Moving a sharepoint workflow between solutions – “Failed on Start (retrying)”

The following blog is talking about an issue I had with moving a workflow. It does not talk about how to create workflows. If you want some really good information on that you can find some great tutorials by Robert Shelton here http://rshelton.com/archive/2007/10/05/sharepoint-document-workflow-with-visual-studio-workshop-documents-download.aspx. I really recommend going through his walk-throughs if you are learning how to build SharePoint workflows in Visual Studio 2008.

Today I attempted to take a workflow built with Visual Studio 2008 and move it to a WSPBuilder workflow. The reason I wanted to do this was because I had a nice Visual Studio solution for SharePoint already going utilizing WSPBuilder. Plus, the Visual Studio workflow solutions make it hard to deploy the association aspx pages easily without another web project.

Here are the details of the two solutions:
Solution 1 (original) – built with the Visual Studio 2008 SharePoint Sequential Workflow template
Solution 2 (new) – built with Visual Studio 2008 WSPBuilder.

So, to start I created a new sequential workflow in WSPBuilder and this built out my feature and initial workflow. Next, I created my aspx page in the layouts folder (this is the association page for my workflow). Next, I created the .cs page that will be the code behind for this aspx page. Now, this isn’t like a regular code behind because we aren’t in a web project and we are using SharePoint. This code behind will be in the GAC and the aspx page needs to reference it like this:

<%@ Page language=”C#” MasterPageFile=”~/_layouts/application.master”  
Inherits=”DocumentAssociationForm, DocumentApprovalWorkflow, Version=1.0.0.0, Culture=neutral, PublicKeyToken=5017f6ecf55b35e7″  %>

Notice my “Inherits” points to the assembly information and my MasterPageFile points to the application.master (because this page will live in the _layouts folder.

Now that I have my association pages built I needed to build out the workflow. I copied over the workflow I created in the original solution and placed it in the WorkflowCode folder (replacing the file that was put there by WSPBuilder when I created the workflow feature). Then I just made sure my feature.xml and elements.xml file were configured correctly to run this workflow. So, at this point I thought everything was ready to go. I used WSPBuilder to deploy my solution and I activated my feature on the site. I instantiated the workflow with my association data form I created and I was ready to test the workflow out. I went to run the workflow on an item in my list and I got this error: “Failed on Start (retrying)”. This made no sense to me because the same workflow worked fine from my original solution. So, was WSPBuilder causing the error? Did I mess up something? I really didn’t know. I then went to the SharePoint logs and found this error in them: “The workflow failed validation”.

Ah, my first clue. When a workflow fails validation, that usually means a .rules file is messed up. But, my .rules file seemed to be setup just fine. So, back to the drawing board.

After eliminating every possibility I went and took a look at the project files (.csproj) and I opened them up in notepad. I compared my new solutions file against my old solution and I noticed something. At the very end of the file on the original solution there was this line:

<Import Project=”$(MSBuildBinPath)\Microsoft.CSharp.targets” />
  <Import Project=”$(MSBuildExtensionsPath)\Microsoft\Windows Workflow Foundation\v3.5\Workflow.Targets” />

But, this was not in my new solution. So, it looks like the project file didn’t get notified to work with WF and version 3.5 specifically. I fixed the project file manually and everything worked.

Wow, that was a pain. But, in the end I have this workflow integrated into my larger solution so everything gets deployed at the same time. Thus, I don’t need a separate project just for my workflow and I don’t need a separate project just for my association form. It is really nice, but I almost put my head through the wall figuring it out.

SharePoint site mysteriously stopped showing up

At my client today the weird stuff started happening. We have a Publishing Portal that gets created through a wsp solution. The main page of this portal needs anonymous access so we have to turn that on at the Web Application level. The same exact solution is on our development machines and we went to move it to a staging environment. We created the custom portal site in the staging environment and didn’t do anything else in SharePoint except turn the anonymous access on at the Web Application level (just like we do on the development machines). However, when we went to the site the search box didn’t show on the page – there was just blank space in it’s place.

I am refering to the search box on the actual pages in the site:

When I did a view source on the page and looked for the search section where this should of been I saw this:

<td valign="top">
<div WebPartID="00000000-0000-0000-0000-000000000000"
    HasPers="true" id="WebPartWPQ1" width="100%"
    OnlyForMePart="true" allowDelete="false" style="" >
 </div>
</td>

So, I found this very strange and chalked this up to portal features not being installed correctly on the box. This is because this particular search box only works with a portal. According to it’s documentation it looks for features turned on at the farm and if it doesn’t find any there it looks for the featre at the web app and it continues this process through the site collection and then finally to the actual site.

Then, while I was researching this the client came to me and said there are more issues. They went to do an approval process, on the actual publishing page, and got an error that said “Unexpected error occured”. In fact, we also lost all of our css styles too, which was really weird.

At this point I realized I was going down the wrong path with the portal features, we seem to have a much bigger issue. So, I had them turn their logging up to verbose (this might have been overkill, but I wanted to make sure we got our log).

To do this: go to central administration – operations – diagnostic logging and then set the event throttling like below:

After the event logging was turned on we recreated the error one more time and I went to the logs. The logs had messages about not being able to connect to the SQL Server Session Database.

So, at this point I realized it is a connection issue, which means app pool issue usually. When we went to the app pool account we saw that it was set to Network Services. However, the client had already set this to the right account. So, how did it turn back?

Basically, in SharePoint you try not to modify IIS directly if you have the choice. You try to let SharePoint do it for you. In this case the app pool account was set directly in IIS. However, the services account wasn’t set on SharePoint. This is at Central Administration – Operations – service accounts. You must set this account correctly. In our case we were setting anonymous access to our web application, through Central Administration. This means SharePoint has to make modifications to IIS directly. Well, when SharePoint makes these modifications to IIS, it sets IIS exactly like you have it setup in Central Administration. So, changes you make directly in IIS can be overwritten. The following article gives a good explanation of this:

http://blogs.msdn.com/joelo/archive/2007/01/02/relationship-between-the-iis-metabase-and-sharepoint-configuration-database.aspx

So, what did we learn today – do not modify your IIS when dealing with SharePoint sites. Myself and this particular client actually knew this, but there are other mitigating factors in which they have to modify IIS. They just were not aware of the service accounts in Central Administration. If you do modify IIS directly be really careful and make sure SharePoint can “recreate” the settings if you make any changes in Central Administration. Also, if you are dealing with a farm, be careful that changes get propogated throughout the web front ends.

When to dispose of objects in SharePoint API

When developing against the SharePoint API we have many ways of obtaining object references to sites or site collections. We can use the “SPContext.Current.Site” to get our current site collection. We can use “SPSite siteColl= newSPSite(http://server)” to open a particular site collection. We can use “SPContext.Current.Web” to open the current site. We can use “SPWeb web = siteColl.OpenWeb()” to open a specific site on the site collection.

However, did you know you need to dispose of the objects some of the time, but not all of the time? SPWeb and SPSite implement IDisposable, so it would seem reasonable that we should dispose of them. On a side note: I recommend using “using” statements to dispose of objects (it is the easiest), but try catch blocks work also.

So, which ones do we need to Dispose of and which ones should we leave alone. The answer is anything that uses the current context should not be disposed of. Thus, “SPContext.Current.Web” should not be disposed of and “SPContext.Current.Site” should not be disposed of. However, the other statements (which are basically newing up the objects) should be disposed of every time. The reason is SharePoint keeps an internal list for everything in the “SPContext”. Then, when the page is completed it tries to dispose of all the objects in that list. If you have already disposed of these objects then things will break. However, if you new the object up yourself, it is not put into these internal lists and you will need to dispose of it or you will start getting log errors and warnings about too many SP objects open.

And, as one of my collegues at RDA (Deepak Gupta) recently pointed out to me, there is a commonly missed usage of this – loops.

Example:

foreach(SPWeb web in SPSite.AllWebs)

{

     …do some processing

    web.Dispose(); //this is required

}

Developing SharePoint Solutions in Visual Studio

SharePoint is built on top of .net, but is not like a regular webpage. SharePoint uses a process of storing information in databases and using files on the server (in the 12 hive – which we will talk about later). When a SharePoint site is created it uses the files in the 12 hive to dynamically create all the information in the database to create the actual site. Some of the information it stores in the database is just references to files on the server (in the 12 hive). Thus, when developing for SharePoint we should modify the files in the servers 12 hive to create customizations (actually, we should never modify the out-of-the-box files on the 12 hive, we should just create new files for our customizations). This way our SharePoint pages will have references to our custom files in the servers 12 hive and will use those custom files to create the site, as well as keep the references to these custom files. Keeping these references around is a concept call ghosting. If you try to customize SharePoint any other way (SharePoint designer, site templates, etc…) you will un-ghost your site. This basically means you are losing the references to the 12 hive and it makes a copy of everything directly in the database. This can be a bad thing because you won’t have one place to make changes for all your sites. On a side note: I have heard the argument un-ghosting is not that bad because of master pages (as long as all your sites reference the same master page). I don’t buy this argument, because there are changes you can make outside of the master page that you want to keep ghosted still. An example of this would be your web zone layout in your aspx pages.

So, you can create site definitions and features in .net and push them out to the servers 12 hive in order for SharePoint to use them. Site definitions are custom xml files and aspx pages that create the webpages for SharePoint and reference the Features to activate within the SharePoint site. Features, which can be activated on the site definition, wrap all the actions we can do on SharePont. If you want to create lists – wrap them in a feature that can be activated. If you want to create webparts – wrap them in a feature that can be activated. Basically, if you want to create things in SharePoint – wrap them in a feature that can be activated. One of the great advantages of building custom Site Definitions is that we have complete control over which features get automatically activated on our site (thus, the features do not have to be manually turned on in the SharePoint site settings).

Site Definitions should always be used to create sites if you want to do true development modifications. You should not use Site Templates if you are a developer (believe me, you will regret using Site Templates if you ever have to modify them or if something goes wrong in them). SharePoint is built upon this concept of Site Definitions and Features for all of the out-of-the-box sites that you can create. Thus, following the SharePoint model of creating Site Definitions is the best option available to us as developers.

Below I am showing how to create a custom SharePoint solution in Visual Studio. I am leveraging a third party tool called WSPBuilder. Using WSPBuilder, or any third party tool, is not required to build solutions, but I have found that it makes life easier. This solution will create features and site definitions and package them up into a wsp file. A wsp file is like an install file for SharePoint (in fact, it actually is a cab file renamed). The acual feature we will be creating in this demo is a feature to deploy a master page to the master page gallery in the SharePoint site. However, following these techniques you can deploy any feature and activate it in a site definition.

1.   Installs

      The recommended way to develop in SharePoint is a Virual Environment – whether VPC or VMWare or any other kind of Virtual Environment. The reason is you need to develop on a SharePoint machine which needs to be on a server. And, it is really hard to get a dedicated server machine just to develop on. If you want to share a server with other developers it is really hard to develop in SharePoint because of the constant App Pool Recycles that are needed. Also, a virtual environment lets us make backup points for the whole server, so if we mess up the environment we have a good snapshot to roll back to. Thus, a virtual environment, in which each develop can have their own virtual server, with their own virtual SharePoint environment is the best way to develop in SharePoint.

a.      WSS or MOSS server

b.      Visual Studio 2005 or 2008

c.      WSPBuilder (version 1.01)

 

2.    Start Visual Studio and choose a new WSPBuilder project.

 

 

3.    Create a folder structure for the solution that matches the folder structure for the SharePoint 12 hive (this is really important – WSPBuilder works on the premise that the Visual Studio solution matches the structure of the SharePoint 12 hive). For now we are just going to create the following folders:

a.   1033 – holds the xml files that will tell SharePoint about our new site definitions

b.   Features – holds the new features we are going to build (i.e.: lists, webparts, master pages, etc…)

c.    Layouts – holds our reference files (i.e.: images, css, js, etc…)

d.   Site Templates – holds our site definitions

 

Server 12 hive structure

 

 

VS Solution

 

 

4.       Create a new site definition based off the team site definition in SharePoint.

a.   Copy the team site definition folder from the 12 hive:

C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\SiteTemplates\sts

b.   Paste the folder in the Visual Studio solutions SiteTemplates folder.

c.   Rename the sts folder to something unique (DemoCompanySiteDef for this example)

d.   Delete the defaultdws.aspx file because we aren’t going to use it for this example.

 

e.    Open the default.aspx file and change the master page to a new master page. Call this custom.master (yes, I know that this is not the name of our master page. We are going to configure SharePoint to do a string replacement of custom.master with DemoCompany.master in later steps). We will create this master page in the next step.

 

f.     Open the Onet.xml file (located in the xml folder) and clean it up for our simple site definition.

The Onet.xml file is the file that tells SharePoint how to create the site definition. Whenever you create a new site in SharePoint it reads from the Onet.xml file to determine how to create the site. It is important to note that SharePoint is reading from the Onet.xml file when the site is being created. So, if you already have created a site, you can’t go back to the Onet.xml file, make changes, and expect it to show up on your already created sites. On a side note: I heard that Onet stands for Office Net.

                                   i.     Scroll down to the Configurations XML nodes and remove 2 configurations:

1.       ID=1  - Blank

2.       ID-2 – DWS

      Be very careful not to remove ID=0 – Default because this is the configuration we are going to use for this site definition. When removing the other two get rid of the whole xml tag as well as everything within their xml tag.

                                   ii.    Scroll down to the Modules XML nodes and remove 2 modules:

1.       DefaultBlank

2.       DWS

      Once again, be very careful not to remove the Default module because this is the module we are going to use for this site definition. When removing the other two get rid of the whole xml tag as well as everything within their xml tag.

5.    Create the new master page feature

The master page will be created as a feature. The reason for this is that if the master page is in a feature the page can be added to the master page gallery of SharePoint. This allows you to set out-of-the-box pages to the master page or your own custom site definitions. It also allows you to turn on or turn off the master page features on specific sites. Basically, having it in a feature (rather than directly in the site definition or in the layouts folder) gives you much more flexibility.

a.    Create a new folder under features and call it something unique for your master page. For this example we will call it DemoCompanyMasterPage.

b.    Create a folder under the DemoCompanyMasterPage called “MasterPages”.

                            i.      Copy the general SharePoint master page from the 12 hive: C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\GLOBAL\default.master

                           ii.      Paste the file under the “MasterPages” folder in the Visual Studio solution.

                           iii.     Rename the file to something unique (DemoCompany.master for this example).

c.      Create two xml files under the DemoCompanyMasterPage folder.

                           i.      Feature.xml – this will give a unique GUID to the feature (which we will reference later). This file also gives some important information about the Scope of the feature. Lastly, this file tells SharePoint where all the other files that make up this feature lives. This file must be called feature.xml (this is because SharePoint will read all files called feature.xml from the Features folder).

Please note that the Scope=”Site”. This is very important. This is telling SharePoint that this feature is for the site collection. If that Scope=”Web” then it would be telling SharePoint that this feature is for an individual site.

 

                           ii.      Elements.xml – this file is telling SharePoint to place the new master page into the master page gallery. This file does not have to be called elements.xml (unlike the feature file which has to be called feature.xml). It just has to match the name located in the “ElementManifest” of the feature.xml file.

 

6.    Modify the site definitions onet.xml file to tell the site to install and use this new masterpage feature when a new site is created.

a.    Open the onet.xml file in Visual Studio (located at SiteTemplates/DemoCompanySiteDef/xml).

b.     Find the xml node that says Configuration ID=0 Name=”Default”.

c.     Add in a reference for the new master page: 

Note: SharePoint will do a string replacement of any aspx page with ~masterurl/custom.master with the reference in CustomMasterUrl. And, it will replace any aspx page with ~masterurl/default.master with the reference in MasterUrl.

d.   Find the SiteFeatures XML node.

e.   Add a new Feature node in there that matches the GUID of the feature.xml file we created for our master page.

 

This section of the onet.xml basically tells the site to activatethis particular feature on this particular site when it is being created. The reason it goes into the SiteFeatures (instead of the WebFeatures) is because this particular feature was scoped for a Site Collection (refer to step 5c where we created the feature and scoped it for a Site Collection). If the feature was scoped for an individual site then it would go in the WebFeatures.

 

7.    Create the WebTemp*.xml file

WebTemp*.xml files tell SharePoint about site definitions. When a new site is being created in SharePoint the “Select Templates” list box is populated by anything it can find in the WebTemp*.xml files. There can be multiple of these files in SharePoint as long as they are in the 12/Template/1033/XML folder. SharePoint will basically read this folder for any file that starts with the word WebTemp.

a.    Add a new folder under 12/Template in our Visual Studio solution called 1033. Then, add a new folder under 1033 called XML.

b.    Add a new file to the 12/Template/1033/XML folder in our Visual Studio solution. This file must start with WebTemp. We will call this file WebTempDemoCompany.xml.

 

c.       Add in the proper xml to tell SharePoint about our new site definition

 

A few things to note:

-    The Name must match the site definition folder exactly. We used the name DemoCompanySiteDef. This matches our site definition name exactly.

 

-    The Configuration ID must match the configuration ID in our onet.xml file of our Site Definition. We used configuration id 0.

-     The display category will create a new tab on the new site screen in SharePoint.

-     The ID has to be unique and SharePoint has already reserved certain IDs. To be safe use an ID over 10000 and make sure it is unique.

 

8.    Now, at this point (in a regular SharePoint solution) we would create manifest.xml and ddf files. These are files SharePoint needs to install the solution properly. However, since we are using WSPBuilder, it will do that for us.

a.     Right click the project and click WSPBuilder – Build

 

b.    Verify your Web Application is created on your local SharePoint environment. Note: the Site Collection should not be created at this point, just the Web Application. And, you only need to do this once.

c.     If the build is successful, right click the project and click WSPBuilder – Deploy.

This is needed because the build command in WSPBuilder just builds a wsp file. This file can be sent to other machines and you can use stsadm scripts to install it on other SharePoint farms. However, in development, we need a quick way to deploy it to the current SharePoint farm. The WSPBuilder – Deploy command will deploy the wsp file on all the Web Applications in the local SharePoint farm.

 

9.    Open up SharePoint Central Administration and create a new Site Collection with the site definition we just created.

 

Now you can go back and modify the master page or default.aspx page as you wish. Just be careful not to mess with the ContentPlaceHolders too much. SharePoint reserve these for certain actions on some of its internal pages. After every change you make you can rebuild by clicking WSPBuilder – Build and update the solution by clicking WSPBuilder – Deploy. Then you can just go to the website and see your changes. Just be aware that everytime you deploy WSPBuilder does an app pool recycle (to ensure the changes get pushed out through IIS), so you might have to wait a few seconds to see your changes (during that few seconds the site could say Service Unavailable).

It is also important to note that the first time you deploy with WSPBuilder it does a true SharePoint install. However, every subsequent time it does a SharePoint solution update. Solution updates work for most changes, however, there are some things it can’t do. So, if you run into errors it is sometimes good to do WSPBuilder – Uninstall and then a WSPBuilder-Deploy to do a real install in certain situations when you are making changes.

If you are curious to what a SharePoint solution is doing, then just check out the files in the 12 hive:

C:\Program Files\Common Files\Microsoft Shared\web server extensions\12

You will notice that all the files we created in our Visual Studio solution have been “pushed” down to the folders in the 12 hive. The WSP knows to “push” these files there. The WSP also knows to tell SharePoint that these files exist. This is exactly how SharePoint develops it’s out of the box site definitions (they are all located in the 12 hive and you can actually go and see them).

Because this blog is starting to get long I am not going to show how to create other features, but the principals are the same:

-      Create the feature and note the “Scope” you set it to

-      Go to your Onet.xml file, of your site definition, and place a reference to the GUID of the feature in the appropriate xml tag (SiteFeatures or WebFeatures).

-      Now that feature is associated with the particular Site Definition every time a new site is created with the selected Site Definition

The other features you can create in SharePointis a pretty long list. These include custom lists, webparts, custom actions, features that call other features, custom code that runs after another feature is created, etc… The key point I am making in this blog is that when you learn how to create another type of feature, you can just added into this solution and tie it to your site definition in the onet.xml file.

Side Notes:

1.    Any file you place in the Layouts folder can be reference in the aspx pages as “/_layouts/{filename}”. I usually recommend placing another folder under Layouts in the Visual Studio solution (just because it makes it easy to find later – you can call this folder the name of your company or the name of your project or anything that is obvious that it contains your custom reference files). Then you can reference everything in there by “/_layouts/{folder}/{filename}”. An example of this is css files in the master page. This is possible because SharePoint sets an IIS path up to the Layouts folder in the 12 hive and calls it _layouts.

2.    If you want to create a custom webpart with WSPBuilder just click Add – new item… WSPBuilder – WebPartFeature. This will create the feature folder, for your webpart, and the .cs class for your webpart. This way you can go directly into coding and not have to worry about the Feature creation. Once you are done coding the webpart just make sure it is in the appropriate scope of your onet.xml file (SiteFeatures or WebFeatures) and it will become available to you in your webpart gallery. Then all you have to do is worry about your dll being installed in the Bin or GAC. If you choose GAC I recommend post-build scripts to move it.

3.    Do not remove Content Place Holders from your master page if you start doing customizations. The reason for this is if you assign other files in SharePoint (such as list pages) to use your master pages, they will assume certain Content Place Holders exist. If they are not there, your site will just break.

 

Update

Since I wrote this I was informed that my demo only works for the top level page (i.e.: the site collection). If you use the code above on a sub-site, it won’t work. The reason is the example deals with the “Site” scope only. There are two ways to fix this for sub-sites:

1. Change the example to “Web” scope. This will place the master page in the master page gallery for each sub-site. To do this you need to change the feature.xml file to scope=Web. And, you need to change the onet.xml file to have the feature registered under the “WebFeatures” instead of the “SiteFeatures”.

2. In the Configuration node of the onet.xml, where we set the CustomMasterUrl and the MasterUrl to _catelags/masterpage/DemoCompany.master – we should replace those to point to ~SiteCollection/_catalogs/masterpage/DemoCompany.master. This approach will just point all the site definitions to use the master page at the root site collection level. 

Speaking in the Northern Virginia area

I will be speaking at two events in the Northern Virginia area. Both of my subjects have to do with SharePoint custom development.

Saturday May 18th – Nova Code Camp : Using .net as a development platform for creating SharePoint solutions

Friday June 27th – SharePoint User Group of Northern Virginia : Developing publishing portal solutions for SharePoint in .net