Windows Workflows that restart themselves?

A few weeks ago I got assigned a fun bug. We have a WF-based order system and it seems that occasionally orders that had been send earlier would start all over by themselves.
And sure enough, when looking at the logging I saw that the order would start over. So how did this happen?

First off I noticed it only happened with orders that required a callback in their process. So orders that, at one point or another, would be idle and waiting for an external event. All other orders never gave a problem.
Then I also noticed that the orders restarted themselves after the system itself had been idle for around 30 minutes.
So workflows that were idle were restarting after 30 minutes of system inactivity. They were re-processed when a new order was sent (you'd then see a whole slew of old orders restarting).

Whenever something happens on IIS after around 30 minuted of inactivity I assume it's always related to the fact that after 20 minutes of inactivity (by default) IIS will stop the web service/application to spare resources. The next request that comes in will cause IIS to startup the web service/application again. So it's pretty much safe to assume the problem is related to the web service stopping and restarting.

So why would an order start all over again, after the web service is being restarted. The most likely reason would be that the Workflow Runtime was unable to save its state. So without knowing where the workflow left off, but knowing that the worklfow does exist, it seems logical the workflow would just restart.

We checked the config and it did have the WorkflowPersistenceService configured and loaded. However, it's "UnloadOnIdle" setting was missing. Meaning it defaults to false. Meaning workflows don't unload when they are idle. And more importantly: a workflow is only persisted when you explicitly tell it to, or when it is unloaded.
Since neither happened on our system the workflows never stored their state and restarted when the web service restarted.

Of course this was not figured out that quickly. I assumed the operational engineers would use the configuration we'd send them, so I never bothered to check it. If I had, I would have solved this bug in a matter of minutes. Now it took us days of prodding, testing and praying. Until another developer mentioned he had noticed the config looked almost the same, but not quite the same. *sigh*.
Bugtracking rule #1: Always check the config!

Error loading workflow

Have you ever seen an error message like this when developing a Windows Workflow application?

I have. Lots of times.

I’m not sure what is causing this. We have a project that contains the workflow and we have several other projects that contain the activities we use. Also we do a little inheritance, where all workflow inherit from a base workflow and all activities inherit from a base activity. The only thing they inherit are properties and functions. We do not have a basic workflow with activities which other workflow then add to. Although I would really love to have that!

So far I discovered two possible solutions workarounds:

1. Rebuild the projects containing the workflow classes.
2. Restart Visual Studio.

Solution #1 is preferable, because otherwise, if you have a large solution, the time it takes for Visual Studio to load your solution can get really annoying. So what I did was create a simple small MSBuild script that cleans and then builds all my workflow project. It starts by cleaning the projects and then builds the projects. First building the projects containing the activities and then the projects containing the workflows. Most of the time that worked for me. And you don’t even need to close anything. You can even keep the workflow designer open with the error message showing. After the build it will refresh and if successful the workflow will be shown.

But sometimes that does not work. Then I would have to quit Visual Studio, run my build script and open Visual Studio again.

It’s also very possible that you have a genuine problem or bug. It is pretty easy to create a situation where you end up being unable to open the workflow in the workflow designer and you’ll have to edit the code-behind file by hand.

Now I’m sure that our inheritance aggravates some of the problems as inheritance isn’t officially supported in Windows Workflow (although I heard they might have something for that in WF 4.0, but I haven’t checked that out yet), but I suspect you’ll run into these problems every now and then even if you do not use inheritance.

Seeing a workflow in the database doesn’t mean it’s persisted

At work we are building a big application that uses Windows Workflow. We are delivering versions to Test at certain milestones so we don’t have to build everything first before we can see how good we built it.

We ran into an interesting issue I want to share. But first let me tell you about some technical details. As I said, we use Windows Workflow to use workflows in our application. The application receives an order (through a webservice), does dome checking and authenticating and then finds the correct workflow to handle the order and start that workflow. The workflows we make inherit from a base class. This base class contains 2 or 3 dependency properties that every workflow should have (such as the order data) and also contains some methods that every workflow can use. We also have a base class for some activities, but these mainly contain methods that most other activities will need. A workflow also needs to implement a certain interface, so we can load all the workflows with our Dependency Injection container and can determine which interface we need for which order.

So far so good, aside from some other problems I want to blog about soon it all seemed to work perfectly. Our first milestone was a couple of simple workflows that started, did some stuff and ended. Nothing special and they worked perfectly. If you used Workflow Monitor to look in the database you’d see the workflow. So to us that meant the workflow was successfully persisted.

Then for the second milestone we created a bunch of more complex workflows. These workflows did a call to a webservice and had to wait on the call-back from that webservice. In workflow this is handled very easily using the ExternalDataExchangeService and the HandleExternalEventActivity. We coded it up and started testing.

When the call-back came in, some stuff happened like validation, and eventually the call-back data came to the part where we’d ask the workflow runtime to give use a pointer to the workflow instance that’s waiting on the data and then use the ExternalDataExchangeService to deliver the data to the workflow so it can go on with it’s process. But instead of just working, the workflow runtime told use that the requested workflow couldn’t be found in the persistence store. We checked with the Workflow Monitor and we saw the workflow. It’s status set to running (or idle). But the runtime insisted it was not there.

Huh??

I did some digging and finally noticed an error in our log files. Some error about an interface not being serializable which caused the persistence to fail. Turns out that what we see in the Workflow Monitor is Workflow Tracking and not Workflow Persistence. In fact, unless you explicitly tell it to do so, workflows will not be persisted. You can either explicitly tell the Workflow Runtime to persist your workflow, or you can use some activity that will cause it to be persisted. Which is what (as far as I know) most activities do that will cause your workflow to be paused. So in our first milestone nothing caused the workflow to be persisted, but in our second milestone we caused the workflow to pause at the HandleExternalEventActivity which meant the workflow had to be persisted, which then failed.

So what was the problem? Well, all fields and get+set properties on a workflow or activity have to be serializable. So it must have the [Serializable] attribute tacked onto its class or interface and onto every class or interface that is inside that class or interface. Since we set up some references to some logic classes (Logging Service, Order Service, etc) in the base class, they would also need to be serializable.

We solved the problem by refactoring all the service fields into read-only properties (‘get’ properties) and not store the reference in the workflow or activity, but rather ask our Dependency Injection container each time we need one through a static method.
We also removed some other classes that where stored as private fields and just instantiate them within a method as they are needed and a few classes were made serializable.

The biggest downside of this is that Windows Workflow demands a very strict versioning scheme, so this means that we now have a few more assemblies that require careful versioning.