At least temporarily, I've decided not to write my own persistence store, due to problems with setting the state from scratch. Since the desire not to take the application down for several hours due to an upgrade still held, I opted to do on-the-fly rebuilding of the workflow instances when could no longer be restored from Persistence correctly. This was a bit tricky to implement, due to the fact that we were using the WorkflowMediator described last year on OdeToCode, which was not designed to solve this problem.

Part of the problem was due to the dependency order demanded by the project organization. Namely this:

As should be obvious, I can't make the mediator depend on the Rebuilder, because that would create a loop in my graph, and Visual Studio won't allow it (for good reason). However, there is one way, I can get the rebuilder accessible to the Mediator without breaking anything. Delegates. I simply change the method declaration:

        public CombinedResults RunWorkflow(Guid workflowId)
        

to:

        public CombinedResults RunWorkflow(Guid workflowId, Func rebuild)
        

Func is a special type in .NET which defines that the argument is a delegate, in this case a delegate that takes a single argument (Guid) and returns a boolean. Calls to this method now just include a reference to a function which matches that signature. I now declare the method as follows:

        public CombinedResults RunWorkflow(Guid workflowId, Func rebuild)
        {
            WorkflowInstance instance;
        
            try
            {
                instance = workflowRuntime.GetWorkflow(workflowId);
            }
            catch (PersistenceException p)
            {
                if (rebuild != null)
                {
                    rebuild(workflowId);
                    instance = workflowRuntime.GetWorkflow(workflowId);
                }
                else
                {
                    throw p;
                }
            }
        
            BusinessResultsInContext = BusinessResults.Pass();
            bool result = WorkflowScheduler.RunWorkflow(instance.InstanceId);
        
            return CombinedResultsInContext;
        }
        

For more information of the ResultsInContext methods, just read the Ode To Code article linked above.

The new part is th try-catch block. The basics are simple, if the load fails due to a problem in the persistence store, I want to rebuild the workflow, if I've been given a rebuild delegate, and the try to load it again. If this fails, I allow the exception to bubble up. Rebuilding is simple:

        public bool RebuildSingleCourse(Guid instanceId)
        {
            if (course.WorkflowGUID == Guid.Empty) return true;
            _persistenceStore.InstanceStates.DeleteAllOnSubmit((from i in _persistenceStore.InstanceStates
                                                                                 where i.uidInstanceID == course.WorkflowGUID
                                                                                 select i));
            _persistenceStore.SubmitChanges();
        
            var parameters = new Dictionary() { // define Dictionary stuff in here
                };
        
            WorkflowInstance instance = _workflowRuntime.CreateWorkflow(typeof(WorkflowType), parameters, instanceId);
            instance.Start();
            CourseService.Instance.RaiseStartCourseEvent(instanceId);
            bool result = _workflowScheduler.RunWorkflow(instanceId);
        
            StateMachineWorkflowInstance statemachine = new StateMachineWorkflowInstance(_workflowRuntime, instanceId);
            try
            {
                statemachine.SetState(GetStatus(instanceId));
                result = _workflowScheduler.RunWorkflow(instanceId);
                statemachine.WorkflowInstance.Unload();
                return true;
            }
            catch (Exception e)
            {
                return false;
            }
        }
        

There is one tricky part to this: the GetStatus method in the SetState. What we do, is we store the status as an enumerated type in another table (as I said yesterday, we have all the data necessary to re-instantiate these objects elsewhere). If you want to juggle different versions of the WorkflowType library, you can follow the instructions in the blog post I linked to yesterday, his solution is good (and was the basis of my own), but our data was already organized such that I didn't need the same complexity he had.

This is far from an ideal solution. The rebuild does take about a second, though hopefully those rebuilds will be rare, but due to the way workflow was implemented, this is an easier solution for many. I still want to build a custom persistence store, but this solution provides me the reliability I want right now, with far less work.