Team of developers wants to help build process migration feature support

My team has been using flowable for the past year to model and execute many of our critical business processes. Flowable has played a key role in helping us build a solid technology platform for our business. Huge thank you to everyone who contributes to this project!

We need to support process instance migrations, and my dev team has dedicated time in mid September to start building this feature. We’ve looked at how other similar tools do it (Camunda) and have some ideas of how we think it could be done.

I noticed that your roadmap was recently updated to include this feature. Since our timelines seem to overlap, I think it would be great if we could collaborate getting this feature built. I have some fantastic devs on my team and would love for us to build something together that can benefit the community.

How would you recommend we proceed? Do you have a technical plan for how you’d like to see this feature built? Have you categorized the work in such a way that you could offload some of the development to us?

Feel free to PM me as well. Maybe we could jump on a call and talk through the options.

Thanks,
Chris

3 Likes

Hi Chris,

Thanks for your post!
We are currently working on the process instance migration feature, I’ll add another post today pointing to a branch where this work is done. The process instance migration feature is built on top of the change state logic that we added in Flowable 6.2.x.

The idea is that there will be a Java API to trigger process instance migration manually, and that you can provide a JSON migration document in a deployment to trigger process instance migration automatically when doing the deployment.

How do you think you will use this feature in most cases? Using the Java / REST API or by adding a JSON migration document to a deployment? Do you think you need a UI to create the migration document instead of writing the JSON by hand?

I think there are some good options to work together on this. Let’s explore the options here, so everyone can benefit from it and then we can discuss the details on a call.

Thanks!

Tijs

1 Like

Hi Chris,

This is the branch where we are currently developing the process instance migration feature:

https://github.com/flowable/flowable-engine/compare/master...DennisFederico:processInstanceMigration

Maybe you can have a look at the approach we are taking there and come back with ideas?

Best regards,

Tijs

Thanks Tijs,

I’ll take a look a the branch and get back to you with thoughts/ideas.

To answer some of your previous questions:

How do you think you will use this feature in most cases

Ultimately, we’d like to get users onto the most recent process definition version as quickly as possible. This may take the form of auto-migrating old workflows where possible as well as the need to write/specify custom steps that need to be performed as part of the migration.

Using the Java / REST API or by adding a JSON migration document to a deployment?

We have infrastructure in place that allows us to move a portion of traffic to new process definition versions. We perform the rollout in a very deliberate/controlled manner. For us, deploying new code or a new process definition is different from “turning on” a new process definition. I need to look at the code to understand your approach, but we probably wouldn’t want to couple a migration with the deployment of a process definition.

Do you think you need a UI to create the migration document instead of writing the JSON by hand?

Creating the migration document by hand would be fine at first. A UI is pretty low on our priority list.

Thanks,
Chris

I just took a quick look through the code. You’re miles ahead of what we were planning to do as a MVP. Very nice work by Dennis Federico.

The functionality exposed through the runtime service looks like it would be sufficient for our initial use cases. I expect that we may want to migrate a curated/batched set of process instance ids in the future. It’s not a requirement now, and I think would be easy enough to build on top of what you have.

The migration document looks straightforward and is definitely something we could craft by hand. Most of our migrations will involve very few custom activity mappings, because most of them are 1:1 (ie activityFoo -> activityFoo, activityBar -> activityBar). I noticed the doMigrateProcessInstance code commented with:

If there’s no specific mapping, we check if the new process definition contains it already

That will make it even easier for us to create (and review) the migration documents.

I see that you update the process definition id in the history tables when the migration is complete (and I understand why). We’ll want a more advanced mechanism than logs to audit process instances, and know if (and when) they were migrated from a previous definition. Maybe you already support that. Either way, it’s not a blocker for us and is something we could build/insert into the migration process.

You’re pretty far along. Any thoughts on how we could help without getting in your way?

Thanks,
Chris

Just some ideas:

I can imagine a future where more complicated migrations are possible. For example, we may need/want to include custom code that runs as part of an activityFoo -> activityBar migration or support more than a 1:1 mapping. These are definitely more advanced use cases and I’m not yet sure how important that would be to us. This is maybe one area we could collaborate on in the future.

I didn’t see any of the REST endpoint implementations. Those would be easy for us to implement, but I’d be worried about our timelines impacting something that is easy for you to just knock out.

We could help write some of the technical documentation.

If the code is in a semi stable state, we could pull the code, run some migrations and then see how things run in our automated testing environments.

I’d love to hear your thoughts on how we could help.

Thanks,
Chris

Hi Chris,

Thanks for the feedback.
The work in the branch shows the ideas we have with the process instance migration API, but we are enhancing it as we speak with more advanced cases, like moving from a user task on root process level to a user task in a nested sub process, multi instance support, parallel gateway support etc.
Additional logic to define which process definition id the historic activity / task etc was migrated from would be useful for Flowable as well I think, so that’s something we could add.

Including custom code that runs as part of an activity migration would be a nice feature indeed. This is definitely something we could collaborate on.

The REST endpoint implementation will be done when we have completed the initial process instance migration work. I expect it will be there in 2 weeks or so.

Helping with the technical documentation would be really nice. There’s a lot involved in process instance migration, so the documentation is really important.

We think the current code is stable for the simple cases, like migrating to the same activity id in the new process definition, but also migrating to another user task on the same scope in the process definition. If you could already give this a try that would be really helpful as well.

Best regards,

Tijs

Thanks Tijs,

It’s looking like we’ll have some time to contribute mid Sept. I’ll check with my team and report back when I have a more solid timeline.

Thanks,
Chris

Hi there, a thumbs up from our side for this feature. We are integrating flowable into one of our key applications and will have plenty of short and long lived processes where new requirements and resulting migrations are inevitable.

About use cases for this feature we haven’t done a deep analysis yet but think that we need a mix, migrate running processes automatically to their latest version and only cherry picking specific ones to migrate. An API to trigger the migration based on a defined definition would be good so we can integrate it in our app and let the user decide which one to migrate.

Regards,
Holger

Tijs,

We’re going to bring in a little bit of this work in our next sprint starting Oct 1. We’ll start with testing what you have in our automated test environment, as well as updating the documentation and go from there. Does that sound good?

Thanks,
Chris

Hi Chris,

Ok great to hear. We started with a very basic process instance migration documentation that we will deliver as part of the 6.4.0 release on Monday or Tuesday next week. We started working on adding more support with parallel and inclusive gateways, multi instance, sub process / call activity and more. Looking forward to get feedback on the work that has been done until now.

Best regards,

Tijs