I have few thoughts on contributing to Flowable again. But want check if this is something of value to the project.
Idea - Horizontally scale Flowable engine independent of the SQL Database.
High-level thoughts
1 ) Each engine instance only talks to local SQL DB on the node.
2 ) All instances push events into a scalable NoSQL history database.
3 ) Use raft(or zk/etcd) to maintain the Leadership of the cluster of Flowable instances.
4 ) The leader will be responsible to redistribute the Workflows from a failed instance to other instances.
5 ) Redistributed workflows will be replayed from event history and execution continues.
The key here is the support needed in the Flowable engine for replaying the workflow. Is this possible? What kind of effort is needed to do such a thing? Are there any constraints needed on the workflows to accomplish this? Can we even deterministically replay workflows in flowable from the event history?
Also, this can be achieved by using scalable SQL database. But still, there will be contention to own the workflow instances. Correct me if I am wrong here.
This is a huge project. Purely for my research and experiments. Kindly answer my queries.
Since cockroachdb is sql and uses the postgres wire protocol it is not too hard to port, but it may depend on some of the sql used in flowable as there are some constraints on complex queries (I don’t know the depth of this as I have not really looked at it in depth or tested it)
The answer to all your question is yes (except constraints) at least from my 1000 feets point of view without knowing much details. I used process instance replay and playback in simulations. (have a look on flowable-crystalball module or blogs Will we meet our KPIs? – crystal processes )
I would expect that you will need e.g. service call mocking (I used in in simulations too) and ++.
@h7kanna I’m a newbie for flowable. This is a very good point, but no flowable team member join to discuss…
How about your research about flowable, I’m interested in it. tks
The original question is interesting, and part of our effort on MongoDB and CoackroachDB has been to see what different behaviours you get with these newer DB architectures. Transactionality is still important if you want to avoid a lot of modeling of compensation flows, and rolling your own equivalent is always going to be a major challenge and a lot of experimentation. Meantime, there’s plenty of other areas we want to explore in the BPM space, but we’ll be interested in h7kanna’s investigations.
Yes, as Flowable uses database transactions this means that when some error happens along the way, the state is rolled back to the latest stable state. The engines have been architected for this purpose exactly: to bring processes from one state to the other in a guaranteed ‘business atomic’ way (i.e. not technically atomic, but a group of steps moves together forward or not), in a way that makes it very easy to add these behaviors to your own application.
