Hi All,
I have a multi AZ DB setup using RDS.
If flowable is trying to acquire async jobs or trying to write a state transition in DB and at that very moment, if the primary DB goes down how does Flowable behaves?
Since we are using Spring boot and Hikari, the connection pool has a max Life time which is set to currently 1 minute to give RDS time to failover to another node.
But during this 1 minute, will we lose jobs? Will Flowable retry?
How can we recover these lost jobs?
If all is set up correctly, the job will be in the database and it will be acquired when a valid connection to the db is created. This is mostly the job of the Hikari threadpool, not Flowable’s.
Thanks @joram for your reply. But in my test scenario the job was already acquired and then I emulated the failover.
I could see this error in the logs
Error updating database. Cause: org.postgresql.util.PSQLException: ERROR: cannot execute INSERT in a read-only transaction. The error may exist in org/flowable/db/mapping/entity/Execution.xml.### The error may involve org.flowable.engine.impl.persistence.entity.ExecutionEntityImpl.bulkInsertExecution-Inline.### The error occurred while setting parameters.### SQL: insert into ACT_RU_EXECUTION
So basically node 1 which was the writer node failed over to node 2. Node 1 is currently a reader node. And the maxlifetime of the connection pool is not reached so the connection pool has not acquired the new connection to node 2 yet.
Is there a way to re-trigger this job again? Will it retry?
Yes, the current transaction will be rolled back (or will never be committed) and the job will be retried later.