Lock Owner Issue

Hi Colleagues,

In our MySQL db we see that all the job requests are going to single lock owner and hence jobs are not moving at all until we set this lock_owner as NULL and make it available for next job in the pipeline. Kindly help us to get this solved as soon as possible.
As of now we are executing the stuck jobs manually.
The locks at OS level looks like below.
Most of the jobs are captured by single lock_owner here: 3f372c61-ad0d-4f98-875a-3f2d761b1d62

Regards,
Preeti Shirur

Lock owners of a job are reset by the reset thread (see flowable-engine/ResetExpiredJobsRunnable.java at main · flowable/flowable-engine · GitHub), by default it should reset jobs when the exp_time is passed.

Are you seeing exceptions in the log around resetting? Which Flowable version are you running?

Hi Joram,
The lock owner of a job are getting reset by reset thread.
Whenever the job acquisition happens, we can identify the new lock_owner is getting locked to the job but the job execution is not happening(in our case we have call activity to a third party API). We are able to execute the job manually by clicking on execute button in admin.
We usually notice this issue only in parallel gateway so far even though we have set all events and call activities are asynchronous and joining gateway as asynchronous plus exclusive.
You can refer the screen shot below for your reference.

Regards,
Preeti Shirur

That’s very strange, when a lock_owner is set, that job is taken by that particular async executor to be executed. Which button are you referring to?

That all sounds not out of the ordinary. Do you have a reproducible model or set of steps? With the current info, nothing sounds off to be honest.

HI Joram,

I am referring to the below mentioned execute job button, with which we are executing jobs manually whenever it gets stuck.

Then it’s even weirder … some things that come to my mind:

  • You are running with the async executor enabled on all Flowable instances?
  • How many Flowable instances do you have?
  • How long does your job run? Is it a long-running job?

Hi JOram,

Please find my answers below.

  • You are running with the async executor enabled on all Flowable instances?–>Yes
  • How many Flowable instances do you have?–>only one
  • How long does your job run? Is it a long-running job?–>there are few jobs which runs for few seconds and also there are jobs which run for 6 hours. So we have set the LOCK_EXP time to 24hours since default was only 5minutes.

Regards,
Preeti Shirur

Hi Joram,

Can you please help us?

Regards,
Preeti Shirur

Hey @preetishirur1,

How is you async executor core and max pool size set?

Having jobs that run for 6 hours is something that we do not recommend. This means that a database transaction will be open for 6 hours. This can lead to various problems due to transactions running too long.

Cheers,
Filip