Environment:
-
Flowable Version: 7.2.0
-
Cluster: 2 Nodes (8 CPUs each) sharing a single database.
-
Scenario: Parent Process A uses script tasks with
Thread.sleepto stagger calls to Child Process B. Process B contains several script tasks that alsoThread.sleepfor 1 minute. -
The Issue: Lack of load balancing. Node 1 locks every job created under its own
LOCK_OWNER_ID.
Technical Observations:
-
Job Acquisition: Even though Node 1 only has 8 CPUs and can only execute 8 script tasks concurrently, it sets itself as the
LOCK_OWNER_for every single job (40+ job) in theACT_RU_JOBtable immediately upon creation and for future jobs created from the same process. -
Queue Capacity Ignored: I have set
flowable.process.async-executor-thread-pool-queue-capacity=10andflowable.process.async-executor.thread-pool-queue-size=10. I expected Node 1 to stop acquiring/locking once the 10 queue slots were full, but it continues to lock the entire burst of jobs. -
Node 2 Idle: Because Node 1 has “kidnapped” the jobs by locking them, Node 2’s acquisition thread sees 0 available jobs and remains idle.
-
Task Configuration: All tasks are
async=trueandexclusive=false.
How can I force Flowable to distribute these “excess” locked jobs to Node 2 so the cluster is actually utilized?
Process A
This text will be hid<?xml version="1.0" encoding="UTF-8"?>
>> DEBUG: Thread '${threadName}' is freezing for 10 seconds..."// This freezes the executor thread
Thread.sleep(10000)
println “>>> DEBUG: Thread ‘${threadName}’ is now awake!”]]>
<![CDATA[String threadName = Thread.currentThread().getName()
println “>>> DEBUG: Thread ‘${threadName}’ is freezing for 20 seconds…”
// This freezes the executor thread
Thread.sleep(20000)
println “>>> DEBUG: Thread ‘${threadName}’ is now awake!”]]>
<![CDATA[String threadName = Thread.currentThread().getName()
println “>>> DEBUG: Thread ‘${threadName}’ is freezing for 10 seconds…”
// This freezes the executor thread
Thread.sleep(10000)
println “>>> DEBUG: Thread ‘${threadName}’ is now awake!”]]>
<![CDATA[String threadName = Thread.currentThread().getName()
println “>>> DEBUG: Thread ‘${threadName}’ is freezing for 20 seconds…”
// This freezes the executor thread
Thread.sleep(20000)
println “>>> DEBUG: Thread ‘${threadName}’ is now awake!”]]>
<bpmndi:BPMNDiagram id=“BPMNDiagram_parentprocessCopy”>
<bpmndi:BPMNPlane bpmnElement=“parentprocessCopy” id=“BPMNPlane_parentprocessCopy”>
<bpmndi:BPMNShape bpmnElement=“startEvent1” id=“BPMNShape_startEvent1”>
<omgdc:Bounds height=“30.0” width=“30.0” x=“100.0” y=“163.0”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-08EB496C-2449-4CBB-8ECB-A2C5B4DDEB2E” id=“BPMNShape_sid-08EB496C-2449-4CBB-8ECB-A2C5B4DDEB2E”>
<omgdc:Bounds height=“79.99999999999999” width=“100.0” x=“389.99999418854725” y=“59.99999910593034”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-1E351B2A-4031-4470-8F76-0BA2F4818468” id=“BPMNShape_sid-1E351B2A-4031-4470-8F76-0BA2F4818468”>
<omgdc:Bounds height=“40.0” width=“40.0” x=“164.99999754130843” y=“149.99999776482585”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-59F36139-D31A-4237-A41E-B7E2D30B3E97” id=“BPMNShape_sid-59F36139-D31A-4237-A41E-B7E2D30B3E97”>
<omgdc:Bounds height=“40.0” width=“40.0” x=“644.9999812245372” y=“164.99999508261692”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-D228A6D8-790D-454A-BA0C-14F524FBA53A” id=“BPMNShape_sid-D228A6D8-790D-454A-BA0C-14F524FBA53A”>
<omgdc:Bounds height=“28.0” width=“28.0” x=“704.5999648779635” y=“170.99999754130843”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“testnode02” id=“BPMNShape_testnode02”>
<omgdc:Bounds height=“79.99999999999997” width=“100.0” x=“389.99998837709455” y=“149.9999955296517”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-B0A3FB4A-953A-4D35-A790-0154E2FE9159” id=“BPMNShape_sid-B0A3FB4A-953A-4D35-A790-0154E2FE9159”>
<omgdc:Bounds height=“79.99999999999994” width=“99.99999999999994” x=“389.999982565642” y=“254.99999620020395”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-D69DEFA4-C91D-4BFE-823E-BD46E9212CAE” id=“BPMNShape_sid-D69DEFA4-C91D-4BFE-823E-BD46E9212CAE”>
<omgdc:Bounds height=“80.0” width=“100.0” x=“629.9999593198324” y=“1064.9999877065422”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-3BE09084-B9E0-4900-9917-65C3009CEE7E” id=“BPMNShape_sid-3BE09084-B9E0-4900-9917-65C3009CEE7E”>
<omgdc:Bounds height=“80.0” width=“100.0” x=“614.9999816715721” y=“824.9999754130846”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-1E758DB1-5BE3-42A7-A394-F29D53CD7F0C” id=“BPMNShape_sid-1E758DB1-5BE3-42A7-A394-F29D53CD7F0C”>
<omgdc:Bounds height=“55.000000372529” width=“64.00000055134296” x=“299.9999955296517” y=“150.49999757111075”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-3BFA51A2-7FF3-4F72-A8EA-C1224DD98C9D” id=“BPMNShape_sid-3BFA51A2-7FF3-4F72-A8EA-C1224DD98C9D”>
<omgdc:Bounds height=“50.00000043213362” width=“51.0000007301569” x=“299.9999955296517” y=“209.9999968707562”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-26E82376-A659-4C88-8701-22AE9C8D76D2” id=“BPMNShape_sid-26E82376-A659-4C88-8701-22AE9C8D76D2”>
<omgdc:Bounds height=“43.000000551342964” width=“50.0” x=“554.9999917298557” y=“539.9999919533731”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-AEE75275-0AC8-42BE-843F-24EF4F858742” id=“BPMNShape_sid-AEE75275-0AC8-42BE-843F-24EF4F858742”>
<omgdc:Bounds height=“50.00000043213356” width=“51.0000007301569” x=“554.4999831020835” y=“749.9999776557092”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-FCDC240D-DD77-4A3F-8BEF-BA25621CA7D2” id=“BPMNShape_sid-FCDC240D-DD77-4A3F-8BEF-BA25621CA7D2”>
<omgdc:Bounds height=“80.0” width=“100.0” x=“614.999990835786” y=“479.99997854232856”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-0BE53FDE-D653-4E2F-B1EC-19AA851C1005” id=“BPMNShape_sid-0BE53FDE-D653-4E2F-B1EC-19AA851C1005”>
<omgdc:Bounds height=“80.0” width=“100.0” x=“614.9999816715721” y=“599.9999910593034”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNShape bpmnElement=“sid-23CF6F13-7D40-4B37-903C-29D1C3002634” id=“BPMNShape_sid-23CF6F13-7D40-4B37-903C-29D1C3002634”>
<omgdc:Bounds height=“80.0” width=“100.0” x=“614.9999816715721” y=“704.9999894946815”></omgdc:Bounds>
</bpmndi:BPMNShape>
<bpmndi:BPMNEdge bpmnElement=“sid-340755BA-22ED-4D68-8EB9-854A09719704” id=“BPMNEdge_sid-340755BA-22ED-4D68-8EB9-854A09719704” flowable:sourceDockerX=“20.0” flowable:sourceDockerY=“20.0” flowable:targetDockerX=“25.50000036507845” flowable:targetDockerY=“25.00000021606681”>
<omgdi:waypoint x=“198.63799607658757” y=“176.31021674848748”></omgdi:waypoint>
<omgdi:waypoint x=“299.99999552965164” y=“223.20284387781257”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-B40DD071-A1DB-4F95-B4F1-D4226E72120B” id=“BPMNEdge_sid-B40DD071-A1DB-4F95-B4F1-D4226E72120B” flowable:sourceDockerX=“50.0” flowable:sourceDockerY=“39.99999999999999” flowable:targetDockerX=“20.0” flowable:targetDockerY=“20.0”>
<omgdi:waypoint x=“489.9499941885473” y=“118.86999930000309”></omgdi:waypoint>
<omgdi:waypoint x=“650.4627352160829” y=“179.51612407393947”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-20FA64AA-7671-4550-92F0-3C6D7912560A” id=“BPMNEdge_sid-20FA64AA-7671-4550-92F0-3C6D7912560A” flowable:sourceDockerX=“32.00000027567148” flowable:sourceDockerY=“27.5000001862645” flowable:targetDockerX=“50.0” flowable:targetDockerY=“39.999999999999986”>
<omgdi:waypoint x=“363.9499960809944” y=“181.5499973731391”></omgdi:waypoint>
<omgdi:waypoint x=“389.9999883770945” y=“184.44444062333915”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-5C96B7D1-A024-4852-B6FA-EBF512123036” id=“BPMNEdge_sid-5C96B7D1-A024-4852-B6FA-EBF512123036” flowable:sourceDockerX=“20.0” flowable:sourceDockerY=“20.0” flowable:targetDockerX=“50.0” flowable:targetDockerY=“39.99999999999999”>
<omgdi:waypoint x=“200.65307448464947” y=“165.69230547744849”></omgdi:waypoint>
<omgdi:waypoint x=“389.9999941885472” y=“113.7117637293974”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-6A3A91C1-104E-4C18-BADA-8552CC9962CC” id=“BPMNEdge_sid-6A3A91C1-104E-4C18-BADA-8552CC9962CC” flowable:sourceDockerX=“50.0” flowable:sourceDockerY=“39.999999999999986” flowable:targetDockerX=“20.0” flowable:targetDockerY=“20.0”>
<omgdi:waypoint x=“489.9499883770942” y=“188.8888842838783”></omgdi:waypoint>
<omgdi:waypoint x=“645.4347638847813” y=“185.43369078621043”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-5DD49702-3F62-40C6-9D85-EC902555843B” id=“BPMNEdge_sid-5DD49702-3F62-40C6-9D85-EC902555843B” flowable:sourceDockerX=“50.0” flowable:sourceDockerY=“40.0” flowable:targetDockerX=“20.0” flowable:targetDockerY=“20.0”>
<omgdi:waypoint x=“664.9999816322724” y=“599.9999910593034”></omgdi:waypoint>
<omgdi:waypoint x=“664.999981244138” y=“204.90217686132473”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-FFBEB59C-1CC8-49EC-8E0B-06B5C920FDDB” id=“BPMNEdge_sid-FFBEB59C-1CC8-49EC-8E0B-06B5C920FDDB” flowable:sourceDockerX=“50.0” flowable:sourceDockerY=“40.0” flowable:targetDockerX=“20.0” flowable:targetDockerY=“20.0”>
<omgdi:waypoint x=“664.9999816452759” y=“824.9999754130846”></omgdi:waypoint>
<omgdi:waypoint x=“664.9999812376525” y=“204.90145484310295”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-6020CE75-DD29-4BF1-89AC-4E71903680BD” id=“BPMNEdge_sid-6020CE75-DD29-4BF1-89AC-4E71903680BD” flowable:sourceDockerX=“50.0” flowable:sourceDockerY=“40.0” flowable:targetDockerX=“20.0” flowable:targetDockerY=“20.0”>
<omgdi:waypoint x=“664.999981639641” y=“704.9999894946815”></omgdi:waypoint>
<omgdi:waypoint x=“664.9999812404628” y=“204.90176770475574”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-15E5786E-CE59-4B6F-8F1A-C562B55255A8” id=“BPMNEdge_sid-15E5786E-CE59-4B6F-8F1A-C562B55255A8” flowable:sourceDockerX=“25.50000036507845” flowable:sourceDockerY=“25.00000021606678” flowable:targetDockerX=“50.0” flowable:targetDockerY=“40.0”>
<omgdi:waypoint x=“603.5638727060735” y=“799.9499780878427”></omgdi:waypoint>
<omgdi:waypoint x=“627.2222036597909” y=“824.9999754130846”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-2D8269F6-A43D-478E-B812-CC8861011B0A” id=“BPMNEdge_sid-2D8269F6-A43D-478E-B812-CC8861011B0A” flowable:sourceDockerX=“15.0” flowable:sourceDockerY=“15.0” flowable:targetDockerX=“20.0” flowable:targetDockerY=“20.0”>
<omgdi:waypoint x=“129.85466958315442” y=“176.2966450925603”></omgdi:waypoint>
<omgdi:waypoint x=“167.0512801715902” y=“172.04615218853195”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-D624AFB2-0D41-4B73-BCDE-9D86EA4B1B19” id=“BPMNEdge_sid-D624AFB2-0D41-4B73-BCDE-9D86EA4B1B19” flowable:sourceDockerX=“49.99999999999997” flowable:sourceDockerY=“39.99999999999997” flowable:targetDockerX=“20.0” flowable:targetDockerY=“20.0”>
<omgdi:waypoint x=“489.94998256564185” y=“270.5555513617066”></omgdi:waypoint>
<omgdi:waypoint x=“651.567145474745” y=“191.55074142219925”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-F3B152F4-3046-4D37-A37F-B4574072A3DF” id=“BPMNEdge_sid-F3B152F4-3046-4D37-A37F-B4574072A3DF” flowable:sourceDockerX=“20.0” flowable:sourceDockerY=“20.0” flowable:targetDockerX=“50.0” flowable:targetDockerY=“40.0”>
<omgdi:waypoint x=“196.53715892591333” y=“178.412648168”></omgdi:waypoint>
<omgdi:waypoint x=“614.9999908357859” y=“483.5416467020217”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-BB288F49-8986-4D03-B52E-288C3D9E85CD” id=“BPMNEdge_sid-BB288F49-8986-4D03-B52E-288C3D9E85CD” flowable:sourceDockerX=“25.50000036507845” flowable:sourceDockerY=“25.00000021606681” flowable:targetDockerX=“49.99999999999997” flowable:targetDockerY=“39.99999999999997”>
<omgdi:waypoint x=“350.9499962598086” y=“248.33624317503484”></omgdi:waypoint>
<omgdi:waypoint x=“389.999982565642” y=“268.7991201748517”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-BBCBD707-8CBB-4088-9E02-A307A75F810D” id=“BPMNEdge_sid-BBCBD707-8CBB-4088-9E02-A307A75F810D” flowable:sourceDockerX=“50.0” flowable:sourceDockerY=“40.0” flowable:targetDockerX=“20.0” flowable:targetDockerY=“20.0”>
<omgdi:waypoint x=“664.9999896881741” y=“479.99997854232856”></omgdi:waypoint>
<omgdi:waypoint x=“664.9999817969086” y=“204.90295808422115”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-296EB00B-2935-4B6B-AFBB-257E5E548EED” id=“BPMNEdge_sid-296EB00B-2935-4B6B-AFBB-257E5E548EED” flowable:sourceDockerX=“20.0” flowable:sourceDockerY=“20.0” flowable:targetDockerX=“50.0” flowable:targetDockerY=“40.0”>
<omgdi:waypoint x=“194.07677470008667” y=“180.87313547450918”></omgdi:waypoint>
<omgdi:waypoint x=“631.6086779474654” y=“704.9999894946815”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-26ED17CA-F02D-4453-B2A2-66879B34DEAA” id=“BPMNEdge_sid-26ED17CA-F02D-4453-B2A2-66879B34DEAA” flowable:sourceDockerX=“20.0” flowable:sourceDockerY=“20.0” flowable:targetDockerX=“32.00000027567148” flowable:targetDockerY=“27.5000001862645”>
<omgdi:waypoint x=“203.91455265635298” y=“171.0296751948035”></omgdi:waypoint>
<omgdi:waypoint x=“299.99999552965164” y=“176.25850112478912”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-C4F40405-CFC6-45C4-9966-4C03F06035E6” id=“BPMNEdge_sid-C4F40405-CFC6-45C4-9966-4C03F06035E6” flowable:sourceDockerX=“20.0” flowable:sourceDockerY=“20.0” flowable:targetDockerX=“25.50000036507845” flowable:targetDockerY=“25.00000021606678”>
<omgdi:waypoint x=“192.88024752819922” y=“182.06953832387205”></omgdi:waypoint>
<omgdi:waypoint x=“563.677669321388” y=“749.9999776557092”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-C5A2D04A-9929-4471-A6FF-CCFB9DEA91F4” id=“BPMNEdge_sid-C5A2D04A-9929-4471-A6FF-CCFB9DEA91F4” flowable:sourceDockerX=“20.0” flowable:sourceDockerY=“20.0” flowable:targetDockerX=“14.0” flowable:targetDockerY=“14.0”>
<omgdi:waypoint x=“684.9314048872809” y=“184.99999599774577”></omgdi:waypoint>
<omgdi:waypoint x=“704.5999648779635” y=“184.99999689911272”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-FD3B8E4D-779E-42FD-BD0C-8BC5BE389A5E” id=“BPMNEdge_sid-FD3B8E4D-779E-42FD-BD0C-8BC5BE389A5E” flowable:sourceDockerX=“20.0” flowable:sourceDockerY=“20.0” flowable:targetDockerX=“25.0” flowable:targetDockerY=“21.500000275671482”>
<omgdi:waypoint x=“195.0193815964239” y=“179.9306080664738”></omgdi:waypoint>
<omgdi:waypoint x=“558.3073596163845” y=“539.9999919533731”></omgdi:waypoint>
</bpmndi:BPMNEdge>
<bpmndi:BPMNEdge bpmnElement=“sid-4A9CD1E6-3D3D-4B49-92B2-AB1C49707D6F” id=“BPMNEdge_sid-4A9CD1E6-3D3D-4B49-92B2-AB1C49707D6F” flowable:sourceDockerX=“25.0” flowable:sourceDockerY=“21.500000275671482” flowable:targetDockerX=“50.0” flowable:targetDockerY=“40.0”>
<omgdi:waypoint x=“603.2249737141062” y=“582.949992504716”></omgdi:waypoint>
<omgdi:waypoint x=“621.6858550275167” y=“599.9999910593034”></omgdi:waypoint>
</bpmndi:BPMNEdge>
</bpmndi:BPMNPlane>
</bpmndi:BPMNDiagram>
den