Hi Colleagues,
Executable jobs taking significantly more time than usual.
Anything changed meanwhile in flowable with respect to jobs or task handling?
Each model is suppose to finish in seconds but now its taking hours to complete it.
Kindly let us know if you this is a known issue or happening due to some bug or so?
Immediate assistance is highly appreciated.
Regards,
Preeti Shirur
No, on the contrary, job execution has become way faster. See Handling asynchronous operations with Flowable – Part 1: Introducing the new Async Executor
We need way more info to be able to say something: your configuration, your models, type of service task, etc.
No need to mention this, this goes for everyone ;-). Just keep in mind that all help here is best effort.
Regards,
Preeti Shirur
Hi Joram,
I have attached model screen shot and catalina.out log for your reference.
Regards,
Preeti Shirur
Next time, pls just attach the xml file, that’s way easier to read than screenshots of XML.
I can’t spot anything wrong immediately. The error you pasted however does indicate that concurrent threads were active on the same job. This could potentially explain the total instance being slower.
Can you clarify
Hi joram and experts,
I think I should add some more information on the situation and observations we made. Not being a techie, I will ask our techies to add configuration facts below.
We are running 6 levels of processes to build (install and configure) SAP R/3 systems in a hierarchical manner as follows:
Wrapper->Phase->Subphase->Configuration Scenario->Configuration Process->Configuration Step
All levels but the wrapper are called via Call Activities. All higher levels do nothing else but taking decisions, such as skipping or branching, and executing their children in sequential or parallel manner. Only the lowest level, Configuration Step, accesses the server or system and performs some activity. This is either done via some script (usually PERL or SHELL) or via some Tasklists/Task (ABAP classes).
We have some 500 Configuration Steps involved for any given system, what Preeti has shown above is such a Configuration Step. We have run our processes since many months now, without this issue, first in Flowable 6.4, with a mySQL database, now also in 6.6 on the same database.
The issue occurred first time 3 days back in our productive engine (Flowable 6.4), no changes to the infrastructure were done recently. There we have open jdk 13, mySQL 8.0.19 and Tomcat 9.0.27.
But yesterday the same happened in our development and test engine, which is on Flowable 6.6, and open jdk 16, mySQL 8.0.25, Tomcat 9.0.46 (I will ask our techies to confirm this).
The issue occurred on level Configuration Process, when executing the lowest level Configuration Step, and it was first seen in processes with parallel executions of these Configuration Steps, but later also with sequential executions. All gateways are set to asynchronous.
When the issue happened, observation was the following:
In the process diagram
all Configuration Steps were shown as green. However, in Flowable Admin, there were no sub-processes shown in the respective Tab.
Catalina.out showed:
On the Job-Tab, initially, there was no deadletter job visible. Then, but this is not really confirmed somehow or reproducible as such, executable jobs were shown. I was able to click on “execute job”, but this had no immediate effect. In the end I got the impression, that whatever I did there, did not have any influence on the behaviour, but I wanted to mention this here.
Some time later, there was apparently a deadletter job for all 6 sub-processes. However, when clicking on them in the list, Flowable reacted with an error message (cannot find this job) and catalina.out showed:
2021-06-07 11:59:00.472 ERROR 13744 — [o2-443-exec-114] o.f.u.a.r.c.JobClientResource : Error getting job d23b44a6-c776-11eb-9d97-42010aeec52a
In all cases since then, and we have 3 completely independent process models where it happened. Flowable took 10-50 minutes to actually run each of the Configuration Steps one by one. I.e., while such an execution of 6 parallel Configuration Steps took 30 seconds or so in the past, all of a sudden it took 3 hours or so to execute all of them one by one (in a kind of sequential mode). When it happened first in our productive engine, this behaviour was seen inside a multi-instance (sequential) which ran 5 times, with the same behaviour for all 5 process instances at the very same location/model.
Therefore we thought that it’s the process model, but in the meanwhile, as mentioned above, it has happened in 2 other process models in another engine, i.e. both, on 6.4 and 6.6 respectively.
Regards,
Joachim
Hi joram and experts, related to the screenshot of the Configuration Process in my reply, I wanted to share the XML here, but the file does not qualify for being attached!?
Not surer whether this helps:
<?xml version="1.0" encoding="UTF-8"?>-
-
https://wiki.wdf.sap.corp/wiki/x/Ykwyi
-
<flowable:eventListener class=“com.sap.dlm.flowable.ProcessCancelledListener” events=“PROCESS_CANCELLED”/>
+
-
-
flowable:valueNAMESPACE</flowable:value>
-
-
-<flowable:executionListener class=“com.sap.dlm.flowable.DLMScriptExecutionListener” event=“start”>
-<flowable:field name=“language”>
-
</flowable:string>
</flowable:field>
-<flowable:field name=“scriptURL”>
-
</flowable:string>
</flowable:field>
</flowable:executionListener>
-
-
<flowable:in target=“ecsKey” source=“ecsKey”/>
<flowable:in target=“simulate” source=“simulate”/>
<flowable:in target=“LEAD_PRODUCT_TYPE” source=“LEAD_PRODUCT_TYPE”/>
<flowable:in target=“SYSTEMTYPE” source=“SYSTEMTYPE”/>
<flowable:in target=“SYSTEMROLE” source=“SYSTEMROLE”/>
<flowable:in target=“INSTALLTYPE” source=“INSTALLTYPE”/>
<flowable:in target=“CLI_TYPE” source=“CLI_TYPE”/>
<flowable:in target=“SERVICE_LEVEL” source=“SERVICE_LEVEL”/>
<flowable:in target=“NW_RELEASE” source=“NW_RELEASE”/>
<flowable:in target=“NW_SPLEVEL” source=“NW_SPLEVEL”/>
<flowable:in target=“STATUS” source=“STATUS”/>
<flowable:in target=“hostname” source=“hostname”/>
<flowable:in target=“instanceNumber” source=“instanceNumber”/>
<flowable:in target=“targetClient” source=“targetClient”/>
<flowable:in target=“initiator” source=“initiator”/>
<flowable:in target=“EC2_CLIENT” source=“EC2_CLIENT”/>
-
eCS Data Skip
-
eCS Data Go
-
-<flowable:executionListener class=“org.flowable.engine.impl.bpmn.listener.ScriptExecutionListener” event=“start”>
-<flowable:field name=“language”>
-
</flowable:string>
</flowable:field>
-<flowable:field name=“script”>
-
</flowable:string>
</flowable:field>
</flowable:executionListener>
-
-
-
-
<flowable:in target=“ecsKey” source=“ecsKey”/>
<flowable:in target=“simulate” source=“simulate”/>
<flowable:in target=“LEAD_PRODUCT_TYPE” source=“LEAD_PRODUCT_TYPE”/>
<flowable:in target=“SYSTEMTYPE” source=“SYSTEMTYPE”/>
<flowable:in target=“SYSTEMROLE” source=“SYSTEMROLE”/>
<flowable:in target=“INSTALLTYPE” source=“INSTALLTYPE”/>
<flowable:in target=“CLI_TYPE” source=“CLI_TYPE”/>
<flowable:in target=“SERVICE_LEVEL” source=“SERVICE_LEVEL”/>
<flowable:in target=“NW_RELEASE” source=“NW_RELEASE”/>
<flowable:in target=“NW_SPLEVEL” source=“NW_SPLEVEL”/>
<flowable:in target=“STATUS” source=“STATUS”/>
<flowable:in target=“hostname” source=“hostname”/>
<flowable:in target=“instanceNumber” source=“instanceNumber”/>
<flowable:in target=“targetClient” source=“targetClient”/>
<flowable:in target=“initiator” source=“initiator”/>
<flowable:in target=“EC2_CLIENT” source=“EC2_CLIENT”/>
-
-
<flowable:in target=“ecsKey” source=“ecsKey”/>
<flowable:in target=“simulate” source=“simulate”/>
<flowable:in target=“LEAD_PRODUCT_TYPE” source=“LEAD_PRODUCT_TYPE”/>
<flowable:in target=“SYSTEMTYPE” source=“SYSTEMTYPE”/>
<flowable:in target=“SYSTEMROLE” source=“SYSTEMROLE”/>
<flowable:in target=“INSTALLTYPE” source=“INSTALLTYPE”/>
<flowable:in target=“CLI_TYPE” source=“CLI_TYPE”/>
<flowable:in target=“SERVICE_LEVEL” source=“SERVICE_LEVEL”/>
<flowable:in target=“NW_RELEASE” source=“NW_RELEASE”/>
<flowable:in target=“NW_SPLEVEL” source=“NW_SPLEVEL”/>
<flowable:in target=“STATUS” source=“STATUS”/>
<flowable:in target=“hostname” source=“hostname”/>
<flowable:in target=“instanceNumber” source=“instanceNumber”/>
<flowable:in target=“targetClient” source=“targetClient”/>
<flowable:in target=“initiator” source=“initiator”/>
<flowable:in target=“EC2_CLIENT” source=“EC2_CLIENT”/>
-
-
<flowable:in target=“ecsKey” source=“ecsKey”/>
<flowable:in target=“simulate” source=“simulate”/>
<flowable:in target=“LEAD_PRODUCT_TYPE” source=“LEAD_PRODUCT_TYPE”/>
<flowable:in target=“SYSTEMTYPE” source=“SYSTEMTYPE”/>
<flowable:in target=“SYSTEMROLE” source=“SYSTEMROLE”/>
<flowable:in target=“INSTALLTYPE” source=“INSTALLTYPE”/>
<flowable:in target=“CLI_TYPE” source=“CLI_TYPE”/>
<flowable:in target=“SERVICE_LEVEL” source=“SERVICE_LEVEL”/>
<flowable:in target=“NW_RELEASE” source=“NW_RELEASE”/>
<flowable:in target=“NW_SPLEVEL” source=“NW_SPLEVEL”/>
<flowable:in target=“STATUS” source=“STATUS”/>
<flowable:in target=“hostname” source=“hostname”/>
<flowable:in target=“instanceNumber” source=“instanceNumber”/>
<flowable:in target=“targetClient” source=“targetClient”/>
<flowable:in target=“initiator” source=“initiator”/>
<flowable:in target=“EC2_CLIENT” source=“EC2_CLIENT”/>
-
-
<flowable:in target=“ecsKey” source=“ecsKey”/>
<flowable:in target=“simulate” source=“simulate”/>
<flowable:in target=“LEAD_PRODUCT_TYPE” source=“LEAD_PRODUCT_TYPE”/>
<flowable:in target=“SYSTEMTYPE” source=“SYSTEMTYPE”/>
<flowable:in target=“SYSTEMROLE” source=“SYSTEMROLE”/>
<flowable:in target=“INSTALLTYPE” source=“INSTALLTYPE”/>
<flowable:in target=“CLI_TYPE” source=“CLI_TYPE”/>
<flowable:in target=“SERVICE_LEVEL” source=“SERVICE_LEVEL”/>
<flowable:in target=“NW_RELEASE” source=“NW_RELEASE”/>
<flowable:in target=“NW_SPLEVEL” source=“NW_SPLEVEL”/>
<flowable:in target=“STATUS” source=“STATUS”/>
<flowable:in target=“hostname” source=“hostname”/>
<flowable:in target=“instanceNumber” source=“instanceNumber”/>
<flowable:in target=“targetClient” source=“targetClient”/>
<flowable:in target=“initiator” source=“initiator”/>
<flowable:in target=“CT_UA_SECPOL_NAME” source=“CT_UA_SECPOL_NAME”/>
<flowable:in target=“EC2_CLIENT” source=“EC2_CLIENT”/>
-<flowable:executionListener class=“com.sap.dlm.flowable.DLMScriptExecutionListener” event=“start”>
-<flowable:field name=“language”>
-
</flowable:string>
</flowable:field>
-<flowable:field name=“script”>
-<![CDATA[package com.sap.dlm.groovy;
import org.flowable.engine.;
import org.flowable.engine.delegate.;
import org.flowable.engine.runtime.;
import org.flowable.engine.impl.context.;
import org.flowable.engine.cfg.;
import org.flowable.form.api.;
import org.flowable.form.model.;
import java.util.AbstractMap.SimpleEntry;
import java.util.ArrayList;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.text.ParseException;
import com.sap.dlm.utils.;
import groovy.json.*;
import groovy.util.ConfigSlurper;
HashMap < String, Object > allVariables = new HashMap < String, Object > ();
for (Map.Entry entry in com.sap.dlm.utils.OATProperties.getInstance().entrySet()) {
String name = entry.getKey() as String;
String uName = name.toUpperCase();
if (name.equals(uName)) {
allVariables.put(name, entry.getValue() as String);
}
}
String customTable = execution.getVariable(“customTable”);
String resolvedLink = resolveWebServiceLink(allVariables, customTable);
String resolveWebServiceLink(HashMap < String, Object > allVariables, String serviceLink) {
Pattern variables = java.util.regex.Pattern.compile(’\$\{([^}]+)’);
Matcher aMatcher = variables.matcher(serviceLink);
int start = 0, end = 0;
String resolvedLink = “”;
while (aMatcher.find()) {
// a variable found in the link. This should be replaced with it’s value
String varName = aMatcher.group(1);
if (allVariables.containsKey(varName)) {
resolvedLink += serviceLink.substring(start, aMatcher.start()) + allVariables.get(varName);
start = aMatcher.end() + 1;
} else {
resolvedLink += serviceLink.substring(start, aMatcher.end());
start = aMatcher.end();
}
}
resolvedLink += serviceLink.substring(start);
return resolvedLink;
}
SimpleEntry parameter = new SimpleEntry<String, Object>(“ITAB”, null);
if (searchWebServiceJsonData(resolvedLink, parameter)) {
ArrayList exObjList = new ArrayList();
for (Object excludeEntry : parameter.getValue()) {
for (Map.Entry<String, Object> ent : excludeEntry.entrySet()) {
if (“CT_UA_SECPOL_NAME”.equals(ent.getKey())) {
exObjList.add(ent.getValue());
}
}
}
//secpolicies = new String[exObjList.size()];
//secpolicies = exObjList.toArray(secpolicies);
execution.setVariable(“secpolicies”, exObjList);
println("SUCCESS ");
}
boolean searchWebServiceJsonData(String url, Map.Entry entry) {
String[] hierarchy = entry.getKey().split("\.");
URL serviceURL = new URL(url);
HttpURLConnection conn = serviceURL.openConnection() as HttpURLConnection;
conn.setRequestProperty( ‘Accept’, ‘application/json’ )
if (conn.responseCode >= 200 && conn.responseCode < 300) { // Success
JsonSlurper jsonSlurper = new JsonSlurper();
Object serviceMap = jsonSlurper.parseText(conn.inputStream.text);
int count = 0;
for (String nodeName : hierarchy) {
if (null != serviceMap && serviceMap instanceof Map && serviceMap.containsKey(nodeName)) {
serviceMap = serviceMap.get(nodeName);
count++;
} else {
break;
}
}
if (count == hierarchy.length) {
entry.value = serviceMap;
return true;
}
}
return false;
}]]>
</flowable:string>
</flowable:field>
</flowable:executionListener>
-
eCS Data Branch A
-
-
-<flowable:executionListener class=“com.sap.dlm.flowable.DLMScriptExecutionListener” event=“start”>
-<flowable:field name=“language”>
-
</flowable:string>
</flowable:field>
-<flowable:field name=“scriptURL”>
-
</flowable:string>
</flowable:field>
</flowable:executionListener>
-<flowable:executionListener class=“com.sap.dlm.flowable.DLMScriptExecutionListener” event=“start”>
-<flowable:field name=“language”>
-
</flowable:string>
</flowable:field>
-<flowable:field name=“scriptURL”>
-
</flowable:string>
</flowable:field>
</flowable:executionListener>
-<flowable:executionListener class=“com.sap.dlm.flowable.DLMScriptExecutionListener” event=“end”>
-<flowable:field name=“language”>
-
</flowable:string>
</flowable:field>
-<flowable:field name=“scriptURL”>
-
</flowable:string>
</flowable:field>
</flowable:executionListener>
-
-
<flowable:in target=“ecsKey” source=“ecsKey”/>
<flowable:in target=“simulate” source=“simulate”/>
<flowable:in target=“LEAD_PRODUCT_TYPE” source=“LEAD_PRODUCT_TYPE”/>
<flowable:in target=“SYSTEMTYPE” source=“SYSTEMTYPE”/>
<flowable:in target=“SYSTEMROLE” source=“SYSTEMROLE”/>
<flowable:in target=“INSTALLTYPE” source=“INSTALLTYPE”/>
<flowable:in target=“CLI_TYPE” source=“CLI_TYPE”/>
<flowable:in target=“SERVICE_LEVEL” source=“SERVICE_LEVEL”/>
<flowable:in target=“NW_RELEASE” source=“NW_RELEASE”/>
<flowable:in target=“NW_SPLEVEL” source=“NW_SPLEVEL”/>
<flowable:in target=“STATUS” source=“STATUS”/>
<flowable:in target=“hostname” source=“hostname”/>
<flowable:in target=“instanceNumber” source=“instanceNumber”/>
<flowable:in target=“targetClient” source=“targetClient”/>
<flowable:in target=“initiator” source=“initiator”/>
<flowable:in target=“EC2_CLIENT” source=“EC2_CLIENT”/>
-
-
<flowable:in target=“ecsKey” source=“ecsKey”/>
<flowable:in target=“simulate” source=“simulate”/>
<flowable:in target=“LEAD_PRODUCT_TYPE” source=“LEAD_PRODUCT_TYPE”/>
<flowable:in target=“SYSTEMTYPE” source=“SYSTEMTYPE”/>
<flowable:in target=“SYSTEMROLE” source=“SYSTEMROLE”/>
<flowable:in target=“INSTALLTYPE” source=“INSTALLTYPE”/>
<flowable:in target=“CLI_TYPE” source=“CLI_TYPE”/>
<flowable:in target=“SERVICE_LEVEL” source=“SERVICE_LEVEL”/>
<flowable:in target=“NW_RELEASE” source=“NW_RELEASE”/>
<flowable:in target=“NW_SPLEVEL” source=“NW_SPLEVEL”/>
<flowable:in target=“STATUS” source=“STATUS”/>
<flowable:in target=“hostname” source=“hostname”/>
<flowable:in target=“instanceNumber” source=“instanceNumber”/>
<flowable:in target=“targetClient” source=“targetClient”/>
<flowable:in target=“initiator” source=“initiator”/>
<flowable:in target=“EC2_CLIENT” source=“EC2_CLIENT”/>
-
-
-
-<bpmndi:BPMNDiagram id=“BPMNDiagram_P_PREPARE_USER_CREATION”>
-<bpmndi:BPMNPlane id=“BPMNPlane_P_PREPARE_USER_CREATION” bpmnElement=“P_PREPARE_USER_CREATION”>
-<bpmndi:BPMNShape id=“BPMNShape_sid-9EC8360C-7D3F-41C5-BD66-CD9E4D79A118” bpmnElement=“sid-9EC8360C-7D3F-41C5-BD66-CD9E4D79A118”>
<omgdc:Bounds y=“243.98147772129244” x=“269.0” width=“40.0” height=“40.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-CAE9DC18-EBCA-49D4-9CFC-42C7D1C230AC” bpmnElement=“sid-CAE9DC18-EBCA-49D4-9CFC-42C7D1C230AC”>
<omgdc:Bounds y=“248.00000106204627” x=“1065.0” width=“28.0” height=“28.00000000000003”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-7F2DEE84-8542-49B7-8C5F-8353B1B45057” bpmnElement=“sid-7F2DEE84-8542-49B7-8C5F-8353B1B45057”>
<omgdc:Bounds y=“45.0” x=“480.0” width=“100.0” height=“80.00000000000003”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-F46068F7-B223-4A27-9B29-C6C87629976C” bpmnElement=“sid-F46068F7-B223-4A27-9B29-C6C87629976C”>
<omgdc:Bounds y=“242.0000010620463” x=“960.0” width=“40.0” height=“40.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-F7B4918D-D5F8-47C0-A0F5-B966C8276338” bpmnElement=“sid-F7B4918D-D5F8-47C0-A0F5-B966C8276338”>
<omgdc:Bounds y=“242.99999618530273” x=“390.265625” width=“40.0” height=“40.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-FF40D659-4CCB-4087-9265-EB7AA6D4955E” bpmnElement=“sid-FF40D659-4CCB-4087-9265-EB7AA6D4955E”>
<omgdc:Bounds y=“242.00000106204627” x=“611.44319650383” width=“40.0” height=“40.00000000000003”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-5ECAA56F-F26F-45F8-8932-021BEEC74F50” bpmnElement=“sid-5ECAA56F-F26F-45F8-8932-021BEEC74F50”>
<omgdc:Bounds y=“135.0” x=“480.0” width=“100.0” height=“80.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-DC0AFFFA-C577-438E-940E-36CEB86F99EC” bpmnElement=“sid-DC0AFFFA-C577-438E-940E-36CEB86F99EC”>
<omgdc:Bounds y=“223.98147772129244” x=“484.0” width=“100.0” height=“80.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-0D132E41-5577-4531-8180-1FED33BF5B64” bpmnElement=“sid-0D132E41-5577-4531-8180-1FED33BF5B64”>
<omgdc:Bounds y=“450.0” x=“484.0” width=“100.0” height=“80.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-85356B6E-16D0-4C28-B233-5F9E3AA9BED4” bpmnElement=“sid-85356B6E-16D0-4C28-B233-5F9E3AA9BED4”>
<omgdc:Bounds y=“555.0” x=“480.0” width=“100.0” height=“80.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-B0F14ED3-A5B9-4E46-8972-1047EFA5C06E” bpmnElement=“sid-B0F14ED3-A5B9-4E46-8972-1047EFA5C06E”>
<omgdc:Bounds y=“247.0000010620463” x=“150.0” width=“30.0” height=“30.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-333C11B3-ADB4-4B2D-A401-EF5FB40D358A” bpmnElement=“sid-333C11B3-ADB4-4B2D-A401-EF5FB40D358A”>
<omgdc:Bounds y=“330.0” x=“480.0” width=“100.0” height=“80.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_sid-F5D5DCDA-43A8-4313-8E49-2C06427BB57F” bpmnElement=“sid-F5D5DCDA-43A8-4313-8E49-2C06427BB57F”>
<omgdc:Bounds y=“242.0000010620463” x=“769.4166801604954” width=“40.0” height=“40.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNShape id=“BPMNShape_C_FILL_TABLES_USOBX_C_AND_USOBT_C_FOR_PROFILE_GENERATOR” bpmnElement=“C_FILL_TABLES_USOBX_C_AND_USOBT_C_FOR_PROFILE_GENERATOR”>
<omgdc:Bounds y=“345.0” x=“739.4166801604954” width=“100.0” height=“80.0”/>
</bpmndi:BPMNShape>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-9BD16AA9-2D4E-4FC8-AC84-87A64306313E” bpmnElement=“sid-9BD16AA9-2D4E-4FC8-AC84-87A64306313E”>
<omgdi:waypoint y=“243.01719127980533” x=“410.2428279806606”/>
<omgdi:waypoint y=“85.00000000000001” x=“410.0625”/>
<omgdi:waypoint y=“85.00000000000001” x=“480.0”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-9DC99A03-7799-4E02-AE67-1BE111657030” bpmnElement=“sid-9DC99A03-7799-4E02-AE67-1BE111657030”>
<omgdi:waypoint y=“242.99999618530273” x=“410.265625”/>
<omgdi:waypoint y=“175.0” x=“410.265625”/>
<omgdi:waypoint y=“175.0” x=“480.0”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-3952B752-B7FC-41C3-9845-AA09DB30467B” bpmnElement=“sid-3952B752-B7FC-41C3-9845-AA09DB30467B”>
<omgdi:waypoint y=“263.82090434433474” x=“308.7898280564846”/>
<omgdi:waypoint y=“263.160168128818” x=“390.42619837695315”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-E22E0E4B-2372-49C8-A2ED-6330D60F9AD1” bpmnElement=“sid-E22E0E4B-2372-49C8-A2ED-6330D60F9AD1”>
<omgdi:waypoint y=“282.90178660790343” x=“410.30947115311096”/>
<omgdi:waypoint y=“490.0” x=“410.765625”/>
<omgdi:waypoint y=“490.0” x=“483.99999999997345”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-26C03242-1497-4C07-AB76-4781C5C62C93” bpmnElement=“sid-26C03242-1497-4C07-AB76-4781C5C62C93”>
<omgdi:waypoint y=“282.917301012478” x=“410.29535674366616”/>
<omgdi:waypoint y=“597.9999961853027” x=“410.765625”/>
<omgdi:waypoint y=“596.256766846439” x=“479.9999999999978”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-0F6250E0-0B9E-45FE-822A-550D69398A16” bpmnElement=“sid-0F6250E0-0B9E-45FE-822A-550D69398A16”>
<omgdi:waypoint y=“262.0000010620463” x=“179.94999484297517”/>
<omgdi:waypoint y=“262.0000010620463” x=“224.5”/>
<omgdi:waypoint y=“263.3853788502032” x=“269.5632738848647”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-F6746133-8086-4CB7-92BA-1E35BCD63879” bpmnElement=“sid-F6746133-8086-4CB7-92BA-1E35BCD63879”>
<omgdi:waypoint y=“262.0000010620463” x=“999.9399344096871”/>
<omgdi:waypoint y=“262.0000010620463” x=“1065.0”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-85F3CCEF-538A-4723-985B-63DDB963E32B” bpmnElement=“sid-85F3CCEF-538A-4723-985B-63DDB963E32B”>
<omgdi:waypoint y=“282.84802128185476” x=“410.3584156943817”/>
<omgdi:waypoint y=“370.0” x=“410.765625”/>
<omgdi:waypoint y=“370.0” x=“480.0”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-E62BE427-49C5-468D-AE68-346AB24D5832” bpmnElement=“sid-E62BE427-49C5-468D-AE68-346AB24D5832”>
<omgdi:waypoint y=“385.0” x=“839.3666801604635”/>
<omgdi:waypoint y=“385.0” x=“980.0”/>
<omgdi:waypoint y=“281.90807673339356” x=“980.0”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-EEE16E87-0FC3-4886-AF0F-EF2244810F9D” bpmnElement=“sid-EEE16E87-0FC3-4886-AF0F-EF2244810F9D”>
<omgdi:waypoint y=“263.1569975256844” x=“430.05075767293033”/>
<omgdi:waypoint y=“263.5848694481308” x=“483.99999999999795”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-B779AD0A-0C4C-4381-8D88-7251377F60CB” bpmnElement=“sid-B779AD0A-0C4C-4381-8D88-7251377F60CB”>
<omgdi:waypoint y=“595.0” x=“579.95”/>
<omgdi:waypoint y=“595.0” x=“631.44319650383”/>
<omgdi:waypoint y=“281.9029824378444” x=“631.44319650383”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-8D3260D1-C50C-4B79-B789-33E34D3823D6” bpmnElement=“sid-8D3260D1-C50C-4B79-B789-33E34D3823D6”>
<omgdi:waypoint y=“370.0” x=“579.9499999999744”/>
<omgdi:waypoint y=“370.0” x=“631.44319650383”/>
<omgdi:waypoint y=“281.9091993939255” x=“631.44319650383”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-EB457BE6-578D-4043-8EAD-AC529C0B3180” bpmnElement=“sid-EB457BE6-578D-4043-8EAD-AC529C0B3180”>
<omgdi:waypoint y=“85.00000000000001” x=“579.9499999999884”/>
<omgdi:waypoint y=“85.00000000000001” x=“630.8636428905911”/>
<omgdi:waypoint y=“242.02121931588098” x=“631.3779239431123”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-44827565-7816-4840-BE55-534F9A4C9112” bpmnElement=“sid-44827565-7816-4840-BE55-534F9A4C9112”>
<omgdi:waypoint y=“262.0000010620463” x=“809.3614489740289”/>
<omgdi:waypoint y=“262.0000010620463” x=“960.0”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-674E4B8D-BA28-4CF0-8921-5BC52747D708” bpmnElement=“sid-674E4B8D-BA28-4CF0-8921-5BC52747D708”>
<omgdi:waypoint y=“175.0” x=“579.95”/>
<omgdi:waypoint y=“175.0” x=“631.44319650383”/>
<omgdi:waypoint y=“242.00000106204627” x=“631.44319650383”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-8CE74EEC-D496-40B1-AEC8-3303D033A2EA” bpmnElement=“sid-8CE74EEC-D496-40B1-AEC8-3303D033A2EA”>
<omgdi:waypoint y=“262.0000010620463” x=“651.3868861474477”/>
<omgdi:waypoint y=“262.0000010620463” x=“769.4166801604954”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-D7BA7CB5-67EF-4665-8009-520E4405CA62” bpmnElement=“sid-D7BA7CB5-67EF-4665-8009-520E4405CA62”>
<omgdi:waypoint y=“262.9647434954359” x=“583.9499999999991”/>
<omgdi:waypoint y=“262.3975931093153” x=“611.8417850223955”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-E4CED190-AC09-46BE-94FE-25A2211D2C06” bpmnElement=“sid-E4CED190-AC09-46BE-94FE-25A2211D2C06”>
<omgdi:waypoint y=“490.0” x=“583.95”/>
<omgdi:waypoint y=“490.0” x=“631.44319650383”/>
<omgdi:waypoint y=“281.9043560423212” x=“631.44319650383”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-4FB9565F-637C-43CE-99F4-E9F44E0F5E95” bpmnElement=“sid-4FB9565F-637C-43CE-99F4-E9F44E0F5E95”>
<omgdi:waypoint y=“243.98147772129244” x=“289.0”/>
<omgdi:waypoint y=“42.000003814697266” x=“289.0”/>
<omgdi:waypoint y=“42.000003814697266” x=“980.0”/>
<omgdi:waypoint y=“242.0000010620463” x=“980.0”/>
</bpmndi:BPMNEdge>
-<bpmndi:BPMNEdge id=“BPMNEdge_sid-4179537F-6496-4898-A56B-8BA899BF0FEC” bpmnElement=“sid-4179537F-6496-4898-A56B-8BA899BF0FEC”>
<omgdi:waypoint y=“281.94189789382034” x=“789.4166801604954”/>
<omgdi:waypoint y=“345.0” x=“789.4166801604954”/>
</bpmndi:BPMNEdge>
</bpmndi:BPMNPlane>
</bpmndi:BPMNDiagram>
Hi Team,
Please find the below version details.
Details for PRD
openjdk-13.0.1
apache-tomcat-9.0.27
mysql-8.0.19
Details for TST:
jdk-15.0.1
apache-tomcat-9.0.39
MySQL 8.0.16
With regards,
Keerthi M
engine PRD is on Flowable 6.4, TST on 6.6
On all engines we have:
Async Executer Settings:
flowable.process.async.executor.default-async-job-acquire-wait-time-in-millis=5000
flowable.process.async.executor.default-timer-job-acquire-wait-time-in-millis=5000
flowable.cmmn.async.executor.default-async-job-acquire-wait-time-in-millis=5000
flowable.cmmn.async.executor.default-timer-job-acquire-wait-time-in-millis=5000
flowable.process.async.executor.async-job-lock-time-in-millis=86400000
Number of process instances: 1567
Number of Jobs:
Out expert thinks number of open files can be increased.
Tomcat Connector settings:
top output:
Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 65976772 total, 51589728 used, 14387044 free, 283620 buffers
KiB Swap: 5242876 total, 189276 used, 5053600 free. 37566424 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2916 mysql 20 0 5072608 1.255g 12868 S 0.997 1.994 3672:42 mysqld
10539 root 20 0 71.295g 0.010t 42700 S 0.332 16.51 57:13.26 java
TCPIP Status (netstat -n | awk ‘/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}’):
CLOSE_WAIT 65
ESTABLISHED 83
TIME_WAIT 6
Java:
That’s a lot of info 
What are your settings for the number of threads for the async executor? With lots of jobs, this is the most influential settings.
The
“update ACT_RU_EXECUTION set LOCK_TIME_ = ? where ID_ = ? and (LOCK_TIME_ is null OR LOCK_TIME_ < ?)” exception means that multiple threads are competing for the same lock. That’s not necessarily bad, but the fact the database is timing out is not expected.
Additionally, we’ve recently made lots of changes in the async executor:
Are all your call activities automatic steps and async? If so, the last link will be very useful.
Hi Joram,
Just wanted to let you know that in our production environment we are unable to terminate the running process instances now. Its stuck and no action is being performed.
And i see the error a below in catalina.
2021-06-14 07:54:06.798 ERROR 24154 — [io2-443-exec-61] .f.u.a.r.c.ProcessInstanceClientResource : Error executing action on process instance 7e2ead84-ccbf-11eb-a8dd-42010aeec52a
org.flowable.ui.admin.service.engine.exception.FlowableServiceException:
at org.flowable.ui.admin.service.engine.FlowableClientService.executeRequestNoResponseBody(FlowableClientService.java:477) ~[flowable-ui-admin-logic-6.4.2.jar:6.4.2]
at org.flowable.ui.admin.service.engine.ProcessInstanceService.executeAction(ProcessInstanceService.java:193) ~[flowable-ui-admin-logic-6.4.2.jar:6.4.2]
at org.flowable.ui.admin.rest.client.ProcessInstanceClientResource.executeAction(ProcessInstanceClientResource.java:144) [flowable-ui-admin-rest-6.4.2.jar:6.4.2]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
May be this is helpful for you?
Regards,
Preeti Shirur
Might be related to
It’s really hard to say anything about this, without knowing more about your setup and amount of data. The exception states that two (or more transactions) are trying to update the same execution, from the looks of it two (or more) subprocesses are trying to continue the same parent process instance.
This should not be a problem, as the database normally picks one transaction and rolls the other back. However, it seems that your db isn’t doing that. What version of MySQL are you using?
Also, what do you mean with termination? What API are you using to do that? The delete API does everything synchronously, so the exception you state there can only happen if another, in-flight, transaction is executing.
Hi Joram,
We are using mysql-8.0.19 version.
Hi Joram,
I tried to terminate the process instance from flowable admin, the url for the same is as below.
https://dlmoattst.wdf.sap.corp/flowable-ui/admin/#/process-definitions
Terminating a process instance from flowable admin was stuck and was throwing below error.
Regards,
Preeti Shirur