When pods get old, flowable is unable to fetch the definition, returning it as null. Once the service is restarted it starts working and returns the definition for IDs which it failed to get before restarting.
Snippet: repositoryService
.createProcessDefinitionQuery()
.processDefinitionId(workflowDefinitionId).singleResult();
Hey @pchandle,
That’s a bit strange. Flowable stores the data in the database, so I don’t see how the pod getting old would fail to retrieve the process definition.
Can you please provide some more information.
Cheers,
Filip
This piece of code is used to create a BPMN image of the definition… But before creating the image it tries to get the definition, it sends it as null. Although the definition is present in act_re_procdef table with valid information in act_re_deployment table as well. This has started recently(1-2months), while flowable was upgraded to 6.8.1 a long time ago. There were no changes made in code.
Any idea what could have gone wrong and how to proceed?
Could you create a jUnit test?
JUnits seems to pass, as at startup it works fine… when service is deployed it can fetch the definition and create BPMN image as expected… it is after few hours, the issue arises. It can’t fetch definition hence unable to create the BPMN image
Do you have an exception or is the result of the singleResult() method null?
Also, what is the workflowDefinitionId at the time when you invoke at the time when it fails?
Valentin
Some logs comparison:
Before service redeploy(old pods running for 5-6+ hours): ProcessDefinition lookup for id=‘GB_Flow_21:3:f8576ffe-d8c6-4f2e-83f3-2c568e484955’ returned null
After service redeploy: ProcessDefinition lookup for id=‘GB_Flow_21:3:f8576ffe-d8c6-4f2e-83f3-2c568e484955’ returned ProcessDefinitionEntity[GB_Flow_21:3:f8576ffe-d8c6-4f2e-83f3-2c568e484955]
These are logs for same workflowDefinitionId.
Generic Exception thrown if this helps:
com.sas.mkt.workflow.client.exception.WorkflowException: The system failed to find the workflow process definition.
at com.sas.mkt.workflow.engine.services.impl.RepositoryServiceWrapperImpl.getEngineProcessDefinition(RepositoryServiceWrapperImpl.java:73)
at com.sas.mkt.workflow.engine.services.impl.RepositoryServiceWrapperImpl.getWorkflowDefinitionDiagram(RepositoryServiceWrapperImpl.java:131)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:723)
at com.sas.mkt.workflow.engine.services.impl.RepositoryServiceWrapperImpl$$SpringCGLIB$$0.getWorkflowDefinitionDiagram()
at com.sas.mkt.workflow.core.services.impl.WorkflowDefinitionServiceImpl.getWorkflowDefinitionImage(WorkflowDefinitionServiceImpl.java:882)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:723)
at com.sas.mkt.workflow.core.services.impl.WorkflowDefinitionServiceImpl$$SpringCGLIB$$0.getWorkflowDefinitionImage()
at com.sas.mkt.workflow.rest.controllers.WorkflowDefinitionController.getWorkflowDefinitionDiagram(WorkflowDefinitionController.java:170)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:257)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:190)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1088)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:978)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
… 72 frames truncated
There is no Flowable in this exception, so it’s hard to tell something useful out of it. Can you share the method RepositoryServiceWrapperImpl.getEngineProcessDefinition?
No flowable does not throw any exception…I did try to catch any exception form flowable earlier… here’s the method:
@OverrideOverride
public ProcessDefinition getEngineProcessDefinition(
String workflowDefinitionId) throws WorkflowException {
ProcessDefinition processDefinition = null;
processDefinition = this.repositoryService
.createProcessDefinitionQuery()
.processDefinitionId(workflowDefinitionId).singleResult();
logger.info(“GetBPMNImage:: ProcessDefinition lookup for id=‘{}’ returned {}”, workflowDefinitionId, processDefinition);
if (processDefinition == null) {
logger.error(“GetBPMNImage:: Could not find process definition for id='{}”, workflowDefinitionId);
throw new WorkflowException(
WorkflowMessageCode.WORKFLOW_DEFINITION_NOT_FOUND);
}
return processDefinition;
}
Hey @pchandle,
it’s correct that flowable does not throw any exception for that query..
For me, there are only 2 possibilities:
- Whatever value workflowDefinitionId has, it’s not a process definition Id.
- At some time between you storing that definitionId and executing the query, the definition got undeployed.
Can you check where the actual value of workflowDefinitionId comes from or check if the DeleteDeploymentCmd got executed?
Greetings
Christopher
Edit: While restarting the pods, do you use the auto-deploy for your process definitions?
Are the pods only for flowable, and not restarting the database?
Hey @WelschChristopher
This is a log when system is able to find procdef: ProcessDefinition lookup for id=‘GB_Flow_21:3:f8576ffe-d8c6-4f2e-83f3-2c568e484955’ returned ProcessDefinitionEntity[GB_Flow_21:3:f8576ffe-d8c6-4f2e-83f3-2c568e484955]
There is no specific code written to undeploy the definition. Also there is no difference in details of act_re_procdef and act_re_deployment table when system works fine and when it dosent. All the details like id_, deployment_id_ seems to vaild and related to each other.
Complete service is restarted ie. workflow service that includes flowable as well. yes, autodeploy does run at service startup.
Hey @pchandle ,
Flowable runs in a multi instance env?
Can you please log the query and it’s parameters when this happens?
To say it bluntly: If there is something fundamental wrong with querying definitions, that would pop up. Queries are not cached and fetch directly from the database. That leaves us only with your setup/configuration or custom code (In my opinion)
Greetings
Christopher
Hi @WelschChristopher , I started flowable internal logs
Failed:
Worked:
it looks like after service restart it is still querying the same Id, and is able to get it… While before service restart it couldn’t. Does flowable maintain internal cache and tries to fetch it from there?
There is not deleteDeploymentCmd in the logs, so it dosent look like it got executed.
Any idea how this could happen and how to fix it?