In our application we have dynamically generated process with a lot of subprocesses. Before we had flow with sync http tasks, but faced problems with FlowableOptimisticLocking Exception and according to business logic we could not retry tasks. So, after refactoring now we have process with parallel subprocesses with async receive tasks which have listeners that execute http requests. In small processes with 7-9 subprocesses everything works fine. But when we have 50+ subprocesses with 6-9 threads in parallel after 15-20 of completed subprocesses tasks begin to have longer timeouts between each other, if in the beginning we have 1-2 seconds, at the end it increases up to 7-10 minutes and even another processes stuck. But when another process B starts - that stuck process begins to run normally for some time and then starts to hang on again, but that process B runs normally till the end if it’s not that large.
We are using Spring Boot and app runs in container. All resources measures are fine and not even close to it’s limits.
Can you please suggest anything: possible root causes of this behaviour or where to look to find it.
Thank you in advance!