Bad performance with async service tasks

Flowable Engine
14

Ok, I looked into your example test and process and came to the following conclusions.

  • First of all, Flowable is configured out of the box to have minimal resources. We don’t like being this tool that gobbles RAM and CPU. This means that a typical Flowable installation needs to be tweaked according to needs.
  • Second, running with an async step is always going to be slower than running it synchronous on a single node (i.e. my laptop in this case). Async starts to pay off when spreading the load across multiple nodes (or when it makes sense to add transaction boundaries).

I created a simple Spring Boot app to verify this, as we’ve seen too many strange things happening when Docker is involved. So basically, created a new Spring Boot app, added your process xml to the ‘processes’ folder under resources and used these dependendies:

 <dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>


    <dependency>
        <groupId>org.flowable</groupId>
        <artifactId>flowable-spring-boot-starter-rest</artifactId>
        <version>6.5.0</version>
    </dependency>
    <dependency>
        <groupId>org.postgresql</groupId>
        <artifactId>postgresql</artifactId>
    </dependency>
    <dependency>
        <groupId>org.codehaus.groovy</groupId>
        <artifactId>groovy-jsr223</artifactId>
        <version>2.5.10</version>
    </dependency>
</dependencies>

Basically, the starter for rest does the same as the flowable-rest app: it adds and exposes the Flowable REST api.

The baseline number I got when executing the sync process with your JMeter script on my laptop was ± 750 process instances/sec.

Now, in this synchronous use case, the threads actually executing the process instances are the Tomcat threads. By default, Spring Boot has a max of 250 threads. This means that the 50 threads from JMeter blasting away can be handled easily, whilst doing the Flowable logic at the same time.

The default async executor of Flowable has 8 threads, which is by no means an equal comparison with the tomcat threads. The JMeter script is blasting away with 50 threads concurrently, so there’s more being fed into the system than is being processed.

Furthermore, when I attached a profiler to the app, it was clear that the default connection pool wasn’t cutting it. This is logical, as instead of 1 connection / tomcat thread the async executor also i taking connections for each async job concurrently.

That all being said, the settings were changed to the following in the application.properties:

spring.datasource.url=jdbc:postgresql:testing
spring.datasource.username=flowable
spring.datasource.password=flowable
spring.datasource.hikari.maximum-pool-size=250

spring.task.execution.pool.core-size=64
spring.task.execution.pool.max-size=64

flowable.history-level=none

logging.file=output.txt

I tried with more threads, but the combined 50 JMeter threads + tomcat threads handling the POST + async executor threads + local postgres db was too much for my machine, hence the setting of 64. On a production installation this would off course be possible.

With these settings, I got ± 400 process instances/second. Which makes sense given the explanation above. Probably could be made faster by swapping the groovy script with a java class. In the benchmarks we’ve published (https://blog.flowable.org/2018/03/05/flowable-6-3-0-performance-benchmark/) we used multiple machines on AWS (with loadbalancers and a proper db).

Most likely, if we zoom in on the particular process here, there can be some settings or tweaks be found that would speed it up even more.

Hope this clarifies some things.

3 Likes