AsyncJobExecutor doesn't work as expected with multitenancy

Flowable Engine
1

Hi team, hi all.
I need to use multitenancy mode “Multi-Engine / Shared Database”.
I need in different thread executors capacity. Because of that I create as many Process Engines as many tenants I have. For example 100 engines.
Each created Engine has own tenantId. Each Async Executor of each Process Engine has tenantId. Each Async Executor has own thread-pool configuration.

I create process definition and run process instance inside concrete Process Engine. All objects created by the Engine is assigned to an engine’s tenant.

But what I have with this approach:

  • Each async job executor creates 3 threads: AcquireAsyncJobsDueRunnable, AcquireTimerJobsRunnable, ResetExpiredJobsRunnable.
  • Each created AcquireAsyncJobsDueRunnable finds next jobs in DB to acquire and execute them.
  • AcquireAsyncJobsDueRunnable calls AcquireJobsCmd that finds the jobs. But it does not use tenantId to find just own jobs, it gets any jobs for any tenants.
  • The same happens with timers and reset jobs in AcquireTimerJobsCmd and FindExpiredJobsCmd.
    But if different executors has different settings then it may be aquired and executed by wrong executor.
    That is right?

I’m looking at ExecutorPerTenantAsyncExecutor.
Looks good at the point of “boolean executeAsyncJob(JobInfo job)” - it determines executor for current tenant. And looks like the job will be executed inside right executor.
But. It creates TenantAware-runnables that inside just sets current tenant. Yes, as I understand it will catch executor by tenant.
But the acquiring operation will find any job. For this job will be determined some executor for current tenant that does not equals job’s tenant.
That is right?
Is this a bug or is it expected? Where I’m wrong?

My target is that: I want to run process and execute jobs with a configured async executors.
Do I need create just one engine with many asyncExecutors (per tenant)?
Do I have to customize all runnables (like AcquireAsyncJobsDueRunnable) that will run custom commands to retrieve jobs belong the asyncExecutor’s tenant?

Thanks.

3

  1. That’s correct. The regular async executor is not multi tenant enabled (to keep the query as fast as possible), so that’s what’s happening here. For this reason there is indeed the ExecutorPerTenantAsyncExecutor or SharedExecutorServiceAsyncExecutor.

  2. It sets a specific runnable for the acquiring: https://github.com/flowable/flowable-engine/blob/master/modules/flowable-job-service/src/main/java/org/flowable/job/service/impl/asyncexecutor/multitenant/ExecutorPerTenantAsyncExecutor.java#L79. This indeed sets the current tenantId, which in conjuction with the MultiSchemaMultiTenantProcessEngineConfiguration will work due to the TenantAwareDatasource: https://github.com/flowable/flowable-engine/blob/master/modules/flowable-engine/src/main/java/org/flowable/engine/impl/cfg/multitenant/MultiSchemaMultiTenantProcessEngineConfiguration.java#L77

  3. No, the TenantAwareDataSource is how it fits together. What you could do to make your use case work, is use one MultiSchemaMultiTenantProcessEngineConfiguration per tenant, but set it to one tenant.

4

Joram, thank you for reply.

As I see TenantAwareDatasource just gets one of the data sources by required tenantId. Hence, all Runnable-threads of AsyncExecutor will just work with own data source. It may have sense for isolated data per tenant (multi schema or multi database). But if data isolation is not needed and only one datasource is used then each Runnable of all AsyncExecutors will acquire/reset jobs and timers not only for own tenant. This is because AcquireJobsCmd is not used tenantId to find jobs.

But what interesting I saw is enabledCategories settings may be used in my situation (this is new in v6.5.1). In this way is possible to configure Process Engine’s JobServiceConfiguration to enable working with concrete categories only. But this requires add jobCategory extension for asynchronous activities in the Process Definition.
Looks like a Silver Bullet for the problem.

But why not just configure tenant for AsyncExecutor and pass the tenant (if it is set) in AcquireJobsCmd (for example) to findJobsToExecute like enabledCategories? With this approach it would be easier to separate AsyncJobExecutors without additional settings in Process Definition for each async elements.

Thanks.
Dmitry F.

5

That’s indeed correct and would fit your use case.

It would complicate the query further (and table scans really need to be avoided there), so it needs to be applied only if needed (like the category). It’s not something that is on our roadmap, but PR’s are of course always reviewed.

6

Sure, this filtering must be applied only when it is specially configured for AsyncExecutor.

Sounds great! Thank you anyway!