Best practice for filtering Kafka messages before process instance creation

Hi everyone:

Flowable version: 7.2

We have several BPMN processes triggered by a shared Kafka topic. Each process uses a dedicated channel with its own consumer group, but only a small subset of messages is relevant to each process — determined by a combination of Kafka headers and payload fields. Previously we handled this with an exclusive gateway after the start event, but this creates a process instance
for every message only to discard most of them — generating a large number of short-lived history records at high volumes.

We want to move the check upstream so Flowable never creates an instance for non-matching messages. We’ve identified two interception points:

Option 1 — InboundEventKeyDetector

Inspect headers and payload and return the event definition key only when conditions are met, otherwise return “”, which will be eventually discarded by an existing InboundEventFilter.

Option 2 — InboundEventFilter via eventFilterDelegateExpression

Inspect headers and payload in an InboundEventFilter and return false for non-matching messages, keeping fixedValue for key detection.


Is there a recommended pattern for this use case? Is one of these preferable, or is there a third approach we haven’t considered?

Thanks!
Pedro.

PS: We are about to upgrade to Flowable 8, in case there is another option there.

Hey @nadaesposible,

The second option “InboundEventFilter via eventFilterDelegateExpression” would be the approach we would recommend to you. This was added for exactly the use case you are asking about.

Cheers,
Filip

Good morning, Filip,

Thank you so much for the quick response and the clear answer. I really appreciate it.

Cheers,
Pedro.