Event registry kafka late events processing

Flowable Engine
#1

Hi,

Regarding event channels, I have an out of order kafka topic which is partitioned as well and we are planning to use it in a inbound channel, reading the flowable documentation and code I didn`t found any feature to handle late events

We wanted to send status updates through intermediate catching events and is required to receive only the newest events and discard the outdated events for the identifier, below a very basic sample diagram with the idea:
image

IE. lets imagine the following data inside the unordered kafka topic
id: 1 , timestamp: 1 , message: “item 1 created”
id: 2 , timestamp: 1 , message: “item 2 created”
id: 1 , timestamp: 3 , message: “item 1 updated second time”
id: 1 , timestamp: 2 , message: “item 1 updated first time” (should be ignored)

There is any plans to add a feature to filter out late events from the topic?

I think that we need to implement a custom logic either a custom InboundEventProcessingPipeline which keeps a state of a window of the data read from the topic and discards late events,

other possibility that I though is to add a condition on intermediary catching event to process only if the incoming event has the timestamp greater than the previous one, is possible to ignore the incoming event via execution listener or other mechanism? do you have any other ideas to solve this issue?

Thank you in advance.

#2

Yes, that would indeed the way to go.
A custom pipeline (reusing most of the default one). that applies this filtering for you.

The tricky thing is what to do when a server goes down, when the events are still in this ‘window’. These could potentially be lost.

That would be possible (solving it in the model), but will get messy quickly.

Is it imperative this is solved on the receiving side? Isn’t there a way this could be solved before the events are passed to Flowable?