Regarding event channels, I have an out of order kafka topic which is partitioned as well and we are planning to use it in a inbound channel, reading the flowable documentation and code I didn`t found any feature to handle late events
We wanted to send status updates through intermediate catching events and is required to receive only the newest events and discard the outdated events for the identifier, below a very basic sample diagram with the idea:
IE. lets imagine the following data inside the unordered kafka topic
id: 1 , timestamp: 1 , message: “item 1 created”
id: 2 , timestamp: 1 , message: “item 2 created”
id: 1 , timestamp: 3 , message: “item 1 updated second time”
id: 1 , timestamp: 2 , message: “item 1 updated first time” (should be ignored)
There is any plans to add a feature to filter out late events from the topic?
I think that we need to implement a custom logic either a custom InboundEventProcessingPipeline which keeps a state of a window of the data read from the topic and discards late events,
other possibility that I though is to add a condition on intermediary catching event to process only if the incoming event has the timestamp greater than the previous one, is possible to ignore the incoming event via execution listener or other mechanism? do you have any other ideas to solve this issue?
Thank you in advance.