Error handling with event task and RabbitMQ

Is there any ‘best practices’ to handling errors which occure working with event task?
I have send and receive event task - checkbox “Triggerable” - where i defined two Rabbit queues. Connection is ok, everything works fine but when i for example get wrong message (based on which I walk through exclusive gateway)

Adnotacja 2020-09-11 135604

I send event to rabbit, get response in which i should get json with field. Then my flow goes based on that field. For example, when i dont get that field, my process throw FlowableException and back to send event task and consumes message again. That takes place in loop and my app crash. What should I do?

Is there any way to handle any kind of exception and for example execute this task 3 more times? I found deadletters but does it work with this kind of task? I used it only with async task.
Is this part of documentation https://flowable.com/open-source/docs/bpmn/ch18-Advanced/#message-queue-based-async-executor right solution?

Yes, it is a service task so regular async applies. Making it async will indeed make it retry 3 times (by default, but configurable) before moving it to a deadletter job.

I have send event task with triggerable. I get Flowable Exception and application loops. As I presented on image above. How can I deal with it?

I have one more question. Send event task with triggerable and async. I send message, then I receive one. I want to check what I received. How can I do it? I put execution listener while task end (EndListener).

@Service
public class EndListener implements ExecutionListener {


    @Override
    public void notify(DelegateExecution execution) {
        if (execution.getVariable("foo", String.class) == null)
            throw new BpmnError("code", "message");
    }
}

After that I receive ListenerExecutionFailedException which I can’t handle with boundary error event.

Logs

2020-09-18 13:57:08.107 WARN 16500 — [nboundChannel-1] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1705) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1651) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1521) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1498) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1489) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1433) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:970) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:916) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1600(SimpleMessageListenerContainer.java:83) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1291) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1197) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: org.flowable.engine.delegate.BpmnError:
at com.example.demo.EndListener.notify(EndListener.java:15) ~[main/:na]
at org.flowable.engine.impl.delegate.invocation.ExecutionListenerInvocation.invoke(ExecutionListenerInvocation.java:35) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.delegate.invocation.DelegateInvocation.proceed(DelegateInvocation.java:35) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.delegate.invocation.DefaultDelegateInterceptor.handleInvocation(DefaultDelegateInterceptor.java:26) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.bpmn.listener.DelegateExpressionExecutionListener.notify(DelegateExpressionExecutionListener.java:45) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.bpmn.listener.ListenerNotificationHelper.executeExecutionListeners(ListenerNotificationHelper.java:79) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.agenda.AbstractOperation.executeExecutionListeners(AbstractOperation.java:78) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.agenda.AbstractOperation.executeExecutionListeners(AbstractOperation.java:69) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.agenda.TakeOutgoingSequenceFlowsOperation.handleActivityEnd(TakeOutgoingSequenceFlowsOperation.java:109) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.agenda.TakeOutgoingSequenceFlowsOperation.handleFlowNode(TakeOutgoingSequenceFlowsOperation.java:95) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.agenda.TakeOutgoingSequenceFlowsOperation.run(TakeOutgoingSequenceFlowsOperation.java:88) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.interceptor.CommandInvoker.executeOperation(CommandInvoker.java:88) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.interceptor.CommandInvoker.executeOperations(CommandInvoker.java:72) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:56) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.interceptor.BpmnOverrideContextInterceptor.execute(BpmnOverrideContextInterceptor.java:25) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.common.engine.impl.interceptor.TransactionContextInterceptor.execute(TransactionContextInterceptor.java:53) ~[flowable-engine-common-6.5.0.jar:6.5.0]
at org.flowable.common.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:72) ~[flowable-engine-common-6.5.0.jar:6.5.0]
at org.flowable.common.spring.SpringTransactionInterceptor.lambda$execute$0(SpringTransactionInterceptor.java:56) ~[flowable-spring-common-6.5.0.jar:6.5.0]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.flowable.common.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:56) ~[flowable-spring-common-6.5.0.jar:6.5.0]
at org.flowable.common.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30) ~[flowable-engine-common-6.5.0.jar:6.5.0]
at org.flowable.common.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:56) ~[flowable-engine-common-6.5.0.jar:6.5.0]
at org.flowable.common.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:51) ~[flowable-engine-common-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.RuntimeServiceImpl.trigger(RuntimeServiceImpl.java:443) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.eventregistry.BpmnEventRegistryEventConsumer.handleEventSubscription(BpmnEventRegistryEventConsumer.java:84) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.engine.impl.eventregistry.BpmnEventRegistryEventConsumer.eventReceived(BpmnEventRegistryEventConsumer.java:71) ~[flowable-engine-6.5.0.jar:6.5.0]
at org.flowable.eventregistry.impl.consumer.BaseEventRegistryEventConsumer.eventReceived(BaseEventRegistryEventConsumer.java:57) ~[flowable-event-registry-6.5.0.jar:6.5.0]
at org.flowable.eventregistry.impl.DefaultEventRegistry.sendEventToConsumers(DefaultEventRegistry.java:63) ~[flowable-event-registry-6.5.0.jar:6.5.0]
at org.flowable.eventregistry.impl.DefaultInboundEventProcessor.eventReceived(DefaultInboundEventProcessor.java:42) ~[flowable-event-registry-6.5.0.jar:6.5.0]
at org.flowable.eventregistry.impl.DefaultEventRegistry.eventReceived(DefaultEventRegistry.java:56) ~[flowable-event-registry-6.5.0.jar:6.5.0]
at org.flowable.eventregistry.spring.rabbit.RabbitChannelMessageListenerAdapter.onMessage(RabbitChannelMessageListenerAdapter.java:61) ~[flowable-event-registry-spring-6.5.0.jar:6.5.0]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1647) ~[spring-rabbit-2.2.10.RELEASE.jar:2.2.10.RELEASE]
… 10 common frames omitted

How can I handle that?
@filiphr, @joram can you help, please? If more detail needed, let me know.

What about a service task after the send event that has the check for the error? Currently, BPMN error codes are not propagated from the trigger() yet.

OK but how can I deal with that error? How can I repeat that send event task? I do not want to make BPMN loop, which will indicate this task after error appear. It will produce infinitive error loop. Can I, in some way, receive something like deadletter which I can resume e.g. by REST API only when I want to?