Channel definition - Configurable Kafka topic

Flowable Engine
1

Hi,
I am using a channel definition to configure an inbound topic:
Is there a way to have a variable/expression for the topic name so that it is configurable per environment? Can it be a property, a bean method etc…?
I am using Flowable 6.6.0 and Spring Boot 2.3
Thanks!

{
    "key": "new_doc_channel",
    "name": "New Document channel",
    "description": "New Document channel",
    "channelType": "inbound",
    "type": "kafka",
    "deserializerType": "json",
    "channelEventKeyDetection": {
        "fixedValue": "new_doc_event"
    },
    "topics": ["hardcoded_topic"]
}
2

This is a great question and something we ran into when deploying our application into different environments where the topic names varied per environment. After much digging through the source code, we found we could control the topic names by subclassing the KafkaChannelDefinitionProcessor class (in the org.flowable.eventregistry.spring.kafka package) and introducing our implementation as a Spring bean named kafkaChannelDefinitionProcessor. Overriding the method:
protected Collection<String> resolveTopics(KafkaInboundChannelModel channelDefinition)
allows the topic names in the channel definition to be replaced with other values at runtime.

3

Hey @mykeul and @jwashington,

Most of the values in the Channel definitions can use SPeL (Spring Expression Language) to resolve the values. There is no need to override the KafkaChannelDefinitionProcessor.

If you look at

You’ll see that it invokes resolveExpression for every single value in the topics.

resolveExpression uses the Spring value resolver.

So finally if you want to use a property in your topic you can define it like ${application.test.kafka-topic}.

You can see some examples in this commit.

Cheers,
Filip

4

Thanks @jwashington and @filiphr for your answers.

I used the property directly in the channel definition file and it works perfectly.

"topics": ["${kafka.topic}"]

Thank you very much @filiphr for your detailed answer.

Michel

5

By the way what’s the pros and cons of using channel/event definition files vs programmatically?

6

The use of SPel, as @filiphr rightfully pointed out, would probably fit most needs of replacing topic names through property settings and/or referencing Spring beans with either no or minimal programming. If for some reason SPel couldn’t meet the requirements, then creating your own KafkaChannelDefinitionProcessor would provide the most amount of control.

1 Like
7

Hi @filiphr
As I said before, your solution with Spel works perfectly:

However, If I change the spring property name, the application cannot start and is still searching for the old (not existing) property name.
Only if I remove the DB, then the application starts.
What would be the solution to change a property name without the need to remove the DB?

Thanks,