Classloading issues running flowable 6.4.0 on Karaf


#1

Hi flowable team,

we use flowable 5 together with apache karaf 4.0.x and it’s working quite well. Now we would like to migrate to flowable 6.4.0, but we have classloading issues when bootstrapping the engine.

All required flowable bundles are installed:
325 │ Active │ 80 │ 6.4.0 │ Flowable - BPMN Converter
326 │ Active │ 80 │ 6.4.0 │ Flowable - BPMN Model
327 │ Active │ 80 │ 6.4.0 │ Flowable - CMMN API
328 │ Active │ 80 │ 6.4.0 │ Flowable - CMMN Model
329 │ Active │ 80 │ 6.4.0 │ Flowable - Engine Common API
330 │ Active │ 80 │ 6.4.0 │ Flowable - Engine Common
331 │ Active │ 80 │ 6.4.0 │ Flowable - Content API
332 │ Active │ 80 │ 6.4.0 │ Flowable - DMN API
333 │ Active │ 80 │ 6.4.0 │ Flowable - DMN Model
334 │ Active │ 80 │ 6.4.0 │ Flowable - Engine
335 │ Active │ 80 │ 6.4.0 │ Flowable - Form API
336 │ Active │ 80 │ 6.4.0 │ Flowable - Form Model
337 │ Active │ 80 │ 6.4.0 │ Flowable - Identity Link Service API
338 │ Active │ 80 │ 6.4.0 │ Flowable - Identity Link Service
339 │ Active │ 80 │ 6.4.0 │ Flowable - IDM API
340 │ Active │ 80 │ 6.4.0 │ Flowable - IDM Engine
341 │ Active │ 80 │ 6.4.0 │ flowable-idm-engine-configurator
342 │ Active │ 80 │ 6.4.0 │ Flowable - Image Generator
343 │ Active │ 80 │ 6.4.0 │ Flowable - Job Service API
344 │ Active │ 80 │ 6.4.0 │ Flowable - Job Service
345 │ Active │ 80 │ 6.4.0 │ Flowable - Task Service API
346 │ Active │ 80 │ 6.4.0 │ Flowable - Task Service
347 │ Active │ 80 │ 6.4.0 │ Flowable - Process Validation
348 │ Active │ 80 │ 6.4.0 │ Flowable - Variable Service API
349 │ Active │ 80 │ 6.4.0 │ Flowable - Variable Service

The engine is configured as follows:

<bean id="processEngineFactory" class="org.flowable.osgi.blueprint.ProcessEngineFactoryWithELResolver" init-method="init" destroy-method="destroy">
		<property name="processEngineConfiguration" ref="processEngineConfiguration" />
		<property name="bundle" ref="blueprintBundle" />
		<property name="blueprintELResolver" ref="blueprintELResolver" />
		<property name="blueprintContextELResolver" ref="blueprintContainerListener" />
</bean>
<bean id="processEngineConfiguration" class="org.flowable.engine.impl.cfg.StandaloneProcessEngineConfiguration" ext:field-injection="true">
		<property name="dataSource" ref="dataSource" />
		<property name="databaseSchemaUpdate" value="true" />
		<property name="disableIdmEngine" value="true" />
		<property name="mailServerHost" value="${smtpHost}" />
		<property name="mailServerPort" value="${smtpPort}" />
		<property name="mailServerDefaultFrom" value="${fromEmail}" />
		<property name="asyncExecutorActivate" value="${jobExecutorActivate}" />
	</bean>

When starting our process-engine with blueprint.xml above, we get following exception:
Caused by: org.flowable.common.engine.api.FlowableException: Error while building ibatis SqlSessionFactory: inStream parameter is null
at org.flowable.common.engine.impl.AbstractEngineConfiguration.initSqlSessionFactory(AbstractEngineConfiguration.java:686)
at org.flowable.engine.impl.cfg.ProcessEngineConfigurationImpl.init(ProcessEngineConfigurationImpl.java:960)
at org.flowable.engine.impl.cfg.ProcessEngineConfigurationImpl.buildProcessEngine(ProcessEngineConfigurationImpl.java:887)
at org.flowable.osgi.blueprint.ProcessEngineFactory.init(ProcessEngineFactory.java:41)
at org.flowable.osgi.blueprint.ProcessEngineFactoryWithELResolver.init(ProcessEngineFactoryWithELResolver.java:42)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:331)
at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:838)
at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591)
… 70 more
Caused by: java.lang.NullPointerException: inStream parameter is null
at java.base/java.util.Objects.requireNonNull(Objects.java:246)
at java.base/java.util.Properties.load(Properties.java:403)
at org.flowable.common.engine.impl.AbstractEngineConfiguration.initSqlSessionFactory(AbstractEngineConfiguration.java:679)
… 81 more

The org.flowable.common.engine.impl.AbstractEngineConfiguration.getResourceAsStream(String) Method does not work in OSGi, from OSGi classloading perspective, this bundle does not see the resources it wants to load (mybatis xml configurations).

Any help would be appreciated.

Regards, Björn


#2

Hi Björn,

We have some basic tests in the flowable-osgi module to test the bootstrapping of the Flowable Engine and starting a simple process instance. Would it be possible to create a unit test in the Flowable OSGi module that shows this issue? Then it would be easier to think of a fix. Or maybe we should create a flowable-karaf module and do this testing from there?

Best regards,

Tijs


#3

the best solution would be to provide a working karaf-example in https://github.com/flowable/flowable-examples. I could provide a sample Maven Module, which demonstrates, how to use Flowable with Karaf (and shows the problems I have with that).

If everything fits for you and it’s working, you provide the module in the flowable-examples repository.

What would be the best way to send the project to you?


#4

I added the project:

Can you have a look?


#5

Could you give us feedack, if you will take care of this issue? Otherwise, we have to re-think our migration plan …


#6

Sorry for the delay in answering. I will give your Karaf demo a try over the next couple of days and come back to you in this forum topic.

Best regards,

Tijs


#7

Hi,

I’ve tested your example project and could also reproduce the issue.
The strange thing is that it does work within the flowable-osgi unit test for some reason, maybe because of an extra restriction in Apache Karaf.
I’ve tried a couple of ways to get the issue resolved with exporting the IDM engine mapping files folder etc, but got the same error with every change. Also couldn’t find a solution yet while searching on loading a resource file from another bundle. A ways to make it work for now is to disable the IDM engine, with setting the disableIdmEngine property to true in the process engine configuration.

Best regards,

Tijs


#8

I also noticed, that resources are loaded from other bundles (e.g. flowable-engine loads resources from flowable-engine-common:org.flowable.common.db.properties), so if using OSGi, those packages should also be added to the OSGi export/import declarations.

But imho, this should not work with the org.flowable.osgi.blueprint.BundleDelegatingClassLoader, which is configured for the ProcessEngineConfiguration. I dont understand, why a special classloader is needed here !?

The disableIdmEngine=true configuration changes the classloading behaviour but does not solve the problem, I still see Nullpointers caused by Resources, which could not be load.


#9

Yes I understand your point. But when I disabled the IDM engine the process engine was being created successfully in my tests. For you the process engine bundle still fails?

Best regards,

Tijs


#10

yes, it fails with

Caused by: java.lang.NullPointerException
	at java.io.Reader.<init>(Reader.java:78)
	at java.io.InputStreamReader.<init>(InputStreamReader.java:72)
	at org.flowable.common.engine.impl.AbstractEngineConfiguration.initSqlSessionFactory(AbstractEngineConfiguration.java:659)

I updated my sample project with disableIdmEngine=true, could you pull my changes and check, why we have different behaviour? (I also added a H2 Datasource, see changes in README.md)

Regards, Björn


Flowable-engine in OSGI (Karaf)
#11

Hello,
I am trying to use flowable-osgi v6.4.0 with Felix v6.0.1. From the bundle I want to configure and use the ProcessEngine from I also have the an input stream NULL:

Caused by: java.lang.IllegalArgumentException: InputStream cannot be null
        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:117)
        at org.flowable.common.engine.impl.AbstractEngineConfigurator.registerCustomMybatisMappings(AbstractEngineConfigurator.java:106)
        at org.flowable.common.engine.impl.AbstractEngineConfigurator.beforeInit(AbstractEngineConfigurator.java:50)
        at org.flowable.common.engine.impl.AbstractEngineConfiguration.configuratorsBeforeInit(AbstractEngineConfiguration.java:852)
        at org.flowable.engine.impl.cfg.ProcessEngineConfigurationImpl.init(ProcessEngineConfigurationImpl.java:912)
        at org.flowable.engine.impl.cfg.ProcessEngineConfigurationImpl.buildProcessEngine(ProcessEngineConfigurationImpl.java:887)

From debugging, I found out that it is related to the IdmEngineConfigurator.getMybatisCfgPath() (i.e. ‘org/flowable/idm/db/mapping/mappings.xml’) which does not exist within that specific bundle, nor is the AbstractEngineConfiguration.getClassLoader() provided (i.e. NULL).
Thus I assume that it is in the same kind of ideas in regard to Classloading issues for Flowable 6.4.0 with OSGi.


#12

yes, when disableIdmEngine=false is set, I have the same behaviour. Looking forward to see this issue fixed :grinning:


#13

I have tried the process-engine module from the demo you posted in my case. The bundle start well but after some time I get the following error:

g! 13:57:52.283 [Blueprint Extender: 3] ERROR org.apache.aries.blueprint.container.BlueprintContainerImpl - Unable to start container for blueprint bundle com.ge.flowbase.flowbase-flowable-process-engine/1.0.0.SNAPSHOT due to unresolved dependencies [(objectClass=javax.sql.DataSource)]
java.util.concurrent.TimeoutException: null
        at org.apache.aries.blueprint.container.BlueprintContainerImpl$1.run(BlueprintContainerImpl.java:373) [org.apache.aries.blueprint.core-1.10.1.jar:1.10.1]
        at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:45) [org.apache.aries.blueprint.core-1.10.1.jar:1.10.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

Maybe another symptom bound to the class loading issue


#14

Hi all,

I am very interesting by this.
I would like to do that, so I took this sample and try it.
Unfortunatly, it’s does not work with the final command (i.e. diag process-engine)
Unable to initialize bean processEngineFactory
org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to initialize bean processEngineFactory
at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:593)
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:703)
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:666)
at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:90)
at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:360)
at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:190)
at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:716)
at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:413)
at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:278)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106)
at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:45)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.flowable.common.engine.api.FlowableException: Error while building ibatis SqlSessionFactory: null
at org.flowable.common.engine.impl.AbstractEngineConfiguration.initSqlSessionFactory(AbstractEngineConfiguration.java:686)
at org.flowable.engine.impl.cfg.ProcessEngineConfigurationImpl.init(ProcessEngineConfigurationImpl.java:960)
at org.flowable.engine.impl.cfg.ProcessEngineConfigurationImpl.buildProcessEngine(ProcessEngineConfigurationImpl.java:887)
at org.flowable.osgi.blueprint.ProcessEngineFactory.init(ProcessEngineFactory.java:41)
at org.flowable.osgi.blueprint.ProcessEngineFactoryWithELResolver.init(ProcessEngineFactoryWithELResolver.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:331)
at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:838)
at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:591)
… 21 more
Caused by: java.lang.NullPointerException
at java.io.Reader.(Reader.java:78)
at java.io.InputStreamReader.(InputStreamReader.java:72)
at org.flowable.common.engine.impl.AbstractEngineConfiguration.initSqlSessionFactory(AbstractEngineConfiguration.java:659)
… 32 more


#15

Hi Björn,

Finally, I’ve been playing around with your example a bit more.
I’ve made some changes on the current Flowable master (6.4.1-SNAPSHOT):

With these changes the Flowable Engine boots ok now in your example. You can also include the IDM engine. Let me know if it also works for you.

Best regards,

Tijs


#16

@donkon, did the changes I referred to in my last post solve the issue for you as well?

Best regards,

Tijs


#17

@tijs yes, the changes solved the issue for setting up the engine in Karaf, the following was required in my flowable-karaf-demo:

I added more functionality to the flowable-karaf-demo to see a bpmn workflow running in Karaf, unfortunatilly I encountered ClassNotFoundExcpetions at runtime (executing a ServiceTask):

org.flowable.common.engine.api.FlowableException: Error while evaluating expression: ${stringOperations.getSize(input)}
	at org.flowable.common.engine.impl.el.JuelExpression.getValue(JuelExpression.java:56) ~[71:org.flowable.common.engine.impl:6.4.1.SNAPSHOT]
....
Caused by: java.lang.ClassNotFoundException: org.flowable.common.engine.impl.de.odysseus.el.ExpressionFactoryImpl
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:?]
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:?]
	at org.flowable.common.engine.impl.javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:204) ~[?:?]
	at org.flowable.common.engine.impl.javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:183) ~[?:?]
	at org.flowable.common.engine.impl.javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:89) ~[?:?]
	at org.flowable.common.engine.impl.javax.el.BeanELResolver.getExpressionFactory(BeanELResolver.java:564) ~[?:?]

I think, the problem is the

Thread.currentThread().getContextClassLoader()

part in ExpressionFactory, which is not OSGi capable.

You can pull from flowable-karaf-demo to reproduce this issue (also notice the updated README.md)


#18

Event though my issue is on Flowable-engine v5.23.0 (as v6.4.0 currently issue), I have some class loading issues in regard to classes/beans used within a task delegate that are imported from another module. (see details in my last post under Flowable-engine in OSGi - Tutorials/examples?).

Has anyone met that situation?


#19

@tijs Any ideas regarding my last post?

@Ceddaerrix I answered in your post


#20

@donkon not yet but will look at it in the next days and come back to you.

Best regards,

Tijs