Yes there has been investigation in the past.
See this forum post.
With version 6 the data persistence for Flowable is fully pluggable allowing for alternative implementations such as recent CroachroachDB (see this blog post) implementation in 6.4.2 and experiments with MongoDB (see this blog post).
And there is this branch of work done by Joram Barrez using Neo4j.