How to write history data to another mysql?

Dear engineers,
I am a developer in China. Our company hopes to use flowable to resolve millions of contents everyday. But as content increased, the database’s pressure is too high to load so much query. So we consider to move history data to another mysql server . I read the source codes ,and I found if I want to move history data to another MySQL server , I need to rewrite HistoricDataManager and DbSqlSession and so on. Is this way right? If it is right, the work is so huge, and I do not have enough time to do. I sincerely need your help , thank you!
Another way I thought is to make two engines. One is runtime and repository engine, and another one is historic engine. Two engines ’ s main differences is data source .I rewrite both of the historyjobhandler . By using kafka or other mq , the runtime engine’s historjobhandler send the messages in the override method. And the historic engine’s receive the messages in the override method and used all the original jsontransformer to save the data. Is this way feasible.
That’s all . Thank you very much.

The same question was asked as a Github issue and was answered by Tijs pointing you to AsyncHistory. In general the forum is the better place to ask questions, you will reach a much wider audience and allows us to keep the Github issues focused on bugs and features.

There is also this blog post on benchmarks and example code but some of the interfaces have changed slightly so the example code is not a drop in replacement unless you are using an older engine.

One upcoming feature in 6.5.0 that may help you is HistoryCleaning in the future. When enabled, you will be able to set a maximum age to keep history after a process has ended. Then at a configurable interval the engine will automatically clean any expired data from the history tables.

1 Like

Thank you wwitt very much!
You are so kindly.

Hi wwitt,

How exactly the old history data will be deleted by HistoryCleaning job. Is it going to lock database while database entries are deleted (considering table has very huge data/millions of rows), or will it do that in batches so that it does not occur any downtime to a production system?

@hariskhan14 Here are the docs. Once enabled, the history cleanup job executes based on a time cycle, removing everything older than a specified number of days. The default is to run everyday at 1 AM (server time) and remove everything that has completed more than 365 days in the past.

History cleanup is only removing the records, so it does not lock the database. There could be some record locking but that shouldn’t impact anything in the running system or new history being added. Of course, if you have years of history data the database may get very busy on the first run. Subsequent runs would only be deleting one day of history and should execute more quickly.

History cleanup does not do any additional database management tasks, like rebuilding indexes or compacting table space. If your database management system needs that kind of periodic maintenance, you will still need to conduct it.