Bonita cluster installation and configuration

Discover how to create a cluster from scratch, convert a single node into a cluster, allow communication between nodes and manage a cluster.

The cluster feature is a Subscription feature for Enterprise and Performance edition only.

It is possible to manage the lifecycle of a node using the API to connect directly to the node, but this bypasses the load balancer so should be done with care and only in exceptional circumstances.

Note when using the AWS support

Please read the following to ensure that Bonita Cluster works correctly on AWS.

IAM role configuration

If you want to use IAM role for EC2 autodiscovery, you need to attach a role to your EC2 instances with a policy allowing "ec2:DescribeInstances" action. For details about IAM Roles for Amazon EC2 see AWS documentation

Create a cluster from scratch

In this part we will create a cluster from scratch. We will initialize the database on which the cluster will run, then we will configure nodes to run on this cluster.

Allow communication between servers

Bonita cluster uses Hazelcast as the distributed cluster dispatcher layer. Hazelcast needs the port 5701 (by default) to be opened on all the servers that compose the cluster. Ensure that these ports are opened in all the nodes of your cluster.

For security reasons, port 5701 must be reachable only from the other Bonita Runtime nodes of the same cluster, and from nothing at all in a single-node installation. With Docker Compose, every container of a project shares the default network, so a compromised sibling container can reach the port; with Kubernetes, pod-to-pod traffic is unrestricted across namespaces unless a NetworkPolicy says otherwise, and leaving 5701 out of the Service does not block the pod IP. Restrict the port explicitly: a NetworkPolicy that admits only the Bonita pods, a dedicated network that third-party services are not attached to, or a security group or host firewall.

Two properties control the address the Hazelcast member listens on, one per deployment mode:

Cluster. By default a member listens on every interface, so that its peers can reach it. On a host with several networks, the optional property bonita.platform.cluster.hazelcast.interfaces.list in bonita-platform-sp-cluster-custom.properties binds the member to one address, for example 10.0.0.5. The value is a comma separated list of patterns, such as 10.0.0.*; Hazelcast picks one address of the node matching one of them and the member listens on that address only, so several entries are alternatives, not several addresses to listen on. This property is available from Bonita 2025.1-u11 (technical id 10.3.11). Earlier versions read a property of that name without applying it: if your configuration already sets it, make sure its value matches an address of the node, or the node does not start.
Single node (Cluster feature not enabled). A single node also starts a Hazelcast member, for its second-level cache; the member exists only while bonita.platform.persistence.use_second_level_cache is true, its default. From 2025.1-u11 that member listens on loopback (127.0.0.1) only, the default of the property bonita.platform.hazelcast.interfaces.list, so nothing has to be done for it. If a Hazelcast client or Management Center must reach it from another host, in bonita-platform-sp-custom.properties, uncomment the line and remove its value so that it reads bonita.platform.hazelcast.interfaces.list=: the member then follows the network settings of hazelcast.xml, which by default listen on every interface. Uncommenting the line without removing the value keeps the loopback default.

From Bonita 2025.1-u12 (technical id 10.3.12), the Hazelcast member accepts other Hazelcast members only, on a single node as in a cluster: Hazelcast clients, Management Center included, are refused, and the Hazelcast REST API, health check included, and Memcache are not served, even when hazelcast.xml enables them. Nothing in Bonita connects to the member as a client. A deployment that attaches a Hazelcast client or Management Center, or uses the REST API or Memcache, on purpose must set bonita.platform.hazelcast.client-protocol.enabled=true in bonita-platform-sp-custom.properties, with the same value on every node of a cluster, and restrict port 5701 to the hosts that need it: the Hazelcast edition bundled with Bonita has no client authentication, so any client that reaches the port can read and write the platform’s caches and sessions. Bonita logs a warning at every start while the property is true. On a single node, a client that connects from another host also needs the empty bonita.platform.hazelcast.interfaces.list described above. Refusing clients does not replace restricting the port: a host that reaches it can still join the cluster as a member.

Create and initialize the database for Bonita Platform

In this step you will create and initialize the database for the Bonita Platform cluster using the platform setup tool. When done you will have a database with all tables created and with a table CONFIGURATION containing all configuration required for the cluster to start.

Ensure that you meet the requirements.
Create the database.
In case you use Business data, create a database for the Business Data.
Download a Bonita Tomcat bundle and unzip it in your chosen folder.
Edit file setup/database.properties and modify the properties to suit your databases (Bonita internal database & Business Data database). Beware of backslash characters.

In the following steps, you will update the configuration files that are in the setup/platform_conf/initial folder of the platform setup tool.

Edit the file setup/platform_conf/initial/platform_engine/bonita-platform-sp-custom.properties: Uncomment and set the bonita.cluster property to true, as follows: bonita.cluster=true
Edit the file setup/platform_conf/initial/platform_engine/bonita-platform-sp-cluster-custom.properties: uncomment and set the bonita.cluster.name property to a name of your own, e.g. myBPMCluster, This name must be unique on the local network if you are using multicast

Edit the file setup/platform_conf/initial/platform_engine/bonita-platform-sp-cluster-custom.properties: set to true one of bonita.platform.cluster.hazelcast.multicast.enabled or bonita.platform.cluster.hazelcast.tcpip.enabled or bonita.platform.cluster.hazelcast.aws.enabled, as follows: Uncomment the # properties and set only one of them to true, set the others to false depending on how you want your nodes to discover each others. Example:

bonita.platform.cluster.hazelcast.multicast.enabled=false
bonita.platform.cluster.hazelcast.tcpip.enabled=true
bonita.platform.cluster.hazelcast.tcpip.members=ipServer01,ipServer02

If you don’t use bonita.platform.cluster.hazelcast.multicast.enabled, you must uncomment the # properties and set it to false to deactivate it, as follows: bonita.platform.cluster.hazelcast.multicast.enabled=false.

For more information on this take a look at the Hazelcast Documentation.

Copy licenses of all your nodes in setup/platform_conf/licenses.
Run setup.sh init or setup.bat init as described in the platform setup tool page.

At the end of the script, you should see the following line: "Initial configuration files successfully pushed to database". This creates the database tables needed by Bonita platform, stores the configuration into this database, and stores the license files for all your cluster nodes into the database.

If later you need to change the configuration of the node discovery or add new licenses to the Bonita Platform configuration, you can update it by following this guide.

Install a first node

Once you have done all the steps from section Create and initialize the database for Bonita Platform above, run setup.sh configure or setup.bat configure as described in the Bundle configuration to have your Tomcat bundle configured for pointing to the right database.
Once the bundle is configured, and to avoid unsynchronized versions of the configuration files between several nodes, you are advised to delete the folder [TOMCAT_DIRECTORY]/setup and its entire content. Note: if you want to keep it for future configuration changes, just move the folder [TOMCAT_DIRECTORY]/setup and its entire content outside the [TOMCAT_DIRECTORY] to use it as your main platform setup tool.
If your Bonita installation is behind a proxy or is installed inside a Docker container (mainly in TcpIp or Aws discovery modes), you must declare its public address by adding the following property: -Dhazelcast.local.publicAddress=publicaddress, this property should be added in the [TOMCAT_DIRECTORY]/setup/tomcat-templates/setenv.sh or [TOMCAT_DIRECTORY]/setup/tomcat-templates/setenv.bat
When the installation is complete, start Tomcat on the node. This starts Bonita Platform:
```
./start-bonita.sh
```

Then start the cluster in the load balancer.

If you are using apache-tomcat load-balancer with mod_jk, make sure that jvmRoute attribute is set at your Engine

<Engine name="Catalina" defaultHost="localhost" jvmRoute="node01">

and that the jvmRoute attribute value matches your worker name in workers.properties. See the official documentation: https://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html

In the server.xml file, for example, change the line to: Engine name="Catalina" defaultHost="localhost" jvmRoute="node01"

Check that the log file contains messages of the following form:

March 22, 2016 5:07:07 PM INFO: com.hazelcast.cluster.ClusterService [10.0.5.3]:5701 [myBPMCluster]

Members [1] {
        Member [10.0.5.3]:5701 this
}
[...]
March 22, 2016 5:09:18 PM INFO: org.apache.catalina.startup.Catalina start Server startup in 30333 ms

Then deploy a basic process and check that it runs correctly, to validate the installation.

Add a node to the cluster

You can add a new node to a cluster without interrupting service on the existing nodes.

Copy the entire Tomcat directory to another machine.
If Hazelcast Node discovery is configured with TCP, update the configuration in database using the platform setup tool, as follows:
1. Run the setup.sh pull or setup.bat pull. This will retrieve the configuration of your platform under platform_conf/current folder.
2. Edit the file platform_conf/current/platform_engine/bonita-platform-sp-cluster-custom.properties and add the node to the list of members as follows for example: bonita.platform.cluster.hazelcast.tcpip.members=ipServer01,ipServer02,ipServer03
Start the Tomcat on the new node, running ./start-bonita.sh script

Update the load balancer configuration to include the new node. The log file will contain messages of the following form:

March 22, 2016 5:12:53 PM INFO: com.hazelcast.cluster.ClusterService [10.0.5.17]:5701 [myBPMCluster]

Members [2] {
        Member [10.0.5.3]:5701
        Member [10.0.5.17]:5701 this
}
[...]
March 22, 2016 5:12:28 PM INFO: org.apache.coyote.http11.Http11Protocol start Starting Coyote HTTP/1.1 on http-7280
March 22, 2016 5:12:28 PM INFO: org.apache.catalina.startup.Catalina start Server startup in 30333 ms

In the log, you can see how many nodes are in the cluster, and their IP addresses and port number. This node that has been started is indicated by this. The new node is now available to perform work as directed by the load balancer.

Convert a single node installation into a cluster

In this case you already have a Bonita Platform running as single node installation, you will change the configuration to make it able to have multiple nodes.

Update the configuration in database

Some properties of the Bonita Platform need to be changed, through Bonita platform setup tool, in order to make your installation work as a cluster node.

Download Bonita Tomcat bundle, that contains the platform setup tool, and unzip it in your chosen folder.
Go into the setup folder: cd ./setup/
Configure the Setup Tool as described in the platform setup tool page
Run the setup.sh pull or setup.bat pull. This will retrieve the configuration of your platform under platform_conf/current folder.
Update configuration files that are in the platform_conf/current folder of the platform setup tool.
- In platform_engine/bonita-platform-sp-custom.properties
  - uncomment and set the bonita.cluster property to true.
- In platform_engine/bonita-platform-sp-cluster-custom.properties
  - uncomment and set the bonita.cluster.name property to a name of your own, e.g. myBPMCluster, This name must be unique on the local network if you are using multicast
  - set one of bonita.platform.cluster.hazelcast.multicast.enabled, bonita.platform.cluster.hazelcast.tcpip.enabled and bonita.platform.cluster.hazelcast.aws.enabled to true: uncomment the # properties and set only one of them to true, set the others to false depending on how you want your nodes to discover each others, for more information on this take a look at the Hazelcast Documentation.
Copy licenses of all your nodes in platform_conf/licenses
Run the setup.sh push or setup.bat push. This will update in database the configuration of your platform.

Configure nodes to run on this cluster

The configuration of the node you were using is still valid. You should be able to run it without any issue.

If your Bonita installation is behind a proxy or is installed inside a Docker container, please refer to the

Install a first node part.

Configure sessions and session cookies

Since Bonita 2023.2, HTTP sessions are handled (and shared between nodes) using Spring Session instead of Tomcat. As a result, in order to override the default session timeout or the session cookies attributes, Tomcat configuration may be irrelevant. But the following properties can be added and configured in the file platform_engine/bonita-platform-sp-cluster-custom.properties instead:

server.servlet.session.cookie.name=JSESSIONID
server.servlet.session.cookie.same-site=Lax
server.servlet.session.cookie.secure=false
server.servlet.session.cookie.http-only=true
server.servlet.session.cookie.max-age=-1
server.servlet.session.cookie.domain=
server.servlet.session.cookie.path=/
spring.session.timeout=1800

These properties can also be set using environment variables or system properties.

Cluster management

Stop a node

Simply run ./stop-bonita.sh script.

Remove a node from a cluster

This section explains how to perform a planned shutdown and remove a node from the cluster.

Update the load balancer configuration so that no further work is directed to the node. All work that is already in progress on the node that will be shutdown will continue until completion. Do not remove the node completely, because the load balancer needs to be informed when current work is finished.
Allow current activity instances to complete.
Stop the Tomcat server: run ./stop-bonita.sh
Update the load balancer to remove the node from the cluster.

The node is now removed from the cluster.

Work recovery when a node leaves the cluster

When a node leaves the cluster, whether it was stopped in a planned way or failed, the work it had started but not finished is taken over by one of the remaining nodes. Only one node performs that recovery: the others detect it is already being handled and carry on with their own work. The recovery runs in the background on that node, one departed node at a time, and no longer holds up the handling of other cluster events on it.

Work that a recovery could not take over stays in the departed node’s queue. It is taken over again the next time a node leaves or starts, and in the meantime the periodic recovery (bonita.tenant.recover.*) picks up what it covers. A node that resumes after a tenant pause resubmits the work that arrived while it was paused.

Two properties limit how long that takes. Their default values suit most installations; they can be changed in the file platform_engine/bonita-platform-sp-cluster-custom.properties, together with the lease used by the lock service:

bonita.platform.cluster.work.recoveryLockTimeoutSeconds and bonita.platform.cluster.work.recoveryLockLeaseSeconds are available from Bonita 2025.1-u11 (technical id 10.3.11) onwards. bonita.platform.cluster.lock.leaseTimeSeconds is also available in earlier versions.

# Seconds a node waits for its turn to recover the work of a node that has left.
bonita.platform.cluster.work.recoveryLockTimeoutSeconds=30
# Seconds a node may spend handing over the work of a node that has left. The handover stops by
# itself after 90% of this time; work not handed over by then is taken over again when a node next
# leaves or starts, or picked up by the periodic recovery.
bonita.platform.cluster.work.recoveryLockLeaseSeconds=14400
# Seconds a lock on a process instance or a task is held before the cluster releases it.
bonita.platform.cluster.lock.leaseTimeSeconds=600

These properties can also be set as Java system properties, using the name above unchanged, or as environment variables:

BONITA_PLATFORM_CLUSTER_WORK_RECOVERYLOCKTIMEOUTSECONDS=30
BONITA_PLATFORM_CLUSTER_WORK_RECOVERYLOCKLEASESECONDS=14400
BONITA_PLATFORM_CLUSTER_LOCK_LEASETIMESECONDS=600

Keep bonita.platform.cluster.work.recoveryLockLeaseSeconds high, in hours rather than minutes. The handover of a departed node’s work stops by itself after 90% of the lease, so that it never outlives its lock, and the work not handed over by then waits for the periodic recovery. Too low a value therefore cuts short the handover of a large queue. Raise it if the log reports that a handover gave up because it had spent its share of the lease, or that a work recovery lock could not be released because its lease had expired. In the second case, another node may have started the same handover, and some work may have run twice.

A high value has one cost: if a node hangs while holding the lease, the recovery of that node’s work waits that long. A node that stops or fails releases the lease by leaving the cluster.

Raising bonita.platform.cluster.work.recoveryLockTimeoutSeconds is rarely useful: on every departure all remaining nodes but one are expected to give up waiting, and the message reporting it is informational.

Dismantle a cluster

To dismantle a cluster:

Disable processes.
Allow current activity instances to complete.
When each node has finished executing, stop it.
When all nodes have been stopped, update the load balancer to remove the cluster.

The individual nodes can now be used as standalone Bonita server, provided the following change in the configuration is done: Update file bonita-platform-sp-custom.properties located in the platform_engine folder of the configuration, use the platform setup tool to update it and set back the bonita.cluster property to false.

See How to update a Bonita Tomcat Bundle configuration for more details on updating the configuration.

Managing the cluster with Hazelcast

As said before, Bonita cluster uses Hazelcast as the distributed cluster dispatcher layer. Therefore, you can use the Hazelcast tools to manage the cluster topology. See the Hazelcast documentation for details.

Note that a Bonita cluster uses multicast for discovery by default. You can disable this in Hazelcast. If you are using multicast, you must ensure that your production environment is insulated from any test environment that might also contain cluster nodes. This is to ensure the nodes do not discover each other on the network, if they are not supposed to run inside the same cluster.

It is possible to have more than one cluster on the same network. In this case, you must configure the cluster names to be sure that it is clear which node belongs to which cluster. You can configure the cluster name through Hazelcast or by updating bonita-platform-sp-custom.properties located in the platform_engine folder of the configuration, use the platform setup tool to update it.

Troubleshooting

Symptom: I regularly get this warning message when 2 or more nodes are started in cluster:

2016-06-13 11:41:22.783 +0200 WARNING: org.bonitasoft.engine.scheduler.impl.BonitaJobStoreCMT This scheduler instance (...) is still active but was recovered by another instance in the cluster.  This may cause inconsistent behavior.

Root cause: The clocks of the servers are not synchronized.

Resolution: The system time of all cluster nodes must be maintained in synchronization with time servers. It is a good idea to have also the db server system time synchronized too. Synchronize the system time of all nodes and restart application servers.