Microservices orchestration - Zeebe

Opensource microservice orchestration engine, implemented in highly scalable, highly performant and fault tolerant way using state of the art, proven technology building blocks

• Using Raft protocol for consensus and replication

• Using RocksDB, an embeddable high performance key-value store, for its persistency

• All process instances are auditable, and audit records can be sent to Elasticsearch or Kafka

• Using an efficient, high performance binary protocol (gRPC) for the clients to connect and receive tasks for execution

Benefits	Description
Workflow definition	BPMN 2.0, YAML - provides integrators with an easy to use tool to customize workflows
Visual workflow representation	Yes, BPMN workflows are both executable and have visual representation
State storage for active workflow instances	Stored on the machines running Zeebe using embedded RocksDB (no external storage required)
Scalability	Horizontally scalable via partitions (no DB bottleneck)
Fault tolerance	Yes, via replication factor (3 node, 1 can fail, or 5 node and 2 can fail)
Supported programming languages	Ships with Java and Go clients, plus NodeJS, C# and Ruby clients available, using gRPC for the clients
Historic workflow data	Can be streamed to storage systems via exporters (Elasticsearch and Apache Kafka is available). Complete auditable log.

Important features

• Timers surviving node failures without interruption

• Correlation of incoming messages to existing workflow instances across the cluster nodes

• Capability to handle non correlating cases - important for cross payment hub engine flows

• Handle failover situation without interruption (3 nodes: 1 can fail, or 5 nodes: 2 can fail) - utilizing Raft protocol for consensus, hand over “leadership” in case failed nodes and replication of state

Focus on system integrators

• BPMN let implementation team focus on the business flow changes inside the given DFSP, making adoption and customization much easier

• Multi programming language support for component implementation, let implementation team to create the necessary connector components in any of the supported languages, the engine is able to orchestrate the steps. (e.g. multiple AMS integration is required), using gRPC protocol for efficient communication

• Connector components are stateless, easy and simple implementation

• No additional components required in the realtime engines, no databases to maintain, all active process states are stored in the embedded RocksDB data store. Auditlogs, and other events are stored in Kafka clusters before storing them

• Scalability - using partitioning (sharding), the system scales horizontally to create thousands of workflow instances per second

• Fault tolerance enables, that system recovers from a failure without data loss, and using replication to define the number of copies for all instances In case of failures, “leader election” is taking place for the given partitions to continue operations uninterrupted (using Raft Consensus and Replication Protocol)