chapter 1 · Building Arda
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
page 4 of 7 · read 0%

Jan 2026 · Updated Oct 2026 · 11 min read · ARDA Labs · Architecture note

Putting BPMN Behind a Workflow Service Boundary

How Arda's workflow-service sits in front of Zeebe 8.5: what is stored in the engine, in Postgres and in domain services, how a case is submitted and recovered, and where the boundary still leaks.

✎ arda-labs/arda-be · #Arda Labs #Zeebe #BPMN #Workflow #Go #gRPC #Transactional Outbox #Architecture

At a glance

2 RPCs

Workflow gRPC surface

WorkflowCommandService exposes CreateCase and SubmitCase in proto/arda/workflow/v1.

44 BPMN

Embedded processes

144 service tasks and 91 user tasks, all service tasks with retries=3, 56 error events, no timers.

v8.5.5

Pinned Go client

github.com/camunda/zeebe/clients/go/v8 in apps/workflow-service/go.mod; infra README says Zeebe 8.5.x.

1 partition

Helm Zeebe layout

clusterSize 1, partitionCount 1, replicationFactor 1 in arda-infra values.

Problem Context

A BPMN engine holds process state that every caller would otherwise have to understand and survive the outage of. Arda puts one service in front of Zeebe so that fifteen services see two gRPC calls, while case records, inbox rows, decisions and events live in Postgres next to the tenant scope.

BPMN Workflow-Service Boundary (Zeebe 8.5)
Domain Servicesfinance-servicecrm-servicehrm-serviceworkflow-servicegRPC & HTTP FacadeCreateCase / SubmitCaseUser Task AssignmentSLA Timers & Error BoundaryDomain Event HooksZeebe 8.5BPMN 2.0OrchestrationEngine

Putting BPMN Behind a Workflow Service Boundary

Architecture & Design

workflow-service deploys 44 embedded BPMN files at startup, runs Zeebe job workers that call domain services over gRPC, and projects Zeebe user tasks into Postgres through the Elasticsearch exporter. SubmitCase takes a per-case advisory lock, starts the instance, then records the process key; terminal transitions and events are written in one transaction to workflow_outbox, relayed to NATS JetStream.

Key Technical Decisions

  1. 1Only workflow-service holds the Zeebe address: domain code never sees job types, but variable names remain a shared contract.
  2. 2Postgres owns case records, decisions and inbox rows: tenant scope and idempotency are enforced with ordinary unique indexes, at the cost of keeping two stores aligned.
  3. 3Decision is recorded before the engine call: a failed call can be reconciled, but the decision exists even if the engine never saw it.
  4. 4SLA warnings come from a one-minute Postgres scan, not BPMN timers: no process path changes on a missed deadline.
  5. 5Domain side effects are idempotent worker calls, not cross-service transactions: retries=3 is safe only where the domain checks status first.

What the boundary is, and what it is for

A BPMN engine keeps its own state: process instances, jobs, variables, user tasks. Every service that talks to it directly has to learn that vocabulary and has to decide, on its own, what to do when the engine is slow or down. In Arda that would be fifteen services (apps/ has ai, auth-gateway, capital, crm, deposit, finance, hrm, iam, loan, mdm, media, notification, platform, statistical and workflow).

The repo states the rule in several places. apps/workflow-service/README.md: "workflow-service is the only backend service that should connect to Zeebe", and zeebe_addr belongs only there. docs/refactor-program/phase-0/service-topology.md describes it as a target state, and docs/refactor-program/06-phased-execution-plan.md lists "No domain service connects directly to Zeebe" as an exit criterion. The docs do not say what prompted it, so I can only describe what it does today.

The contract is small. proto/arda/workflow/v1/workflow.proto defines one service with two RPCs:

// proto/arda/workflow/v1/workflow.proto
service WorkflowCommandService {
  rpc CreateCase(CreateCaseRequest) returns (BusinessCase);
  rpc SubmitCase(SubmitCaseRequest) returns (BusinessCase);
}

Claiming and completing human tasks, monitoring, incidents and process definitions are HTTP under /api/workflow/*, used by the workflow remote in arda-mfe. The proto has no cancel, claim or complete.

The boundary is only as good as the grep that checks it: in arda-be, github.com/camunda/zeebe is imported only by workflow-service. One leftover: arda-infra/k8s/apps/notification-service.yaml still sets a ZEEBE_ADDR env var, although apps/notification-service/README.md says that service does not connect to Zeebe and no Go code in it reads the variable.

What lives in Zeebe, in Postgres, and in the domain services

The split is easiest to see as a table. I built it from the migrations, the BPMN files in internal/bootstrap/ and the worker code.

ConcernLives inEvidence
Control flow: gateways, who acts next, return loopsZeebe, from 44 embedded BPMN filesinternal/bootstrap/*.bpmn
Job retries and error routingBPMN: all 144 service tasks declare retries="3"; 56 error event definitions, no timers, no message eventsgrep over the same files
Case record ("hồ sơ nghiệp vụ"), status, timelinePostgres: business_cases, case_timeline_eventsmigrations/20260702090000_business_cases.sql
Which case type uses which process, and its roles/SLAPostgres: business_operation_types (bpmn_process_id, bpmn_version, maker_role, checker_role), workflow_assignment_rules, workflow_case_type_stepsseed migrations
Inbox rows for humansPostgres: workflow_tasks, one row per activation20260919110000_workflow_task_activations.sql
Human decisionsPostgres: workflow_task_decisions (UNIQUE (idempotency_key))20260919100000_workflow_registry_v2.sql
Business effect (loan adjustment, journal entry, customer)The owning domain service, reached by gRPC from a job workerinternal/worker/loan_workers.go
Events to other servicesPostgres workflow_outbox, relayed to NATS JetStream20261009090000_workflow_outbox.sql

Zeebe is the source of truth for "where is this instance". Postgres is the source of truth for "what does the business call this, who may act, and what did they decide". The two are joined by business_cases.process_instance_key and by a few service-owned process variables (caseId, tenantId, caseType, ...).

docs/crm-hrm-bpm-data-design.md records the rule that keeps domain data out of the workflow database: only the owning service writes to its own tables, workflow-service "only emits jobs", and the cost is explicit: the 1-transaction atomicity of the old EPAS design is lost and replaced by "job retries=3 + worker idempotent by caseId".

Cases pin their process definition. The migration that moves LNM_RECOVERY_V2 to the shared common-maker-checker process says in a comment: "In-flight cases keep their pinned old definition". The step registry works the same way: business_cases.registry_version is set at submit and metadata edits create a new version.

Starting a process: the submit path

SubmitCase is where the two stores meet, so it is where most of the care went. The sequence in workflow_command_service.go:

  1. Take a PostgreSQL advisory lock keyed by tenant and case id, on a dedicated connection, so retries on different replicas serialize.
  2. Reload the case. If it is no longer DRAFT, return the stored result when the idempotency key and request hash match, or ErrIdempotencyConflict when only the key matches.
  3. Refuse to start if the case type has no active registry steps (the inbox projector could never find its human tasks).
  4. Build the process variables: service-owned keys (caseId, tenantId, caseType, primaryObjectId, actorUserId, ...) are derived from the stored case and re-asserted last, and client-supplied values for those keys are dropped and logged.
  5. Start the instance, then commit the DRAFT to SUBMITTED transition.
// apps/workflow-service/internal/service/workflow_command_service.go
processKey, err := s.zeebeSvc.StartWorkflow(ctx, *bc.BpmnProcessID, variables)
if err != nil {
    return nil, fmt.Errorf("start workflow: %w", err)
}
updated, err := s.caseRepo.SubmitCase(ctx, id, in.Actor, processKey, in.IdempotencyKey, requestHash)
if err != nil {
    return nil, fmt.Errorf("workflow started but case submit failed: %w", err)
}

Case creation has its own idempotency: unique indexes business_cases_tenant_idempotency_uq and business_cases_tenant_submit_idempotency_uq (both on (tenant_id, key)), plus a stored request hash so a reused key with a different payload is a conflict, not a silent replay (20260825161000_add_idempotency_request_hash.sql).

Callers pick deterministic keys. loan-service builds lnm-<kind>-<id> for create and lnm-<kind>-<id>-submit for submit (adjustment_service.go). The batch disbursement path goes further: it writes a row to its own lnm_disbursement_workflow_outbox in the same transaction as the batch, and a relay calls CreateCase/SubmitCase later with keys derived from the batch id, up to disbursementWorkflowMaxAttempts = 8. The code comment gives the reason: "API latency does not depend on Zeebe availability".

The gap. Zeebe is called before the Postgres commit and there is no compensation. If StartWorkflow succeeds and SubmitCase then fails (database down, process killed), the case is still DRAFT with no process key, while an instance is running in Zeebe. The retry starts a second instance. CancelWorkflow is only called from the HTTP cancel and operate handlers, not from this path. I found no test for this window; workflow_idempotency_test.go only checks that the request hash is deterministic. The advisory lock closes concurrent retries, not this crash.

Job workers and human tasks

Service tasks. Workers run inside workflow-service (cmd/workflow-service/main.go opens one Zeebe job worker per job type) and are pure transport: the handler reads variables, calls the domain over gRPC with a tenant context built from the job's variables, then completes or fails the job. The loan worker's execute step:

// apps/workflow-service/internal/worker/loan_workers.go
if err := w.loanClient.ResolveAdjustment(crmJobContext(job), kind, id, "APPROVE", decidedBy, note); err != nil {
    w.failJob(client, job, "Loan Error: "+err.Error())
    return
}
if err := w.completeJob(client, job, map[string]any{"approvalStatus": "APPROVED"}); err != nil {
    return
}
w.finishCase(job, repository.CaseStatusCompleted)

Three outcomes exist. A validation problem is a BPMN error (throwValidationError, code VALIDATION_FAILED), which the process routes through a boundary event back to the maker. A transient failure calls failJob with retries - 1, and once retries hit zero Zeebe raises an incident. A malformed job fails with retries 0 immediately.

The domain side must tolerate a repeated call, because the worker can crash after the domain commit and before completeJob. For loan adjustments it does: ResolveAdjustment updates WHERE status = 'PENDING', and a zero-row update falls through to CheckAdjustmentReplay, which accepts the same decision as a replay (apps/loan-service/internal/repository/adjustment_repo.go). Other domains have to do the same; docs/crm-hrm-bpm-data-design.md makes it a stated requirement ("worker idempotent"), but I only checked the loan path.

common-maker-checker.bpmn is one shared process whose role groups come from variables (candidateGroups="=[mcMakerRole]"); a case type opts in through business_operation_types.worker_kind and workers dispatch on mcKind. migrations/canary/ holds two canary migrations: LNM_RECOVERY_V2 and batch disbursement register.

Human tasks. These are native Zeebe user tasks (91 zeebe:userTask elements). The README is explicit about the constraint: Camunda 8.5 gateway REST cannot list user tasks, and Tasklist is disabled in arda-infra/platform/helm/zeebe/values.yaml, so UserTaskProjector polls the Zeebe Elasticsearch exporter every 2 seconds for up to 100 active cases and upserts workflow_tasks rows. Completion goes through the gateway REST API, but only after the decision is written down:

// apps/workflow-service/internal/handler/workflow_handler.go
// Record the decision durably before any engine/domain side effect so the
// command can be reconciled if the engine call fails midway.
if err := h.recordTaskDecision(r, jobKey, processInstanceKey, elementID, actor, comment, variables); err != nil { ... }

Authorization happens before that: the task must be in the inbox or verifiable against the claimed process, the actor must be assigned or in the candidate role, and maker-checker separation is enforced. Approve and reject are confirmed when the terminal worker calls FinishCase; submit and return decisions are applied by DecisionDispatcher adapters and only then marked APPLIED.

Failure modes I could trace

FailureWhat the code doesGap
Zeebe unreachable at startupmain.go exits with status 1 if the client cannot be created, and again if any built-in BPMN fails to deploy (all 44 are redeployed on every start)The pod crash-loops until the gateway is back
Zeebe down at submitStartWorkflow errors, nothing is written, case stays DRAFT, retry is safeThe gRPC server maps every non-NotFound error to FailedPrecondition, so callers cannot tell "engine down" from "bad state"; CreateCase errors all map to InvalidArgument
Crash after StartWorkflow, before SubmitCase commitSee previous sectionOrphan instance, second instance on retry
Worker crashes before completeJobThe job is re-delivered to a worker; the domain call must be replay-safeOnly verified for loan adjustments. The repo sets no job timeout or worker options, so library defaults apply
completeJob succeeds, FinishCase failsThe error is only logged by CaseProjectionThe case stays non-terminal in Postgres while Zeebe has ended the instance. I found no sweeper that repairs it
Human clicks complete twiceauthorizeUserTaskComplete rejects a task whose work item is already COMPLETED; the decision key defaults to task-<jobKey>-<decision> (or the Idempotency-Key header) and is UNIQUE, so a retry after an engine failure reuses the recorded decisionIf the engine call succeeds and the DB write after it fails, the work item stays open until the reconciler closes it
Engine completes a task but workflow_tasks stays openWorkflowReconciler (every 2 minutes) closes the row when the engine reports COMPLETED/CANCELED, or when the key is absent while another task of that process is CREATED; otherwise it only touches the rowDepends on the Elasticsearch exporter being current
NATS downEvents accumulate in workflow_outbox; relay retries every 2 seconds; dedupe_key is UNIQUE and the row id goes out as Nats-Msg-IdNo attempt cap or dead-letter: publish_attempts and last_error are recorded, and the row is selected again forever

The workflow outbox is a normal transactional outbox. FinishCase updates business_cases, closes workflow_tasks, marks decisions APPLIED and inserts the event row in one transaction. The relay then claims up to 50 rows with FOR UPDATE SKIP LOCKED, so several replicas do not publish the same row.

SLA warnings do not use Zeebe timers. The BPMN corpus has no timerEventDefinition. Instead a goroutine in main.go calls EmitSLANotifications every minute and enqueues workflow.task.sla_warning / workflow.task.overdue events with the dedupe key task:<id>:sla:<milestone>, for milestones 50, 90 and 100 percent of the created-to-due interval by default. Escalation is a notification; no process path changes when a deadline passes.

Tenant scoping, coupling that remains, and what I would check next

Tenant scoping is applied in Postgres, not in the engine. I found no use of Zeebe multi-tenancy: the Zeebe commands carry no tenant id, and tenantId is an ordinary process variable. Isolation comes from three places:

  • Every CaseRepository read and write starts with verifiedTenant(ctx), taken from the gRPC/HTTP metadata, and returns ErrTenantScopeRequired when it is empty. CreateCase also rejects a request whose tenant_id differs from the verified one.
  • The unique indexes on case code and idempotency keys are per tenant, as above.
  • When a worker calls a domain service, it rebuilds the tenant context from the job's variables (crmJobContext). The comment there is the honest summary: there is no inbound request, so scope "must come from versioned process variables; missing tenant/org intentionally fails closed". That is why client input can never set tenantId: it is in the reserved-key list and is overwritten from the stored case.

The consequence is that one Zeebe cluster (clusterSize: 1, partitionCount: 1 in the Helm values) is shared by all tenants. Only the facade and the network boundary keep a caller away from other tenants' variables.

What still couples the services to Zeebe, even though none of them imports its client:

  • Variable names. adjustmentId, batchId, decision, approvalStatus are contracts between a domain service and a BPMN file. The domain service builds the map; the BPMN reads it. A typo fails at run time, in a gateway expression.
  • Job topics. Every BPMN service task needs a registered worker. TestEveryServiceTaskTopicHasRegisteredWorker checks this at build time against RegisteredJobTopics, because the failure mode is a case parked forever at its last step.
  • Step codes. The inbox and the allowed actions are derived from the BPMN at startup (SeedRegistry); startup fails if an ACTIVE case type has no human steps.
  • Camunda 8.5 specifics. The repo pins github.com/camunda/zeebe/clients/go/v8 v8.5.5 in apps/workflow-service/go.mod, and arda-infra/platform/README.md says "Camunda Platform chart 10.11.3 / Zeebe 8.5.x". The Helm values do not pin an image tag. Because Tasklist is off and 8.5 gateway REST has no user-task search, the Elasticsearch exporter is a hard dependency (ZEEBE_ES_URL). Changing engine versions means revisiting the projector, not just the client.

Known gaps from the repo itself. The README lists "No runtime task claim/complete/reassign facade yet" and "No incident retry/suspend/resume APIs yet"; the code has since gained incident resolve and job retry handlers (operate.go), so that list is partly stale, and called-instance data in the monitor is still pending.

What I would check next. Reproduce the StartWorkflow/SubmitCase crash window with a fault-injection test and decide between a compensating CancelWorkflow and a pending-start row. Add a repair path for FinishCase failures. Set a poison limit on workflow_outbox. Distinguish engine-unavailable from precondition errors in the gRPC status codes.

Verified repository evidence

  • workflow · direct

    Only workflow-service talks to Zeebe 8.5; other services reach it through a two-RPC WorkflowCommandService (CreateCase, SubmitCase), with idempotent submit and a workflow outbox.

    ↳ Zeebe Go client v8.5.5 is imported only by workflow-service; case create/submit are idempotent per tenant; SLA milestones come from a Postgres scan.

    arda-labs/arda-be ↗

Key Engineering Takeaways

  • The boundary removes the engine client from fifteen codebases, not the engine from their design: variables, topics and step codes still couple them.
  • The riskiest step is the one that touches both stores: StartWorkflow runs before the Postgres commit and has no compensation in the repo.
  • Most recovery is polling and reconciliation (projector, reconciler, sweeper, outbox relays), each with a stated gap.