At a glance
2 RPCs
Workflow gRPC surface
WorkflowCommandService exposes CreateCase and SubmitCase in proto/arda/workflow/v1.
44 BPMN
Embedded processes
144 service tasks and 91 user tasks, all service tasks with retries=3, 56 error events, no timers.
v8.5.5
Pinned Go client
github.com/camunda/zeebe/clients/go/v8 in apps/workflow-service/go.mod; infra README says Zeebe 8.5.x.
1 partition
Helm Zeebe layout
clusterSize 1, partitionCount 1, replicationFactor 1 in arda-infra values.
✎ Key Insights in 30 Seconds
Architecture note: How Arda's workflow-service sits in front of Zeebe 8.5: what is stored in the engine, in Postgres and in domain services, how a case is submitted and recovered, and where the boundary still leaks.
Problem Context
A BPMN engine holds process state that every caller would otherwise have to understand and survive the outage of. Arda puts one service in front of Zeebe so that fifteen services see two gRPC calls, while case records, inbox rows, decisions and events live in Postgres next to the tenant scope.
Putting BPMN Behind a Workflow Service Boundary
Architecture & Design
workflow-service deploys 44 embedded BPMN files at startup, runs Zeebe job workers that call domain services over gRPC, and projects Zeebe user tasks into Postgres through the Elasticsearch exporter. SubmitCase takes a per-case advisory lock, starts the instance, then records the process key; terminal transitions and events are written in one transaction to workflow_outbox, relayed to NATS JetStream.
Key Technical Decisions
- 1Only workflow-service holds the Zeebe address: domain code never sees job types, but variable names remain a shared contract.
- 2Postgres owns case records, decisions and inbox rows: tenant scope and idempotency are enforced with ordinary unique indexes, at the cost of keeping two stores aligned.
- 3Decision is recorded before the engine call: a failed call can be reconciled, but the decision exists even if the engine never saw it.
- 4SLA warnings come from a one-minute Postgres scan, not BPMN timers: no process path changes on a missed deadline.
- 5Domain side effects are idempotent worker calls, not cross-service transactions: retries=3 is safe only where the domain checks status first.
What the boundary is, and what it is for
A BPMN engine keeps its own state: process instances, jobs, variables, user tasks. Every service that talks to it directly has to learn that vocabulary and has to decide, on its own, what to do when the engine is slow or down. In Arda that would be fifteen services (apps/ has ai, auth-gateway, capital, crm, deposit, finance, hrm, iam, loan, mdm, media, notification, platform, statistical and workflow).
The repo states the rule in several places. apps/workflow-service/README.md: "workflow-service is the only backend service that should connect to Zeebe", and zeebe_addr belongs only there. docs/refactor-program/phase-0/service-topology.md describes it as a target state, and docs/refactor-program/06-phased-execution-plan.md lists "No domain service connects directly to Zeebe" as an exit criterion. The docs do not say what prompted it, so I can only describe what it does today.
The contract is small. proto/arda/workflow/v1/workflow.proto defines one service with two RPCs:
// proto/arda/workflow/v1/workflow.proto
service WorkflowCommandService {
rpc CreateCase(CreateCaseRequest) returns (BusinessCase);
rpc SubmitCase(SubmitCaseRequest) returns (BusinessCase);
}
Claiming and completing human tasks, monitoring, incidents and process definitions are HTTP under /api/workflow/*, used by the workflow remote in arda-mfe. The proto has no cancel, claim or complete.
The boundary is only as good as the grep that checks it: in arda-be, github.com/camunda/zeebe is imported only by workflow-service. One leftover: arda-infra/k8s/apps/notification-service.yaml still sets a ZEEBE_ADDR env var, although apps/notification-service/README.md says that service does not connect to Zeebe and no Go code in it reads the variable.
What lives in Zeebe, in Postgres, and in the domain services
The split is easiest to see as a table. I built it from the migrations, the BPMN files in internal/bootstrap/ and the worker code.
| Concern | Lives in | Evidence |
|---|---|---|
| Control flow: gateways, who acts next, return loops | Zeebe, from 44 embedded BPMN files | internal/bootstrap/*.bpmn |
| Job retries and error routing | BPMN: all 144 service tasks declare retries="3"; 56 error event definitions, no timers, no message events | grep over the same files |
| Case record ("hồ sơ nghiệp vụ"), status, timeline | Postgres: business_cases, case_timeline_events | migrations/20260702090000_business_cases.sql |
| Which case type uses which process, and its roles/SLA | Postgres: business_operation_types (bpmn_process_id, bpmn_version, maker_role, checker_role), workflow_assignment_rules, workflow_case_type_steps | seed migrations |
| Inbox rows for humans | Postgres: workflow_tasks, one row per activation | 20260919110000_workflow_task_activations.sql |
| Human decisions | Postgres: workflow_task_decisions (UNIQUE (idempotency_key)) | 20260919100000_workflow_registry_v2.sql |
| Business effect (loan adjustment, journal entry, customer) | The owning domain service, reached by gRPC from a job worker | internal/worker/loan_workers.go |
| Events to other services | Postgres workflow_outbox, relayed to NATS JetStream | 20261009090000_workflow_outbox.sql |
Zeebe is the source of truth for "where is this instance". Postgres is the source of truth for "what does the business call this, who may act, and what did they decide". The two are joined by business_cases.process_instance_key and by a few service-owned process variables (caseId, tenantId, caseType, ...).
docs/crm-hrm-bpm-data-design.md records the rule that keeps domain data out of the workflow database: only the owning service writes to its own tables, workflow-service "only emits jobs", and the cost is explicit: the 1-transaction atomicity of the old EPAS design is lost and replaced by "job retries=3 + worker idempotent by caseId".
Cases pin their process definition. The migration that moves LNM_RECOVERY_V2 to the shared common-maker-checker process says in a comment: "In-flight cases keep their pinned old definition". The step registry works the same way: business_cases.registry_version is set at submit and metadata edits create a new version.
Starting a process: the submit path
SubmitCase is where the two stores meet, so it is where most of the care went. The sequence in workflow_command_service.go:
- Take a PostgreSQL advisory lock keyed by tenant and case id, on a dedicated connection, so retries on different replicas serialize.
- Reload the case. If it is no longer
DRAFT, return the stored result when the idempotency key and request hash match, orErrIdempotencyConflictwhen only the key matches. - Refuse to start if the case type has no active registry steps (the inbox projector could never find its human tasks).
- Build the process variables: service-owned keys (
caseId,tenantId,caseType,primaryObjectId,actorUserId, ...) are derived from the stored case and re-asserted last, and client-supplied values for those keys are dropped and logged. - Start the instance, then commit the
DRAFTtoSUBMITTEDtransition.
// apps/workflow-service/internal/service/workflow_command_service.go
processKey, err := s.zeebeSvc.StartWorkflow(ctx, *bc.BpmnProcessID, variables)
if err != nil {
return nil, fmt.Errorf("start workflow: %w", err)
}
updated, err := s.caseRepo.SubmitCase(ctx, id, in.Actor, processKey, in.IdempotencyKey, requestHash)
if err != nil {
return nil, fmt.Errorf("workflow started but case submit failed: %w", err)
}
Case creation has its own idempotency: unique indexes business_cases_tenant_idempotency_uq and business_cases_tenant_submit_idempotency_uq (both on (tenant_id, key)), plus a stored request hash so a reused key with a different payload is a conflict, not a silent replay (20260825161000_add_idempotency_request_hash.sql).
Callers pick deterministic keys. loan-service builds lnm-<kind>-<id> for create and lnm-<kind>-<id>-submit for submit (adjustment_service.go). The batch disbursement path goes further: it writes a row to its own lnm_disbursement_workflow_outbox in the same transaction as the batch, and a relay calls CreateCase/SubmitCase later with keys derived from the batch id, up to disbursementWorkflowMaxAttempts = 8. The code comment gives the reason: "API latency does not depend on Zeebe availability".
The gap. Zeebe is called before the Postgres commit and there is no compensation. If StartWorkflow succeeds and SubmitCase then fails (database down, process killed), the case is still DRAFT with no process key, while an instance is running in Zeebe. The retry starts a second instance. CancelWorkflow is only called from the HTTP cancel and operate handlers, not from this path. I found no test for this window; workflow_idempotency_test.go only checks that the request hash is deterministic. The advisory lock closes concurrent retries, not this crash.
! careful
StartWorkflow uses LatestVersion(); business_cases.bpmn_version is stored but not sent to Zeebe on start.
Job workers and human tasks
Service tasks. Workers run inside workflow-service (cmd/workflow-service/main.go opens one Zeebe job worker per job type) and are pure transport: the handler reads variables, calls the domain over gRPC with a tenant context built from the job's variables, then completes or fails the job. The loan worker's execute step:
// apps/workflow-service/internal/worker/loan_workers.go
if err := w.loanClient.ResolveAdjustment(crmJobContext(job), kind, id, "APPROVE", decidedBy, note); err != nil {
w.failJob(client, job, "Loan Error: "+err.Error())
return
}
if err := w.completeJob(client, job, map[string]any{"approvalStatus": "APPROVED"}); err != nil {
return
}
w.finishCase(job, repository.CaseStatusCompleted)
Three outcomes exist. A validation problem is a BPMN error (throwValidationError, code VALIDATION_FAILED), which the process routes through a boundary event back to the maker. A transient failure calls failJob with retries - 1, and once retries hit zero Zeebe raises an incident. A malformed job fails with retries 0 immediately.
The domain side must tolerate a repeated call, because the worker can crash after the domain commit and before completeJob. For loan adjustments it does: ResolveAdjustment updates WHERE status = 'PENDING', and a zero-row update falls through to CheckAdjustmentReplay, which accepts the same decision as a replay (apps/loan-service/internal/repository/adjustment_repo.go). Other domains have to do the same; docs/crm-hrm-bpm-data-design.md makes it a stated requirement ("worker idempotent"), but I only checked the loan path.
common-maker-checker.bpmn is one shared process whose role groups come from variables (candidateGroups="=[mcMakerRole]"); a case type opts in through business_operation_types.worker_kind and workers dispatch on mcKind. migrations/canary/ holds two canary migrations: LNM_RECOVERY_V2 and batch disbursement register.
Human tasks. These are native Zeebe user tasks (91 zeebe:userTask elements). The README is explicit about the constraint: Camunda 8.5 gateway REST cannot list user tasks, and Tasklist is disabled in arda-infra/platform/helm/zeebe/values.yaml, so UserTaskProjector polls the Zeebe Elasticsearch exporter every 2 seconds for up to 100 active cases and upserts workflow_tasks rows. Completion goes through the gateway REST API, but only after the decision is written down:
// apps/workflow-service/internal/handler/workflow_handler.go
// Record the decision durably before any engine/domain side effect so the
// command can be reconciled if the engine call fails midway.
if err := h.recordTaskDecision(r, jobKey, processInstanceKey, elementID, actor, comment, variables); err != nil { ... }
Authorization happens before that: the task must be in the inbox or verifiable against the claimed process, the actor must be assigned or in the candidate role, and maker-checker separation is enforced. Approve and reject are confirmed when the terminal worker calls FinishCase; submit and return decisions are applied by DecisionDispatcher adapters and only then marked APPLIED.
Failure modes I could trace
| Failure | What the code does | Gap |
|---|---|---|
| Zeebe unreachable at startup | main.go exits with status 1 if the client cannot be created, and again if any built-in BPMN fails to deploy (all 44 are redeployed on every start) | The pod crash-loops until the gateway is back |
| Zeebe down at submit | StartWorkflow errors, nothing is written, case stays DRAFT, retry is safe | The gRPC server maps every non-NotFound error to FailedPrecondition, so callers cannot tell "engine down" from "bad state"; CreateCase errors all map to InvalidArgument |
Crash after StartWorkflow, before SubmitCase commit | See previous section | Orphan instance, second instance on retry |
Worker crashes before completeJob | The job is re-delivered to a worker; the domain call must be replay-safe | Only verified for loan adjustments. The repo sets no job timeout or worker options, so library defaults apply |
completeJob succeeds, FinishCase fails | The error is only logged by CaseProjection | The case stays non-terminal in Postgres while Zeebe has ended the instance. I found no sweeper that repairs it |
| Human clicks complete twice | authorizeUserTaskComplete rejects a task whose work item is already COMPLETED; the decision key defaults to task-<jobKey>-<decision> (or the Idempotency-Key header) and is UNIQUE, so a retry after an engine failure reuses the recorded decision | If the engine call succeeds and the DB write after it fails, the work item stays open until the reconciler closes it |
Engine completes a task but workflow_tasks stays open | WorkflowReconciler (every 2 minutes) closes the row when the engine reports COMPLETED/CANCELED, or when the key is absent while another task of that process is CREATED; otherwise it only touches the row | Depends on the Elasticsearch exporter being current |
| NATS down | Events accumulate in workflow_outbox; relay retries every 2 seconds; dedupe_key is UNIQUE and the row id goes out as Nats-Msg-Id | No attempt cap or dead-letter: publish_attempts and last_error are recorded, and the row is selected again forever |
The workflow outbox is a normal transactional outbox. FinishCase updates business_cases, closes workflow_tasks, marks decisions APPLIED and inserts the event row in one transaction. The relay then claims up to 50 rows with FOR UPDATE SKIP LOCKED, so several replicas do not publish the same row.
SLA warnings do not use Zeebe timers. The BPMN corpus has no timerEventDefinition. Instead a goroutine in main.go calls EmitSLANotifications every minute and enqueues workflow.task.sla_warning / workflow.task.overdue events with the dedupe key task:<id>:sla:<milestone>, for milestones 50, 90 and 100 percent of the created-to-due interval by default. Escalation is a notification; no process path changes when a deadline passes.
Tenant scoping, coupling that remains, and what I would check next
Tenant scoping is applied in Postgres, not in the engine. I found no use of Zeebe multi-tenancy: the Zeebe commands carry no tenant id, and tenantId is an ordinary process variable. Isolation comes from three places:
- Every
CaseRepositoryread and write starts withverifiedTenant(ctx), taken from the gRPC/HTTP metadata, and returnsErrTenantScopeRequiredwhen it is empty.CreateCasealso rejects a request whosetenant_iddiffers from the verified one. - The unique indexes on case code and idempotency keys are per tenant, as above.
- When a worker calls a domain service, it rebuilds the tenant context from the job's variables (
crmJobContext). The comment there is the honest summary: there is no inbound request, so scope "must come from versioned process variables; missing tenant/org intentionally fails closed". That is why client input can never settenantId: it is in the reserved-key list and is overwritten from the stored case.
The consequence is that one Zeebe cluster (clusterSize: 1, partitionCount: 1 in the Helm values) is shared by all tenants. Only the facade and the network boundary keep a caller away from other tenants' variables.
What still couples the services to Zeebe, even though none of them imports its client:
- Variable names.
adjustmentId,batchId,decision,approvalStatusare contracts between a domain service and a BPMN file. The domain service builds the map; the BPMN reads it. A typo fails at run time, in a gateway expression. - Job topics. Every BPMN service task needs a registered worker.
TestEveryServiceTaskTopicHasRegisteredWorkerchecks this at build time againstRegisteredJobTopics, because the failure mode is a case parked forever at its last step. - Step codes. The inbox and the allowed actions are derived from the BPMN at startup (
SeedRegistry); startup fails if an ACTIVE case type has no human steps. - Camunda 8.5 specifics. The repo pins
github.com/camunda/zeebe/clients/go/v8 v8.5.5inapps/workflow-service/go.mod, andarda-infra/platform/README.mdsays "Camunda Platform chart 10.11.3 / Zeebe 8.5.x". The Helm values do not pin an image tag. Because Tasklist is off and 8.5 gateway REST has no user-task search, the Elasticsearch exporter is a hard dependency (ZEEBE_ES_URL). Changing engine versions means revisiting the projector, not just the client.
Known gaps from the repo itself. The README lists "No runtime task claim/complete/reassign facade yet" and "No incident retry/suspend/resume APIs yet"; the code has since gained incident resolve and job retry handlers (operate.go), so that list is partly stale, and called-instance data in the monitor is still pending.
What I would check next. Reproduce the StartWorkflow/SubmitCase crash window with a fault-injection test and decide between a compensating CancelWorkflow and a pending-start row. Add a repair path for FinishCase failures. Set a poison limit on workflow_outbox. Distinguish engine-unavailable from precondition errors in the gRPC status codes.
Verified repository evidence
workflow · direct
Only workflow-service talks to Zeebe 8.5; other services reach it through a two-RPC WorkflowCommandService (CreateCase, SubmitCase), with idempotent submit and a workflow outbox.
↳ Zeebe Go client v8.5.5 is imported only by workflow-service; case create/submit are idempotent per tenant; SLA milestones come from a Postgres scan.
arda-labs/arda-be ↗
Key Engineering Takeaways
- The boundary removes the engine client from fifteen codebases, not the engine from their design: variables, topics and step codes still couple them.
- The riskiest step is the one that touches both stores: StartWorkflow runs before the Postgres commit and has no compensation in the repo.
- Most recovery is polling and reconciliation (projector, reconciler, sweeper, outbox relays), each with a stated gap.