At a glance
6
JetStream relays
finance, workflow, loan, statistical, notification, media
7
Outbox tables
Six drain to NATS, one drains to a gRPC call
2 s
Fixed poll interval
Hard-coded in every relay; no measured latency
2
Relay tests
Both in finance-service; the other relays have none
✎ Key Insights in 30 Seconds
Decision record: A decision record read from the code: how arda-be writes events in the same Postgres transaction as state, how six services relay them to one JetStream stream, where at-least-once ends, and where the implementations disagree.
Problem Context
A handler that commits state and then publishes to a broker can lose or invent events when it dies between the two steps. Arda writes the event as a row in the same Postgres transaction and lets a separate loop move it to NATS JetStream, which turns the problem into delivery that is at-least-once and has to be deduplicated by the reader.
Transactional Outbox on NATS JetStream: What Arda's Six Relays Actually Guarantee
Architecture & Design
Each emitting service owns an outbox table in its schema and inserts into it inside the business transaction (statistical-service being the exception). A relay polls every 2 seconds, claims rows with SKIP LOCKED or a lease, publishes to the shared ARDA_EVENTS stream with the row ID as Nats-Msg-Id, and marks the row after the PubAck. Notification-service consumes through durable queue subscriptions with manual ack, a noti_event_inbox table for dedupe, and an arda.dlq.* subject after the tenth delivery.
Key Technical Decisions
- 1Event row in the state transaction: no dual write, except statistical-service, where a crash between two statements can drop an alert event.
- 2Row ID as Nats-Msg-Id: a republish is suppressed only inside JetStream's duplicate window, so consumers must still dedupe.
- 3Per-service relay copies: each service can tune retries, but four of six retry for ever and two cap attempts, so behaviour has drifted.
- 4Best-effort ORDER BY created_at: no ordering guarantee across retries or replicas, and ADR-007 lists ordering as an open decision.
- 5Inbox table with advisory lock in the consumer: redelivery is safe for the handlers that use ProcessEventOnce, not for the statistical-breach consumer.
The problem: a commit and a publish are two writes
A handler that commits to Postgres and then calls the broker has two failure windows. If the process dies between the two, the state changed and nobody was told. If the order is reversed and the commit fails, consumers react to something that never happened. The outbox pattern removes the second write from the request path: the event is a row in the same database, inserted in the same transaction as the state change, and a separate loop moves it to the broker.
In arda-be that loop exists six times, once per service that emits to NATS. I read all six, plus the one shared library, libs/go/arda-events. This note records what the code guarantees, which is at-least-once delivery into a JetStream stream and at-least-once handling by consumers that dedupe. It does not provide exactly-once, and the repo's own ADR says part of the consumer side is still unproven (more on that below).
Two claims in my older version of this article were wrong and I removed them. The event ID is not a UUIDv7 produced by the envelope: newID() in libs/go/arda-events/events.go is 18 random bytes, base64url-encoded. And I found no processed_events table anywhere in the repo; the consumer-side table is noti_event_inbox. I also had a "< 1 ms" figure for JetStream. Nothing in the repo measures it (the only file under perf/ is api-load.js, an HTTP load script), so it is gone.
The write side: what is really atomic
Each service owns its outbox table in its own schema. The insert takes the caller's *sql.Tx, so it commits or rolls back with the state change. Finance does this inside the posting transaction, after journal lines and balances:
// apps/finance-service/internal/service/posting_service.go
if err := s.postLinesToBalances(ctx, tx, tenantID, resolved); err != nil {
return nil, err
}
if err := s.repo.InsertOutbox(ctx, tx, tenantID, entryID, outboxPayload(entryID, req)); err != nil {
return nil, fmt.Errorf("enqueue outbox: %w", err)
}
if err := tx.Commit(); err != nil {
Workflow, loan and statistical add a dedupe_key with a unique constraint, so a retried command cannot enqueue the same event twice:
// apps/workflow-service/internal/repository/outbox.go
_, err = tx.ExecContext(ctx, `
INSERT INTO workflow_outbox (id, subject, event_code, dedupe_key, payload)
VALUES ($1, $2, $3, $4, $5::jsonb)
ON CONFLICT (dedupe_key) DO NOTHING`, id, subject, eventCode, dedupeKey, encoded)
The keys are deterministic business facts such as task:<id>:assigned:<jobKey> or disbursement:<id>:approved. In loan the outbox row's primary key is the dedupe key.
The exception is statistical-service. EnqueueAlertEvent runs on r.db, not on a transaction, after UpsertAlert has already committed, and the caller only logs a failure (slog.Warn("indicator alert enqueue failed")). A crash between the two statements loses that event. The migration comment says the dedupe key makes enqueue idempotent across COB re-runs, so a re-run would enqueue it again, but that is a repair by re-running, not the atomicity the pattern is named for.
! careful
Only five of the six emitters write the event in the state transaction. Statistical-service enqueues in a separate statement and tolerates the loss.
The relay: claim, publish, mark
Every relay polls every 2 seconds (a constant in each file), publishes to the JetStream stream ARDA_EVENTS (subjects arda.>, file storage, created on first use by whichever service starts first), and sets the outbox row ID as the Nats-Msg-Id header. Finance's version:
// apps/finance-service/internal/service/outbox_relay.go
rows, err := tx.QueryContext(ctx, `
SELECT id, payload FROM fin_outbox
WHERE published_at IS NULL
ORDER BY created_at LIMIT 50
FOR UPDATE SKIP LOCKED`)
// ... scan batch, close rows ...
for _, p := range batch {
msg := &nats.Msg{Subject: ardaevents.SubjectFinanceTransactionPosted, Data: p.payload, Header: nats.Header{}}
msg.Header.Set(nats.MsgIdHdr, p.id)
if _, err := r.js.PublishMsg(msg, nats.Context(ctx)); err != nil {
// UPDATE fin_outbox SET publish_attempts = publish_attempts + 1, last_error = $2
continue
}
// UPDATE fin_outbox SET published_at = now() WHERE id = $1
}
tx.Commit()
The row is marked published only after PublishMsg returns a PubAck. SKIP LOCKED lets replicas split the pending rows; the finance test starts a second relay while the first is blocked inside the publisher and asserts one publish call in total (TestOutboxRelayConcurrentReplicasPublishOnePendingRow).
The six implementations are not the same:
| Service | Table | Claim | Retry limit and backoff | After the limit |
|---|---|---|---|---|
| finance | fin_outbox | FOR UPDATE SKIP LOCKED, lock held across publishes | none, retried every tick | stays pending, publish_attempts grows |
| workflow | workflow_outbox | same as finance | none | same |
| loan | loan_outbox | same as finance | none | same |
| statistical | rpt_outbox_events | same as finance | none | same |
| notification | noti_outbox | status publishing plus 30 s locked_until lease | 10 attempts, delay 30 s x attempts | dead_lettered plus a row in notification_outbox_dlq |
| media | media_outbox_events | one UPDATE ... RETURNING with a 30 s lease | 20 attempts, delay 30 s x attempts capped at 3600 s | status failed, no replay path found |
Loan has a seventh table, lnm_disbursement_workflow_outbox, that drains to a gRPC CreateCase call instead of NATS. It is the most complete one: status PENDING/PROCESSING/DELIVERED/DEAD, a one-minute lease, 8 attempts, exponential backoff capped at 5 minutes, and on DEAD the batch moves to SUBMIT_FAILED and held contract reservations are released in the same transaction. A test walks it through exhaustion and resubmission (TestDisbursementOutboxExhaustionCanBeResubmitted).
Failure modes
Crash after commit, before publish. The row is pending; the next relay tick finds it. This is the case the pattern exists for.
Crash after the PubAck, before the row is marked. In the four SKIP LOCKED relays the mark and the claim share one transaction, so the rollback leaves the row pending and the event is published again. The stable Nats-Msg-Id is the only protection, and JetStream honours it only inside its duplicate-tracking window. No StreamConfig in the repo sets Duplicates, so the server default applies (I am relying on the NATS documentation for that default, not on the repo). Past the window, a republish is stored twice. The finance test (TestOutboxRelayRetryAfterPublishFailureUsesStableMessageID) proves the header does not change between attempts, but it uses a fake publisher, so it shows the contract and not JetStream's behaviour. Notification and media have the same gap through their lease: a worker that dies after publishing leaves publishing until the lease expires, then the row is claimed and sent again.
Poison rows. For finance, workflow, loan and statistical, a row that always fails is retried every two seconds for ever, with only publish_attempts and last_error changing. It does not block other rows, because the loop continues past it, but nothing alerts and nothing parks it. Notification and media park after 10 and 20 attempts.
Ordering. ORDER BY created_at is a preference, not a guarantee. A failed row is skipped while later rows go out; two replicas claim disjoint rows concurrently; notification's retry delay is 30 s times the attempt count, so a retried event arrives after its successors. ADR-007 in docs/refactor-program/phase-0/adr-register.md lists "stream naming/partition/order" under open decisions. Consumers have to treat events as unordered.
Tenant isolation. The outbox is where a missing tenant would first leak into a shared stream (arda.> carries everyone's events). The tables differ: fin_outbox, rpt_outbox_events, media_outbox_events and noti_outbox have a tenant_id column; workflow_outbox and loan_outbox do not, and carry the tenant only inside the JSON payload. Notification's migration 20260825167000_enforce_outbox_tenant_scope.sql drops the column default and adds a CHECK rejecting empty and default, created NOT VALID; 20260826110000_backfill_legacy_tenant.sql then assigns old rows to one fixed tenant UUID and runs VALIDATE CONSTRAINT. The repo's progress notes say the finance and platform checks are still NOT VALID pending an operator-approved assignment.
✎ note
Delivery to the stream is at-least-once, and the dedupe window is the only guard against duplicates created by the relay itself. Everything past that guard is the consumer's job.
The consumer side: dedupe and dead letters
Only notification-service consumes these subjects in this repo. Its shared Consumer binds a durable queue subscription with manual ack:
// apps/notification-service/internal/events/consumer.go
sub, err := c.js.QueueSubscribeSync(c.subject, c.durable,
nats.Durable(c.durable),
nats.ManualAck(),
nats.AckWait(30*time.Second),
nats.MaxDeliver(10),
nats.BindStream(c.stream),
)
// in the loop:
if metadata.NumDelivered >= consumerMaxDeliver {
if err := c.publishDLQ(ctx, msg); err != nil { continue } // not acked, stays redeliverable
msg.AckSync(...)
continue
}
if err := handler(ctx, msg); err != nil { msg.NakWithDelay(consumerAckWait); continue }
msg.AckSync(nats.Context(ctx))
Because the DLQ check runs before the handler, a message gets at most nine handler invocations; the tenth delivery goes straight to arda.dlq.<subject> with a Nats-Msg-Id of arda-dlq:<source id> so that an interrupted ack does not duplicate the dead letter. I found no consumer of arda.dlq.* in the repo: dead letters are stored and nothing reads them yet.
Idempotency is a table, not a promise. ProcessEventOnce takes a transaction-scoped advisory lock on consumer_name:event_id, inserts into noti_event_inbox (primary key (consumer_name, event_id)), runs the handler in that same transaction, and sets processed_at. A redelivered, already-processed event hits the conflict clause, matches no row, and returns nil so the message is acked:
// apps/notification-service/internal/repository/notification_repository.go
tx.ExecContext(ctx, `SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, consumerName+":"+eventID)
err = tx.QueryRowContext(ctx, `
INSERT INTO noti_event_inbox (consumer_name, event_id, subject, tenant_id)
VALUES ($1,$2,$3,$4)
ON CONFLICT (consumer_name, event_id) DO UPDATE SET attempts = noti_event_inbox.attempts + 1
WHERE noti_event_inbox.processed_at IS NULL
RETURNING event_id`, ...).Scan(&claimed)
if errors.Is(err, sql.ErrNoRows) { return tx.Commit() } // already processed
The workflow and loan notification handlers use it, with the payload's event_id (the outbox dedupe key) as the event ID, so the key chosen on the write side is the one that deduplicates on the read side. The statistical-breach consumer does not: it relies on the unique (tenant_id, idempotency_key) on noti_notifications and builds the key from alert_id. Reading the code, UpsertAlert keeps one row, and so one alert_id, per rule, period and slice, while the outbox dedupe key includes the value so that "a materially different value notifies again". I did not run it, but those two keys look inconsistent: the second notification would be accepted by the outbox and then dropped as a duplicate by the consumer. It is the check I would make first.
Trade-offs, limits and what I would check next
Why this shape. The repo's integration plan (docs/refactor-program/04-data-and-service-integration.md) states the intent: state and outbox row commit in one transaction; the publisher claims rows safely, waits for the JetStream ack, then marks published; poison records move to a dead-letter path without blocking. ADR-007 is "accepted-with-follow-up": the PubAck, stable Nats-Msg-Id, bounded redelivery and DLQ/replay reference are landed, and consumer inbox dedupe and replay must still be proven. The docs do not explain why the relays were copied per service instead of moved into arda-events; the practical result is six code paths that have already drifted (the table above).
What is not done yet, from the repo's own notes:
- Replay is manual and narrow:
cmd/notification-replayreplays exactly one dead-lettered outbox row, needs an operator name and the confirmation stringREPLAY_ONE. Notification runtime replay evidence is listed as open. - Only finance's relay has tests. Workflow, loan (NATS relay), statistical, notification and media relays, and
ProcessEventOnce, have none that I could find. - The envelope in
arda-events(schema_version, tenant, trace, actor) is built only by notification-service and ai-service. Finance, workflow, loan, statistical and media publish bare JSON payloads, so schema evolution is by convention, with.v1in some subjects. arda.finance.transaction.postedis published, and I found no consumer in the repo.- Event IDs are inconsistent:
uuidv7()defaults in Postgres for finance, statistical and the loan submission table, random hex in workflow,gen_random_uuid()in notification, a dedupe string in loan. No code depends on time-ordering of IDs. - No latency or throughput figure exists. All I can say from the code is that a published event waits for at most one 2-second tick plus claim and publish time.
When I would not do this. If every event consumer could be reached in the same transaction, a plain table read would be simpler. If events must be ordered per aggregate, this outbox needs a sequence column and a single claimer per aggregate; JetStream subjects alone will not give that.
What I would check next. First the statistical key mismatch. Then a cap and a parking state for the four relays that retry forever, ideally by moving the claim logic into one shared library with the notification-style lease, so a slow publish does not hold row locks. Then a test against a real JetStream that republishes after the duplicate window, to learn what the system does when the header stops protecting it.
Verified repository evidence
events · direct
Designed reliable event-driven domain communication using NATS JetStream and the Transactional Outbox pattern to eliminate dual-write hazards.
↳ Envelope versioning (`ardaevents.Envelope[T]`), transactional outbox table writes within SQL transactions, background relay workers, and idempotent consumer deduplication.
arda-labs/arda-be ↗
Key Engineering Takeaways
- The guarantee is at-least-once into the stream plus consumer-side dedupe; it is not exactly-once.
- Five of six emitters are transactional; the sixth is repaired by re-running a job.
- The relays differ in locking, retry limits and dead-lettering, and only finance's relay has tests.
- The older claims of UUIDv7 event IDs, a processed_events table and sub-millisecond latency were not supported by the repo.