The request timed out. Did the work happen?

The ambiguous outcome

Imagine an order API that writes a database row, publishes an event, and then returns a response. The client sees a timeout. Did the database commit? Did the broker receive the event? Did the response disappear on its way back? A timeout answers none of those questions.

Retries are necessary because networks fail. The harder question is how to make a repeated request express the same intent. Our starting point is a stable request identity, a clearly defined transaction boundary, and a durable record of the work still owed.

The AWS Builders’ Library discussion of idempotent APIs explains why callers need retry behaviour that preserves their intent. The design below is an illustrative order flow, not a report of a client deployment.

Make intent identifiable

Have the caller create an idempotency key before the first attempt. Scope the key to a tenant and operation, and persist a fingerprint of the normalized request. Reusing a key with a different payload should return a conflict. Otherwise a typo in the client can make an old operation appear to succeed with new arguments.

CREATE TABLE request_receipts (
  tenant_id uuid NOT NULL,
  operation text NOT NULL,
  request_key text NOT NULL,
  payload_hash text NOT NULL,
  resource_id uuid NOT NULL,
  response_body jsonb NOT NULL,
  PRIMARY KEY (tenant_id, operation, request_key)
);

The uniqueness constraint is the concurrency primitive. Two application servers can receive the same request simultaneously, so a “read, then insert” check alone is insufficient. Within the business transaction, insert the receipt and resource, or handle the unique-key conflict and return the stored result after verifying the payload hash.

Decide how long a key remains valid. If receipts expire before clients stop retrying, an old request can create a new effect. Key retention is part of the API contract and operational policy.

One transaction, two records

Writing to a database and publishing to a broker are independent operations. If publishing fails after commit, the event is missing. Publishing first creates the inverse problem: a consumer may see an event for a transaction that rolls back.

A transactional outbox stores the business change and the pending event in the same database transaction. A separate dispatcher delivers committed outbox rows to the broker. This moves the unreliable boundary into a recoverable background process.

BEGIN;
  -- Application supplies stable order_id and event_id.
  INSERT INTO orders (id, tenant_id, status)
  VALUES ($1, $2, 'placed');

  INSERT INTO outbox (id, tenant_id, kind, payload)
  VALUES ($3, $2, 'order.placed', $4::jsonb);

  -- Persist request receipt in this same transaction.
COMMIT;

The outbox row should carry an event ID, aggregate ID, schema version, creation time, and enough payload to process the intended event. Avoid requiring a consumer to reconstruct an old event from mutable current state unless that is explicitly the desired contract.

Delivery can still repeat

A dispatcher may publish an event successfully and crash before marking it delivered. On recovery it will publish the event again. The outbox closes the lost-event gap at the database boundary; it does not make the entire distributed workflow exactly once.

Make the consumer record the event ID and apply its local business effect in one transaction. A unique constraint on the consumed event ID prevents a repeated delivery from applying the local change twice. If processing calls another provider, propagate a stable business key or introduce a further durable dispatch record.

Parallel dispatch also needs a claim or lease strategy, bounded retries, and an ordering decision. Order by aggregate when later events depend on earlier state. A global ordering guarantee is often more costly than the business needs.

Every boundary should make its uncertainty visible enough to recover from.

Operate the pending work

Track the age of the oldest undelivered event, delivery attempts, lease expiry, and a dead-letter count. Queue length alone can look healthy while one important event remains stuck. Alert on the delay a business can tolerate, not only on infrastructure errors.

• Crash after the database commit and confirm dispatch resumes.
• Crash after broker acknowledgement and confirm duplicate delivery is harmless.
• Send the same key concurrently and verify one business resource exists.
• Reuse a key with a different payload and expect a conflict.
• Exercise poison events without blocking unrelated aggregates.

The result is a system with an explicit recovery story. That is more useful than a success path that works beautifully until one response disappears.