# Message Queues: At-Least-Once Is the Only Honest Promise

> Every queue advertises delivery guarantees; every distributed-systems engineer eventually learns the ack-crash window makes duplicates inevitable. Why exactly-once is really exactly-once *processing*, and the idempotency patterns that make it true.

URL: https://harshith.in/blog/message-queue-delivery-semantics/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: distributed-systems, systems

Somewhere in every message-driven architecture sits a consumer that
charges a credit card, and somewhere in every incident history is the
day it charged one twice. The root cause is never exotic — it's the
same two-line proof, rediscovered: the consumer processed the message
and crashed *before the ack left*. The broker, seeing no ack, must
choose: redeliver (duplicate) or don't (loss). There is no third
option — that window is physics, not vendor negligence — and every
"delivery guarantee" in every queue's marketing is a policy about
which side of it to fall on.

## The Menu, Honestly Labeled

**At-most-once** — ack/commit *before* processing. Crash mid-work and
the message is gone. Correct choice for data where stale beats
duplicate (metrics, telemetry, cache invalidation hints), and
absolutely nowhere money lives.

**At-least-once** — ack *after* processing. Crash and the broker
redelivers; nothing is lost, duplicates are now your application's
problem. This is the default posture of SQS, RabbitMQ, Kafka
consumers, and reality.

**Exactly-once** — the phrase means something real, but read the fine
print: it's exactly-once *processing within a closed system*, not
delivery. Kafka's version is honest engineering: idempotent producers
(sequence numbers dedupe broker-side retries) + transactions that
atomically commit output messages *and* consumer offsets — so a
consume-transform-produce pipeline that lives entirely inside Kafka
replays without visible duplicates. The moment your consumer touches
anything *outside* (an API call, an email, a database without
transactional tie-in), the ack-crash window reopens at that boundary.
No queue can promise your side effect and its ack are atomic.

## Making Duplicates Boring: Idempotency

Since redelivery is inevitable, the mature design goal is *duplicates
that don't matter*:

- **Natural idempotency** where possible: `SET balance = 700` replays
  harmlessly; `ADD 100` doesn't. Upserts keyed on business identity
  beat inserts.
- **Idempotency keys** everywhere else: a unique ID per logical
  operation (message ID, or better, producer-assigned business key),
  checked-and-recorded **in the same transaction as the effect**:

```sql
BEGIN;
INSERT INTO processed(msg_id) VALUES ($1);  -- unique index: dup ⇒ abort
UPDATE accounts SET balance = balance - $2 WHERE id = $3;
COMMIT;
```

  The transaction is the whole trick — check-then-act without it just
  narrows the window. (Stripe-style APIs push the same pattern across
  the network: the `Idempotency-Key` header is at-least-once made safe
  at an HTTP boundary.)
- **The outbox pattern** for the producing side's mirror-image
  problem ("update the DB *and* publish an event, atomically"):
  write the event into an outbox table in the business transaction;
  a relay (or CDC/Debezium tailing the WAL) publishes it. Dual-write
  without the dual-write bug.

Ordering deserves its own honesty check: queues promise order only
per-partition/per-FIFO-group, retries and DLQ detours reorder
regardless, and consumers scale by partition count — so encode
*causality* in the data (versions, sequence numbers per entity) rather
than trusting arrival order, and pick partition keys (entity ID) so
the ordering you get is the ordering you need. And redelivery needs a
terminus: max-receive-count into a **dead-letter queue**, with the
two operational rules people learn expensively — alert on DLQ depth
(a quiet DLQ is data loss on a timer), and mind visibility-timeout vs
processing-time (work that takes 31 s against a 30 s timeout is a
duplicate *generator*, the "poison pill that isn't" classic).

## Takeaways

- The ack-crash window makes loss-or-duplicate a forced choice;
  at-least-once + idempotent consumers is the industry's answer.
- "Exactly-once" = exactly-once processing inside one system's
  transaction boundary (Kafka txns); external side effects reopen the
  window at the edge.
- Idempotency keys recorded in the same transaction as the effect;
  outbox/CDC for atomic write-and-publish.
- Order is per-partition at best and survives neither retries nor
  DLQs — put sequence/causality in the payload, choose partition keys
  deliberately.
- DLQs with alerts, visibility timeouts sized above real processing
  time — the boring settings that prevent the exciting incidents.