Queue architecture: acknowledgement, idempotency, retries and dead letters

A queue lets one application accept work without waiting for another to finish it. The separation is useful when the target is temporarily saturated, when a process takes longer than an API response allows or when tasks must be distributed among several consumers.

Pending work, however, does not disappear by moving it to an intermediary. Before choosing a product, three things must be defined: what it means to complete a task, how long it can wait and who steps in when it is not processed. This article develops the design principles that follow from those definitions, with references to RabbitMQ and OCI Queue.

When a queue is the right tool

An illustrative case: generating a document after a request is registered. The application stores the request and returns a reference to check its status, while a consumer prepares the document. The user must be able to distinguish «request accepted» from «document ready»; a successful submission does not prove that the work is finished.

A work queue distributes tasks among consumers that compete to process them. The official RabbitMQ work queues tutorial describes the pattern and the recovery of unacknowledged deliveries. If several applications need to keep and re-read the same event history, the right model is different: an event log with retention.

Asynchrony is not appropriate when the operation needs an immediate answer from the target to continue. If an authorization must be validated before a transaction is accepted, deferring that step changes the service contract. As a criterion, the queue is reserved for work that admits an explicit intermediate state and a recovery procedure.

Acknowledgement after the persisted effect

A message's path has two distinct boundaries: delivery to the intermediary and execution of its effect in the application. In RabbitMQ, publisher confirms and consumer acknowledgements cover each boundary separately. The broker confirming a publication does not mean the consumer has executed the business operation.

The consumer should acknowledge once it has persisted the result that marks the task as done. If it acknowledges earlier, a later crash leaves work lost from the business point of view. If it persists first and crashes before acknowledging, it will receive the message again and must recognize that the work was already done. That is why delivery is, in practice, «at least once».

OCI Queue uses a different interface with the same consequence: the consumer receives the message, processes it and deletes it. During the visibility timeout the message is hidden from other consumers; if it expires without deletion, it may be delivered again. Processing must fit within that window, or the consumer must extend it in time.

Idempotency in the business operation

An operation is idempotent if repeating it with the same intent produces no additional effect. The usual technique combines a stable event identifier with a register of completed operations. The identifier must survive retries: if a new one is generated on each publication, it is no longer possible to recognize the same work.

The check and the change must be coordinated. When both live in the same database, a single transaction records the identifier and applies the change, with a unique constraint that prevents recording the same operation twice. A standalone prior query is not enough: two consumers can query at the same time, find nothing and both apply the effect.

When the effect happens in an external API, the local transaction does not cover it. If the target documents an idempotency key, it should be used; otherwise a reconciliation mechanism is needed to find out what happened. Blindly repeating a call whose response was lost can duplicate the effect even if the queue works correctly.

A third point remains: the gap between saving the initial change and publishing the message. The transactional outbox pattern saves the change and a pending-publication record in the same local transaction, and a later process publishes that record. Publication can still repeat if a confirmation is lost, so the pattern complements, rather than replaces, consumer idempotency.

Retries with a defined delay

Before returning a message to the queue, the error must be classified. A temporarily unavailable dependency justifies another attempt; a document missing mandatory fields needs correction. Treating both the same way consumes capacity and delays tasks that could complete.

The retry policy combines an increasing delay with a limit on attempts or total time. Adding random variation (jitter) to the delay keeps all consumers from calling the target at the same instant. As a design criterion, the mechanism must preserve pending work even if the process restarts.

In RabbitMQ, returning a delivery with requeue and no delay policy can cause a fast redelivery loop. Alternatives depend on the version and queue type: the current quorum queues documentation describes a delayed retry configurable through policies starting with RabbitMQ 4.3. On versions without that feature, a delay topology or a persistent scheduler is evaluated, with prior testing.

Topologies based on TTL and dead lettering need a specific review. Message expiration is not an exact execution time, and combining different expirations can alter ordering and delay. It is advisable to record the planned next attempt and measure when it actually happens.

Dead-letter queue and its owner

The dead-letter queue (DLQ) isolates work that could not complete under the normal policy. In OCI Queue, the service moves to the DLQ the messages that exceed the configured maximum of failed deliveries. The DLQ also has retention: the message must be investigated or recovered before it expires.

In RabbitMQ, a dead letter exchange routes messages to configured destinations, and the receiving queue must be declared and bound explicitly. The dead lettering documentation warns that the transfer can fail and that the default behavior does not guarantee a safe transfer; the at-least-once mode of quorum queues requires explicit configuration.

A DLQ without an owner is a store of lost work. Alerts need an assigned person or team, and each message must keep the failure reason and its correlation identifier. Reprocessing keeps the event's original identity when it represents the same operation; if the content is corrected, the record shows who authorized it and what changed.

Testing at the process boundaries

Useful tests inject failures at the boundaries: when the effect was saved but the acknowledgement did not arrive, and when a dependency stops responding. The acceptance criterion is the business result, not only the intermediary's counters.

• Consumer interrupted before and after persisting the result, with no duplicated effect.
• Publication with an uncertain outcome, repeated with the same identifier.
• Permanent error that reaches the DLQ and raises an alert with an owner.
• Delays that survive a restart and reprocessing that preserves traceability.
• Alerts on the age of pending work, in addition to queue depth.

Xterra Engine designs messaging architectures with these failure tests included from the first delivery.