Project 5 — Stream Job Queue

What you'll walk away with

Explain the core ideas behind Project 5 — Stream Job Queue
Run the sample Redis command or code and verify its output
Apply the technique correctly to the Tutorial Platform and production scenarios

Build the mental model

A Stream job queue should be designed for at-least-once delivery. Send small payloads with immutable IDs, fetch durable details from the database, and protect side effects with idempotency keys. Define retry counts, exponential backoff, poison-message dead-lettering, pending reclaim, and retention.

Connect it to a real scenario

Build an email worker group using blocking reads, bounded concurrency, and ACK after successful delivery. Increment failure metadata; after the threshold, append the original ID and reason to a dead-letter stream, then ACK. On shutdown, stop new reads and finish or leave in-flight jobs pending.

Pair at-least-once delivery with idempotency
Reclaim pending jobs with bounded retries
Include DLQ, retention, and observability

Try the working example

typescript

const jobs = await redis.xReadGroup('emailers', workerId,
  [{ key: 'stream:email', id: '>' }], { COUNT: 10, BLOCK: 5000 });
for (const job of jobs ?? []) {
  await withIdempotency(job.id, () => sendEmail(job.message));
  await redis.xAck('stream:email', 'emailers', job.id);
}

You should see

You get crash-recoverable workers protected against duplicate side effects.

5-minute try-it

Design a three-retry DLQ flow and an operator replay checklist.

One important caution

Aggressive trimming without considering pending entries can remove message data needed for recovery.

Redis — Streams — Redis

Easy traps

Aggressive trimming without considering pending entries can remove message data needed for recovery.
Validate sample commands on a local or test instance with recoverable data before applying them to production Redis.

Exercise

Design a three-retry DLQ flow and an operator replay checklist.

You'll know it worked when: You get crash-recoverable workers protected against duplicate side effects.