Queues and Background Jobs

Build the mental model

Slow work — email, video processing, reports, AI calls — should not block a web request. Move it to a queue and let a separate worker run it.

Producer — the web server that creates a job and drops it onto the queue.
Queue — holds pending jobs until a worker is free.
Worker — a separate process that picks up jobs and runs them independently.

Cron jobs run on a schedule (backups, digests, cleanup) rather than in response to a request. Webhooks flip the direction: an external service calls your app.

Webhooks arrive more than once — plan for it

Verify a webhook's signature, expect retries from network timeouts, and make handling idempotent. Skipping this is how a duplicate webhook becomes a duplicate charge.

Idempotency means processing the same request twice gives the same result as once — track handled IDs and skip anything already seen.

Queue: A holding area for pending jobs, waiting for a worker to pick them up and process them.

Worker: A separate process that pulls jobs off a queue and runs them independently of the original request.

Idempotency: The property that processing the same request more than once produces the same result as processing it once.

Webhook: An HTTP callback an external service sends to your app when an event happens on its side.

text

QUEUE, WORKER, AND WEBHOOK FLOW
-------------------------------
QUEUE, WORKER, AND WEBHOOK FLOW
----------------------------------

  [ WEB REQUEST ]                 [ EXTERNAL SERVICE ]
        |                                |
        | producer enqueues job          | webhook call
        v                                v
     [ QUEUE ]                   [ YOUR APP ENDPOINT ]
        |                                |
        v                          verify signature
   [ WORKER ]                      check idempotency key
        |                                |
        v                        already seen? -> skip
   [  RESULT  ]                  new? -> process once

  CRON: same worker pattern, triggered by a clock, not a request.

Connect it to a real scenario

Keep a set of processed request IDs. Before doing real work, check whether the incoming ID is already in that set.

First call with a fresh ID processes normally. A repeat call with the same ID gets skipped. A different ID processes again.

In production, back this with real storage

A real idempotency guard needs a database table or key-value store — not an in-memory Set — so it survives restarts and works across multiple instances.

Slow or non-critical-path work belongs in a queue processed by a worker, not inside the request-response cycle.
Cron jobs are timer-triggered background work; webhooks are externally-triggered — both need signature verification and idempotency.
Idempotency means handling the same request twice yields the same result as once — track handled IDs and skip duplicates.

Try the working example

javascript

function makeIdempotencyGuard() {
  const processed = new Set();
  return function handleRequest(requestId) {
    if (processed.has(requestId)) {
      return { action: "skipped", reason: "already processed", requestId };
    }
    processed.add(requestId);
    return { action: "processed", requestId };
  };
}

const handle = makeIdempotencyGuard();

console.log(JSON.stringify(handle("charge-9f1a")));
console.log(JSON.stringify(handle("charge-9f1a")));
console.log(JSON.stringify(handle("charge-7bd2")));

You should see

{"action":"processed","requestId":"charge-9f1a"}
{"action":"skipped","reason":"already processed","requestId":"charge-9f1a"}
{"action":"processed","requestId":"charge-7bd2"}

5-minute try-it

Extend makeIdempotencyGuard so each processed ID also stores a timestamp, and add a second function that reports how many unique requests were processed versus how many duplicates were skipped.

One important caution

Doing slow work (email, video processing) directly inside a request handler, making users wait or triggering platform request timeouts.

Trusting a webhook arrives exactly once — a retried webhook without an idempotency check can duplicate a charge or an order.

Stripe docs — Idempotent requests — Cloud & Deployment