CONCEPT · 60 minutes
Design a distributed job pipeline with FastAPI, Celery, Redis, and DAG-based task scheduling — grounded in the python-orchestration-system.
The first version of my python-orchestration-system had a fatal design: when a user submitted an SDLC task, the FastAPI endpoint called the task executor directly — inline, synchronous, waiting. The endpoint held the connection open while the task ran. A single long-running task blocked a worker thread. Under load, the queue of waiting submissions backed up behind the thread pool. When a task crashed mid-execution, the HTTP response was a 500 and the caller had no idea what had run and what had not.
The fix was not clever — it was the obvious one, and I had avoided it because it looked like more infrastructure. Separate the submission from the execution. Accept the job, return an ID, run the work elsewhere. The caller polls or subscribes; the executor is its own process. That decoupling is a job pipeline.
A job pipeline appears in nearly every production backend that needs to process work asynchronously — sending emails, running ML inference, transforming data, orchestrating multi-step SDLC tasks. The common shape is always the same: an API that accepts work, a queue that holds it, workers that execute it, and a store that tracks the result. The interesting design questions are all about what happens when things go wrong.
Think of a job pipeline like a hospital triage system. A patient arriving at the emergency desk does not wait in the lobby while a doctor treats them — they are registered immediately, assigned an ID, and placed in a queue by urgency. Doctors work through the queue independently. If one patient's treatment takes hours, the desk keeps accepting new arrivals. The decoupling is the whole point: arrival and treatment happen at different rates, by different actors.
The key abstraction: Separate job submission from job execution. The submitter should never wait for the job to complete — it should get an ID and poll or subscribe for the result. This makes the system resilient to slow jobs, crashing workers, and traffic spikes.
This module walks through the full job pipeline architecture as I built it in python-orchestration-system: FastAPI for submission, Celery + Redis for queue and worker management, DAG-based task dependency scheduling, and the failure-handling patterns that separate a toy pipeline from one you can trust.
A job pipeline has five components. Each is distinct; mixing them creates coupling that makes failure handling impossible.
flowchart TD
A["API Layer\nFastAPI\n— accepts submissions\n— returns job_id immediately"]
B["Message Broker\nRedis\n— holds pending tasks\n— persists across worker restarts"]
C["Workers\nCelery\n— pull from broker\n— execute task logic\n— horizontally scalable"]
D["Result Store\nRedis + Postgres\n— tracks status\n— holds output or error"]
E["DAG Executor\n— watches completions\n— enqueues ready dependents"]
A -->|"send_task()"| B
B -->|"pull task"| C
C -->|"write result"| D
E -->|"enqueue ready tasks"| B
C -->|"notify completion"| E
A -->|"poll"| D
style A fill:#3b82f6,color:#fff
style B fill:#dc2626,color:#fff
style C fill:#d97706,color:#fffAPI layer (FastAPI): accepts job submissions, returns a job ID immediately, exposes status and result endpoints. Never executes job logic. This is the contract boundary — callers interact only with this layer.
Message broker (Redis): holds pending tasks as queue entries. Redis is the right default here: it is fast for queue operations, supports multiple queues out of the box, and Celery integrates with it natively. For very high durability requirements, RabbitMQ is the alternative — Redis can lose messages on crash if persistence is not configured.
Workers (Celery): pull tasks from the broker, execute them, write results back. Workers are horizontally scalable — add more to handle more concurrent jobs. Each worker is stateless; it only needs the task definition and access to the result store.
Result store: tracks job status (pending, running, succeeded, failed) and the result payload. In python-orchestration-system, I use Redis for short-lived task results and Postgres for durable job records. Redis is fast for polling; Postgres is queryable for audit and retry logic.
DAG Executor: handles dependency ordering — task B can only start after task A completes. A flat queue cannot express this. The executor watches task completions and enqueues successors when their dependencies are satisfied.
The synchronous ingestion trap: The most common mistake is running slow logic inside the FastAPI endpoint — parsing a large file, calling an external API, doing complex validation. If it takes more than ~100ms, it belongs in a worker. An endpoint that takes 10 seconds to respond under load will exhaust the server's connection pool.
The submission returns a job ID; the caller uses it to poll for completion:
The critical point: send_task returns immediately. The endpoint does not wait for the task to complete. The caller receives a job ID and can poll at any interval.
In python-orchestration-system, SDLC tasks have dependencies: the implement step cannot start until the spec is finalized; the review step cannot start until implement is done; all steps must complete before the report is written. These dependencies form a directed acyclic graph.
A flat Celery queue cannot express this. If you enqueue all tasks at once, some will start before their dependencies are ready. The solution: a DAG executor that only enqueues a task when all its upstream dependencies have succeeded.
The execution model mirrors topological sort. Start with all tasks that have no dependencies (zero in-degree). As each task completes, decrement the in-degree of its successors. Enqueue any successor whose in-degree reaches zero.
Why a DAG and not a simple chain? A chain (A then B then C) is a DAG where every node has exactly one predecessor. DAGs allow parallel branches: tasks C and D can both depend on A without depending on each other. In my SDLC harness, multiple test tasks run in parallel after implement completes — a chain would serialize them unnecessarily.
Before the failure-handling section, verify you can answer these from memory:
Reflective prompt: In python-orchestration-system, I keep job records in Postgres even though Celery's Redis backend can track task status. Why might I want both? What does each store give me that the other does not? Write a two-sentence answer before moving on — the answer is about durability vs. speed, and which one each query pattern needs.
Every job pipeline fails. Tasks crash, workers restart, the broker restarts. The question is not whether failures happen — it is how you recover from them without running a job twice or losing it entirely.
Redis and Celery provide at-least-once delivery: if a worker crashes before acknowledging a task, the broker redelivers it to another worker. The task may run more than once. This is the default and the correct trade-off for most jobs — it is far easier to make jobs idempotent than to build a distributed exactly-once delivery system.
An idempotent job produces the same result whether it runs once or ten times. The easiest approach: each job writes to a keyed output. Running the same job twice with the same key overwrites the result rather than creating duplicates.
The silent failure trap: Without a dead-letter queue, exhausted tasks vanish. The job status stays "pending" forever, or goes to "failed" with no path to recovery. Always route permanently failed tasks to a DLQ — a separate queue holding tasks for inspection and manual retry. I monitor DLQ depth as a key alert in python-orchestration-system: a non-zero DLQ depth means something is broken that needs human attention.
Apply the job pipeline design to a concrete problem.
The task: Design a job pipeline for a report-generation system. Users submit a report request (what data to include, which date range). Reports take 30–120 seconds to generate. Users should be able to check status and download the result. The system should handle 100 concurrent report requests.
<CodeExample title="Design Prompt Starter — Report Generation Pipeline" language="markdown" code={`## Requirements Functional:
Non-functional:
Apply job pipeline design principles to realistic scenarios
1. A job submission endpoint takes 15 seconds to respond because it runs a database migration synchronously. What is the correct fix?
Correct Answer: B
Any work that takes more than ~100ms belongs in a worker. The endpoint should accept the request, enqueue the migration as a Celery task, and return immediately with a job ID. A background thread within the same process still consumes a thread from the server's pool and crashes the response if it fails — it does not provide the isolation a separate worker process gives.
2. In a DAG executor, task B depends on task A, and task C also depends on task A. Tasks B and C do not depend on each other. How should the executor handle A completing?
Correct Answer: B
When A completes, the executor decrements the in-degree of all tasks that listed A as a dependency. Both B and C had in-degree 1; A completing brings both to 0. Both are ready to execute and should be enqueued simultaneously — this is the parallel branch that makes DAGs more powerful than simple chains.
3. A Celery task fetches an external API and writes the result to Postgres. The external API occasionally returns 503. What retry strategy is appropriate?
Correct Answer: B
Exponential backoff (1s, 2s, 4s) gives the external API time to recover between retries. acks_late=True ensures the task is requeued if the worker crashes mid-execution — without it, a crash between 'task started' and 'acknowledged' silently drops the job. max_retries=3 is a reasonable limit before routing to a dead-letter queue.
4. What is the difference between queue depth growing and DLQ depth growing as monitoring signals?
Correct Answer: B
Queue depth growing means throughput is insufficient — more jobs arriving than workers can process. Scale out workers or add queue partitioning. DLQ depth growing means jobs are reaching their retry limit and failing permanently — investigate the application logic or external dependency causing the failures. They require completely different responses.
5. Why is acks_late=True important for a task that writes a file to object storage?
Correct Answer: B
Without acks_late, Celery acknowledges the message as soon as the worker receives it. If the worker then crashes before the file write completes, the message is gone and the job is silently lost. With acks_late=True, the broker only removes the message from the queue after the task function returns successfully. A worker crash causes redelivery to another worker.
6. Two users submit a report request with identical parameters. Your pipeline uses a SHA-256 hash to detect duplicates. User B submits while User A's report is still pending. What should the system return to User B?
Correct Answer: B
The idempotency key (parameter hash) maps to a single job ID regardless of job status. User B gets User A's job_id and polls it. When the report completes, both users receive the same result. This is the correct behavior: the parameter hash guarantees that two identical requests should always return the same result, whether already computed or still running.