Fire, Forget, Regret
PyCon NL 2026 · 15 October 2026 · Jaarbeurs, Utrecht
TL;DR
- Venue: PyCon NL 2026, Utrecht
- Topic: Why long-running Python pipelines lose work when they fail, and how checkpointing fixes it
- Takeaway: Choose between
BackgroundTasks, a queue or a durable pipeline based on how long your job runs - Library: stitch-durable
A talk about what happens to long-running Python pipelines when something goes wrong halfway through. It climbs three rungs:
- Fire-and-forget fails quietly. FastAPI's
BackgroundTasksand bare fire-and-forget tasks don't survive restarts, don't retry, and give you no way to check whether anything finished. A task can even be garbage collected mid-execution and silently disappear. - Task queues help, with a gap. ARQ, Celery and Dramatiq add retries and persistence, but if step six of ten crashes you start again from step one.
- Durable execution. Tools like Temporal, DBOS and Hatchet persist every completed step, so a crashed pipeline resumes instead of repeating.
Live demo
A real AI news pipeline that searches for fresh AI news, downloads each article, asks a local LLM whether it is a real breakthrough or hype, and stores the verdict in MongoDB. I broke it on purpose: kill -9, wifi off, model stopped, database down, and a rate-limited model server. The plain version crashed or redid finished work. The durable version picked up where it left off with no duplicates, using one decorator.
The talk closes with a practical way to choose between BackgroundTasks, a queue, or a checkpointed pipeline based on how long your job runs.
Library
The demo grew into stitch-durable, a standard-library-only checkpointing library for async pipelines.