Most data pipelines do not fail because the work is hard. They fail because nobody decided, up front, what should happen when something goes wrong. Reliability is not luck. It is a stack of small design choices you make before the first row ever moves.
A pipeline that pages you at 3am is almost always a pipeline that trusted the world to behave. The network stayed up. The upstream schema never shifted. Every file landed on time and clean. Real data is messier than that, and the teams who sleep through the night are the ones who planned for the mess instead of hoping their way around it. If you are still fighting to get raw inputs into usable shape, start with turning messy data into a product advantage. This piece picks up where that one ends: the point where a pipeline has to run every day, on its own, without anyone watching.
Make every step safe to re-run
The most useful property a pipeline step can have is idempotency. A step is idempotent when running it twice produces the same result as running it once. Load the same file two times and you get one clean table, not doubled rows. Reprocess yesterday and you overwrite yesterday, you do not quietly append a second copy.
This single property changes how a bad night feels. When every step is safe to re-run, recovery is boring: run it again. You are no longer awake trying to work out how far a half-finished job got before it died, or hand-deleting duplicate records under pressure. You retry, and the system heals itself.
Idempotency comes from a few habits. Write to a partition or a key you can overwrite, not a blind append. Use deterministic IDs so the same input always maps to the same row. Stage work somewhere temporary and swap it into place only when the whole step succeeds, so a crash halfway through leaves nothing behind. Retries then lean on the same foundation. Transient failures are normal: a database blinks, an API rate-limits you, a node restarts. Wrap each external call in a retry with exponential backoff and a little jitter, so you wait longer between attempts and do not stampede a service that is already struggling. Because the step is idempotent, retrying carries no risk of doubling work.
Backfills are the same idea pointed at the past. When you fix a bug or add a column, you often need to reprocess weeks of history. If your pipeline is parameterized by date and every run is idempotent, a backfill is just the normal job aimed at old partitions. If it is not, a backfill is a weekend and a prayer.
A pipeline you cannot safely re-run is not a pipeline. It is an incident with a schedule.
Externo
Agree on a contract, then enforce it
Most pipeline breakages start somewhere upstream, in a system another team owns. They rename a column, switch a currency, or start sending nulls where they never did before. Your job ran fine yesterday and breaks today, through no change of your own.
A schema contract is the fix. It is a written, checked agreement about the shape of the data moving between a producer and its consumers: which fields exist, their types, which are required, and what a valid value looks like. When incoming data violates the contract, you catch it at the door with a clear message, instead of letting bad values ripple silently into dashboards and models downstream. The point is not bureaucracy. It is turning a vague, hours-long mystery into a five-second answer. "Upstream dropped the email field this morning" is a fix. "The numbers look weird" is a night.
Contain the blast radius
One bad source should never take down all the others. Yet the default pipeline fails in exactly that way: a single unreachable API or malformed file throws an error, the whole run aborts, and forty healthy datasets go stale because one sibling misbehaved.
Isolate failures so problems stay local. Run independent sources as independent tasks with their own boundaries. Let one fail and quarantine its bad records without stopping the rest. Keep the last good data in place rather than replacing it with an error. A well-built pipeline degrades in pieces, it does not collapse all at once. Externo's engineering work leans hard on this idea across data and product alike: design the seams so that when one part breaks, and something always eventually breaks, the damage stays contained and obvious rather than wide and mysterious.
Alert on symptoms, not noise
Here is why most people get paged at 3am: their alerts fire on low-level errors instead of real problems. A retried timeout that already recovered should not wake anyone. A table that is twelve hours stale absolutely should. The trick is to alert on the symptoms your users would actually notice, not on every internal hiccup along the way.
- Freshness: the data is older than it should be, so a table that updates hourly has not moved in six hours.
- Volume: today's row count sits far above or below the normal range, which usually means a partial load or a duplicated one.
- Nulls and validity: a field that is normally full is suddenly half empty, or values fall outside their allowed set.
- Contract breaks: the incoming schema no longer matches what you agreed to accept.
- End-to-end success: the whole run finished and published, not merely that no single task happened to throw.
These signals map to what people actually care about: is the data here, is it complete, and is it correct. A page that means something is a page you can trust, and one you can safely ignore the rest of the time.
None of this holds without tests. Treat pipeline code like the production software it is. Unit-test the gnarly transforms with tricky inputs: empty batches, duplicate keys, awkward time zones, the null that only shows up in production. Run the whole pipeline against a small fixed sample in CI and compare the output to a known-good result. Add data-quality checks that run on every load, not just when someone remembers. Tests are how you keep tonight's fix from becoming next month's outage, and they are the same discipline that lets you ship higher up the stack, like your first LLM feature, without the chaos.
Put these habits together, idempotent steps, retries with backoff, easy backfills, enforced contracts, isolated failures, and alerts that track symptoms, and the pager goes quiet. Not because nothing ever fails, but because failures become small, local, and safe to fix in daylight. That is the whole goal: data your team can trust, and a pipeline calm enough that nobody has to babysit it at 3am.
Data Pipelines FAQ
