[SAMPLE] Idempotency is the only pipeline feature that matters

This is a sample post shipped with the site scaffold so the typography can be checked against real-shaped content. Delete it before launch.

Every data pipeline eventually gets run twice. A worker dies mid-batch and the orchestrator retries. Someone discovers a bug in June's numbers and backfills the month. A deploy goes sideways and you rerun the day by hand at 2am. The question is never whether a run will be repeated — it's whether repeating it is safe.

The property, stated plainly

A pipeline step is idempotent when running it twice with the same inputs produces the same state as running it once. Not similar state — the same state. That rules out three habits that feel harmless in development:

• Appending to a table without a dedup key
• Using wall-clock time as an input ("process everything since now minus one hour")
• Writing partial output before the step has fully succeeded

What it looks like in practice

The cheapest implementation is delete-then-insert scoped to the partition being processed:

BEGIN;
 
DELETE FROM fact_orders
 WHERE order_date = '2026-06-17';
 
INSERT INTO fact_orders
SELECT o.id,
       o.customer_id,
       o.total_cents,
       o.order_date
  FROM staging_orders o
 WHERE o.order_date = '2026-06-17';
 
COMMIT;

Run it once, you get June 17th. Run it five times, you still get June 17th. The partition boundary is the contract: as long as every step owns a disjoint slice of the output, retries are boring — which is the highest compliment you can pay an operational property.

In an orchestrator, the same idea shows up as parameterizing every task by its logical date rather than the current time:

@task
def load_orders(logical_date: date) -> None:
    """Load exactly one day; safe to rerun for any day, any number of times."""
    partition = read_staging("orders", day=logical_date)
    replace_partition("fact_orders", day=logical_date, rows=partition)

Why this beats every other feature

Monitoring, lineage, cataloging — all useful, all secondary. When a pipeline is idempotent, incidents end with "rerun it." When it isn't, incidents end with a forensic reconstruction of which rows were double-counted. One of these is a five-minute fix. The other is a postmortem with an action-items section nobody finishes.