When your maintenance job can't outrun your writer: an Iceberg snapshot-bloat outage

The page

At 21:30 UTC, PagerDuty: bronze.activity_events is stale (SLA: 120min). The standard bronze refresh DAG was failing with a ClickHouse memory error, "would use 4.74 GiB, maximum: 4.66 GiB," and the fast tier was blowing its 20-minute run timeout. Two different failure signatures, one cause.

The cause I already knew about

activity_events and session_events are written by Kinesis Firehose's native Iceberg destination. Firehose creates one snapshot per commit and, at this volume, commits every 20 to 40 seconds (it flushes on its 128 MB buffer threshold, which is already the maximum). That's about 4,300 snapshots per table per day, forever, unless something expires them.

Nothing was expiring them. The maintenance DAG built for exactly this shipped paused, pending a supervised first run, a reasonable gate given that its predecessor once corrupted 32 tables. But the supervised run had died with AttributeError: 'Table' object has no attribute 'maintenance'. The Airflow environment shipped pyiceberg 0.7.1, and the code was written against 0.11. The gate was unsatisfiable, so the pause quietly became permanent.

Six days before the outage I'd measured the consequence and filed a ticket: about 20k snapshots and 20 MiB of metadata.json per table. Every ClickHouse icebergS3() query plan re-parses that file from S3 (I'd deliberately disabled the metadata cache in May, since it under-accounts RAM by 10 to 20x). A 20 MiB parse per plan inflated planning memory past the 5 GB cap hardcoded in the bronze insert macro, and stretched per-table refresh from seconds to minutes. That's both failure signatures at once.

The part I didn't know: expiration loses the race, deterministically

Draining the backlog was supposed to be routine. There's a writer-safe CLI for exactly this. It hung. Ten minutes of silence inside a single commit.

The reason is worth internalizing if you run Iceberg next to a hot writer. pyiceberg's expire_snapshots() commit doesn't just rewrite metadata.json. It runs a file-reachability pass that reads the manifest list of every snapshot in the table to find newly orphaned files. At 20k snapshots that's 20k+ S3 reads per commit attempt. Meanwhile, Glue's optimistic concurrency rejects the commit if the table version changed since it was loaded, and the writer commits every 20 to 40 seconds. When your commit window is minutes and the writer's interval is seconds, you don't lose the race sometimes. You lose it every time. Retries redo the full scan. Smaller batches redo the full scan. No parameter fixes a window that's wider than the writer's interval.

I pivoted to a metadata-only commit: stage a raw snapshot-removal update on a transaction, skip the reachability scan entirely, and leave the orphaned manifest-list files for a later sweep, since they're invisible to readers and cost pennies to clean up eventually.

# Illustrative shape of the fix, not the production code.
from pyiceberg.table import Transaction
 
def expire_snapshots_metadata_only(table, snapshot_ids_to_remove):
    with Transaction(table) as txn:
        txn.remove_snapshots(snapshot_ids_to_remove)
        # No file-reachability scan here. Orphaned manifest-list
        # files are invisible to readers and get swept later.
        txn.commit()

That cut the attempt to about 42 seconds, almost all of it spent parsing the 20 MiB metadata file, and it still lost five races in a row.

The end of the funnel: pause the writer. The producer is a Lambda consuming a Kafka stream; disabling its event source mappings pauses consumption with offsets preserved. Zero data loss, the backlog just waits. With the writer quiet, both tables drained about 20k snapshots each in under a minute. Re-enable, replay, re-trigger the refresh DAGs, alerts clear. Total ingestion pause: about ten minutes, on data that was already five hours stale.

The fixes

• The expire primitive is now metadata-only, with a unit test asserting the maintenance API's reachability scan is never touched. At steady state (around 2k snapshots, 2s parse) it wins the race against the live writer easily. The deterministic loss only shows up once the backlog is huge, which is precisely the state the daily DAG is supposed to prevent.
• pyiceberg pinned to 0.11.1, unblocking the maintenance DAG's supervised run and its unpause.
• The 5 GB insert cap is now 10 GB with disk spill at 5 GB, so metadata pressure degrades to slower-but-successful instead of failing outright.
• Fast-tier timeout went from 20 to 45 minutes. It was sized for healthy metadata, not for the state it's meant to survive.
• Snapshot retention properties are now set on both tables, and a full recovery runbook exists, including the writer-pause procedure.

What I'd tell you now

1. A paused safety-critical job needs an owner and a deadline. "Shipped paused, pending a supervised run" rotted into "permanently off" the moment the supervised run hit an unrelated blocker. If a pause gate becomes unsatisfiable, that's an incident waiting to happen, not a backlog item.
2. Maintenance cost has to scale with the thing it bounds. Snapshot expiration whose commit window grows with snapshot count is a tool that only works when you don't need it. Check your maintenance path at the scale it's supposed to protect you from, not at the scale it was tested at.
3. Optimistic concurrency failure modes are bimodal. Against a hot writer you either win quickly or lose deterministically. There's little in between. If retries aren't converging, stop retrying and change the geometry: shrink your window, or quiet the writer.
4. Know your zero-loss pause lever before you need it. Kafka retention plus a pausable consumer gave me a ten-minute, fully replayable producer pause. That lever being cheap and already known is what turned a stuck recovery into a finished one, instead of a much longer night.