The dbt test suite that cost almost as much as the pipeline

Every dbt run in this pipeline executed the full test suite: schema tests, data quality checks, row-count assertions, freshness checks, all of it, every time. DAGs ran every 10 to 15 minutes across three environments, so that added up to hundreds of test queries an hour against ClickHouse Cloud.

ClickHouse Cloud bills on compute time. A count(*) over a large table and a referential-integrity join look cheap in isolation. Run them hundreds of times an hour and they start showing up next to the production models on the bill, sometimes bigger.

The fix: not every test needs to run in real time

I split tests into two tiers instead of one flat suite.

Critical tests run on every DAG. A small set that catches anything data-breaking immediately: not-null checks on primary keys, uniqueness on surrogate keys, source freshness thresholds. Fast, cheap, high signal. If one of these fails, you want to know in minutes, not by morning.

Everything else runs once a night. Referential integrity, accepted-value checks, row-count anomaly detection, the full data-quality monitor suite. All of it moved into a single nightly DAG.

# dbt_project.yml (illustrative)
models:
  marts:
    +tags: ["critical"]
 
# schema.yml
tests:
  - not_null:
      column_name: id
      tags: ["critical"] # runs every DAG
  - relationships:
      to: ref('dim_customer')
      field: customer_id
      tags: ["nightly"] # runs once, batched

The nightly DAG uses TriggerRule.ALL_DONE, so it still runs the full suite even if a critical test failed earlier in the day. That matters more than it sounds like it should: your anomaly detector needs a consistent daily history to be worth anything, and a partial run breaks that history the same way a missing day does.

What changed

The nightly run takes about 8 minutes per environment and, because it's one batched run instead of a hundred small ones, it actually produces better data for the anomaly detector than the old setup did. That was the part I didn't expect going in.

What I'd tell you if you're about to do this

Not every test needs to run in real time. Freshness and uniqueness matter on every single run because they catch the kind of failure that compounds. Referential integrity can wait eight hours; nothing bad happens in that window that wouldn't already be visible some other way.

ClickHouse Cloud pricing rewards batching, and I'd guess most serverless warehouse pricing does too. A lot of small queries pay startup overhead over and over. Fewer, bigger queries pay it once. If your test bill looks disproportionate to your pipeline bill, this is usually why.

And if you're running an anomaly detector on top of your test results: it needs volume and consistency more than it needs speed. Batching your tests into one daily run instead of scattering them across the day made the detector's output better, not worse. I didn't design for that. It just fell out of fixing the cost problem.