140 terabytes a month, observed.

The first conversation about this work was a finance review of compute spend. The number was real, the trend was wrong, and the team had a pile of jobs that had grown slower over time without anyone catching it. Nobody's fault — Spark gives you so many ways to be slow that you cannot watch them all.

The pipeline itself: Kafka → Flink stateful enrichment → Iceberg landing → Spark transformation → analytics warehouse. Volume is roughly 140 TB / month across feature and event streams. The work I owned was making it observable, then making it cheap.

Observability:

• Per-job SLO dashboards with throughput, cost, and freshness as first-class metrics.
• Schema-drift detector emitting Kafka events when an upstream changes shape.
• Run-time DAG that tags every Spark stage with the upstream contract it depends on.

Most of the wins came from columnar work. Switching feature generation from row-based UDFs to Arrow-aware vectorized expressions unblocked the planner. After that, an honest review of skew exposed a small handful of partitions doing 80% of the work; rebalancing those was a one-line change once we could see them.

• SLO dashboard template (Grafana).
• Schema-drift detector pattern.
• Spark / Arrow rewrite playbook for the team.

A first-class lineage layer. Right now we track contracts per stage, but we do not have an end-to-end provenance trail from source event to warehouse fact table. That would change incident triage from 12 minutes to 2.