This content originally appeared on HackerNoon and was authored by JEEVAN KRISHNA PARUCHURI
\ The Slack message from my manager was three words long. 'We need to talk.' That's how I found out our Azure spend had crossed a number that was making people uncomfortable in the finance review.
I'd been expecting it, honestly. We'd been scaling fast (more pipelines, more datasets, more teams querying the lake) and I hadn't sat down to seriously look at where the money was going. That was a mistake. When I finally pulled the cost breakdown, the culprit wasn't what I expected.
It wasn't the compute. It was the mess we'd left behind.
My first assumption was that we were over-provisioning Spark clusters. That's usually the go-to answer when cloud bills spike. Someone left a fat cluster running, or the auto-scaling config is too aggressive. I spent half a day pulling cluster utilisation metrics and honestly, they weren't bad. Average CPU was around 60%, memory was fine. That wasn't it.
The real problem was storage. Specifically: 18 months of accumulated small files, expired snapshots we'd never cleaned up, and a Spark streaming job that had been writing 200KB Parquet files into Delta Lake at roughly 4-minute intervals since the day it went live. We had tables with half a million files in them. Running even a basic aggregation on those tables was slow not because of compute but because of the metadata overhead. The driver was spending more time building the file list than running the query.
The storage bill looked fine on its own. But the compute bill for querying badly maintained storage was enormous. That's the part that doesn't show up clearly in cost dashboards until you're looking for it.
The three things we actually fixed
I want to be specific here because vague advice about 'optimising your storage layer' is everywhere and mostly useless. Here's what we actually did.
First: compaction. We set up a weekly Spark job that runs OPTIMIZE on every Delta table in our processed zone, targeting a target file size of 256MB. For tables that receive continuous streaming writes, we run it nightly. This dropped the file count on our worst tables from around 480,000 files to under 2,000. Query performance on those tables improved by roughly 6x. The compute cost for the compaction job itself is trivial compared to what we were spending querying fragmented storage. (A note on versions: we were running Delta Lake 2.1 on Spark 3.3 at the time. OPTIMIZE behaviour, particularly around Z-ordering and bin-packing thresholds, has changed across versions, so check the documentation for your specific version.)
Second: snapshot expiry. Delta Lake keeps a history of all table versions by default, and we had never configured retention. Some tables had 14 months of snapshots sitting in storage doing nothing. We set a 30-day retention policy across the board and ran VACUUM. I won't tell you the exact GB we freed up because honestly the number is embarrassing given how long we let it accumulate. It was a lot.
Third: right-sizing the streaming write interval. The job that was writing tiny files every 4 minutes was doing it because that's what the original author set and no one had questioned it. The downstream consumers didn't need data more frequently than every 30 minutes. We changed the trigger interval. File sizes went from 200KB average to 180MB average. Done.
Before and after: our worst table went from ~480,000 files averaging 200KB to under 2,000 files averaging 180MB. Snapshot history dropped from 14 months to 30 days. Streaming write interval moved from 4 minutes to 30 minutes. Total query time on affected tables improved by roughly 6x.
Those three changes accounted for the bulk of the savings. But I also made a mistake in what I tried next, and it’s worth mentioning because the instinct behind it was reasonable.
What I got wrong in the first pass
My first instinct after the compaction job ran was to immediately move to tiered storage: archive cold partitions to the cool or archive tier, keep only the last 90 days in hot. That logic made sense on paper but I underestimated how often 'cold' data was still being queried. We have a compliance requirement that means audit teams pull 2-3 year old transaction data fairly regularly. Archiving those partitions would have hurt query latency on regulatory lookups, which is exactly the wrong thing to optimise away in a banking environment.
I put tiered storage on hold and instead focused on partitioning strategy. Several of our large tables were partitioned by ingestion date rather than business date. That meant queries filtering by transaction date, which is most queries, were doing full scans across partition boundaries. Re-partitioning those tables was a bigger project, but the long-term compute savings justified it.
The 40% number
The 40% reduction took about three months from the first compaction run to showing up meaningfully in the monthly bill. The breakdown was roughly: 15% from compaction and query efficiency gains, 12% from snapshot cleanup reducing storage costs, 8% from the streaming file size fix, and the remaining 5% from a few other smaller adjustments we made along the way.
We didn't touch the data models. We didn't change schemas. We didn't ask downstream analysts to rewrite queries. The business didn't notice anything except that some reports started running faster.
One thing worth saying plainly
None of this required exotic tooling or architectural heroics. Everything I've described is available in Delta Lake's standard API and is documented. The reason we hadn't done it was not ignorance. Nobody owned it. Compaction, snapshot management, file size hygiene: these are operational disciplines, not one-time tasks. We now have a dedicated maintenance pipeline that runs on a schedule, with alerting if table health metrics fall outside acceptable ranges.
If your cloud bill is growing faster than your data volume, the answer is usually sitting in your storage layer. Pull the file count histograms first. The rest tends to follow.
This content originally appeared on HackerNoon and was authored by JEEVAN KRISHNA PARUCHURI
JEEVAN KRISHNA PARUCHURI | Sciencx (2026-04-24T04:53:53+00:00) How I Cut Our Cloud Bill by 40% Without Touching a Single Data Model. Retrieved from https://www.scien.cx/2026/04/24/how-i-cut-our-cloud-bill-by-40-without-touching-a-single-data-model-2/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.