If you have spent any time around enterprise software, you have probably seen some version of this story.
A company has data sitting in an ERP, a PostgreSQL database, a pile of spreadsheets, or, more likely, all three. Every Monday morning, somebody exports a few CSV files, cleans them up, pastes the results into another spreadsheet, and refreshes the Power BI dashboard before the leadership meeting.
The process is tedious. It is fragile. Everyone agrees that it should be automated.
So the company calls a consultant and asks to have the dashboard refreshed automatically.
Six weeks later, they have a cloud data warehouse, an ETL platform, an orchestration layer, a semantic model, three new vendor contracts, and a monthly bill that changes depending on how often people look at their own data.
Congratulations. You have built a $100,000 cron job.
Nobody Asked for a Data Platform
The original request was probably something like this:
Can we have yesterday’s revenue by region in Power BI before the Monday meeting?
Or maybe it was:
Can we combine the ERP data with the spreadsheet Susan maintains without making Susan spend three hours doing it by hand?
Nobody walked into that meeting and asked for a lakehouse. Nobody asked to begin a multiyear data transformation initiative. Nobody asked for a new department whose primary function is keeping the dashboard infrastructure alive.
They wanted a query to run on a schedule and the results to appear in Power BI.
That is the part of the modern data stack conversation that we seem determined to skip. Before we ask whether the company needs Snowflake, Fabric, Databricks, or a small army of consultants, we should probably ask what the actual job is.
Sometimes the job is not very complicated.
Read the data. Transform it. Put the result somewhere Power BI can find it. Do it again tomorrow.
Linux administrators have been automating that class of problem since roughly the invention of dirt.
How the Bill Gets to $100,000
To be fair, nobody usually proposes spending $100,000 on a scheduled report in one shot. The absurdity arrives one perfectly reasonable line item at a time.
First comes the warehouse. The data needs somewhere “proper” to live, after all.
Then you need an ETL or ELT tool to get the data into the warehouse. Naturally, that tool has its own subscription and usage limits.
Then you need orchestration to make sure the jobs happen in the right order. You need monitoring to watch the orchestration. You need consultants to configure everything. You need someone internally to understand what the consultants configured. You may need additional Power BI capacity because the new architecture has changed how the data is served.
Finally, you need a recurring meeting to discuss why last night’s pipeline failed.
Add the software subscriptions, implementation work, ongoing support, usage-based compute, and internal staff time, and the number gets large very quickly. The $100,000 is not necessarily a single invoice. It is the total cost of turning a small, understandable problem into a platform that requires its own care and feeding.
Every individual purchasing decision can be defended. The complete picture is still ridiculous.
Complexity Looks Like Sophistication
There is an uncomfortable incentive problem here.
Complexity is easy to sell because complexity looks impressive. A large architecture diagram makes a large statement of work feel justified. A scheduled SQL query writing to an open file format does not look nearly as exciting in a slide deck.
Nobody gets promoted for saying, “We can solve this with a small Linux service and a timer.”
The consultant is also in a difficult position. If the customer asks what it means to “do data properly,” the safest professional answer is to recommend the complete enterprise stack. Every box on the diagram solves a real problem. Whether this particular customer has that problem is a separate question.
This is how resume-driven development escapes the engineering department and becomes resume-driven infrastructure. We choose the architecture that sounds most sophisticated instead of the one that most directly solves the business problem.
In Defense of the Humble Cron Job
I am not suggesting that every business should literally throw a Bash script into /etc/cron.daily and call it a data strategy.
Cron is the metaphor here, although sometimes it is also the perfectly reasonable implementation.
The point is that scheduled work is a solved problem. SQL is already an excellent language for describing data transformations. Open formats like Parquet and Delta give us durable, portable output that many tools can read. We do not need to invent an entire platform every time someone wants a dashboard refreshed at 7:00 a.m.
A sane version of this system should be easy to explain:
- Connect to the data where it already lives.
- Run a known transformation on a schedule.
- Write the result into a durable format.
- Make that result available to Power BI.
- Alert somebody if the job fails.
That last point matters, but there is another requirement that matters even more: a failed refresh should not take the dashboard down with it.
Stale, Never Broken
One of my biggest frustrations with modern software is how often we turn a routine failure into a complete outage.
If Tuesday morning’s transformation fails, Power BI should still be able to read Monday’s successful dataset. The operator should get an alert. The business should see that the data is stale. What they should not see is a broken dashboard five minutes before the executive meeting.
That is the difference between a pipeline and a dependency chain.
When the dashboard reads from durable serving files, it does not need the transformation service to be alive every second of the day. The transformation can fail visibly and recoverably without dragging every downstream user into the incident.
Is yesterday’s data ideal? No.
Is yesterday’s data better than an error page? Almost always.
This is boring infrastructure thinking, and I mean that as a compliment. Good infrastructure limits the blast radius when something goes wrong. It does not convert every hiccup into a company-wide fire drill.
Yes, Some Companies Really Do Need the Warehouse
Before the data engineers start sharpening their pitchforks, let me be clear: data warehouses are real tools that solve real problems.
If you are combining hundreds of sources, maintaining years of analytical history, supporting thousands of users, processing massive volumes, or doing near real-time analysis, you may genuinely need the whole stack. Strict lineage, governance, and broad self-service analytics can also justify more sophisticated infrastructure.
Netflix has Netflix-sized data problems.
Most regional manufacturers, construction firms, repair stations, professional services companies, and mid-market operators do not.
Buying the architecture of a hyperscaler does not turn your company into a hyperscaler. It just gives you the invoice of one.
The honest architectural question is not, “What would a Fortune 50 company deploy?”
It is, “What is the smallest reliable system that solves our actual problem?”
Ask the Impolite Questions
Before approving the next grand data initiative, ask a few impolite questions:
- How many source systems are actually involved?
- How often does the data truly need to change?
- How many people consume the result?
- What happens if one refresh is missed?
- Do we really need a second permanent copy of all this data?
- Can the required output be expressed as a handful of SQL queries?
- Which capabilities in this proposal will anybody use during the next twelve months?
- What will the complete system cost after implementation, support, compute, and internal staff time?
Those questions will not always lead to a tiny solution. They will, however, force everyone to connect the proposed architecture to the actual business requirement.
That alone can save a shocking amount of money.
Why We Built Alice
This exact pattern is why we built Alice.
Alice connects to the data a company already has, creates the required transformations, and serves the results to Power BI. The serving files use open formats and can live in the customer’s own storage or in an isolated managed container. The dashboard reads the last successfully delivered data, so a failed pipeline means stale data, not a broken report.
There is no requirement to move everything into a new warehouse first. There is no six-week consulting engagement required to create the first dataset. There is no consumption meter spinning every time somebody refreshes a dashboard.
Alice is not an argument that data warehouses should never exist. It is an argument that you should not have to buy one just to stop Susan from manually cleaning CSV files every Monday morning.
The goal is simple: keep Power BI, keep your data where it makes sense, and remove as much expensive machinery as possible from the space between them.
The Distance Between the Problem and the Solution
The most expensive part of enterprise software is often the distance between the problem and the solution.
Every new platform, abstraction, vendor, and consulting handoff adds cost. It also adds another place where the Monday morning dashboard can fail. Sometimes that complexity is necessary. Far more often than our industry likes to admit, it is optional.
So before you approve the warehouse, the orchestration platform, the implementation project, and the monthly bill nobody can quite predict, ask one final question:
Are we solving a genuinely difficult data problem, or have we built a $100,000 cron job?
