Disclosure: I’m Director of Engineering at Draft.dev, which produces technical content for Haevek. I’m publishing this because the pattern it describes kills more good work than most engineering teams want to admit, and Andrew and Kevin articulate why better than anyone I’ve spoken to.
I came to this topic as a data engineer and BI developer who has spent time building pipelines against real operational constraints. When Andrew Hoeft described a pattern he’d seen across government analytics programs and commercial enterprise deployments, I recognized it.
Across data and AI initiatives, roughly nine out of ten projects that reach a successful proof of concept stall before reaching production at a sustainable cost. The cause Andrew and Kevin identified is consistent across program types and industries: the economics of the production step. What kills these projects is the difference between what something costs to prove and what it costs to run at the scale the business actually requires.
The test the prototype doesn’t run
The proof of concept succeeds because it runs on interesting data. A data scientist curates a sample, exercises the model, and demonstrates that the technique is sound. That’s exactly the right way to establish feasibility. Production data doesn’t look like that.
Take sentiment analysis on customer support transcripts as a concrete example. Building a classifier on a few hundred curated conversations is cheap and the results are compelling. Apply the same process to every support interaction a large consumer operation handles in a day, and somewhere between 90 and 99 percent of those conversations are routine: order tracking, password resets, standard billing questions. The model processes all of them anyway, because there’s no cheap way to identify the signal before you’ve committed to the expensive step. The prototype ran on the interesting fraction. Production runs on everything, and the cost per useful output is orders of magnitude higher than the proof of concept suggested.
Andrew’s framing of the more tractable approach is worth holding on to: treat expensive compute as a scarce resource rather than a general-purpose tool. A system that efficiently finds the ten seconds of genuine signal within ten minutes of noise and routes only that to downstream processing will substantially outperform one that floods the expensive layer with everything. The technique is identical. The architecture around it is what makes the economics work.
The three forces that compound the problem
Andrew and Kevin identified three organizational dynamics that consistently push projects past the cost analysis.
The first is the translation problem. Business needs get passed to engineering teams as vague mandates: detect fraud in financial transactions, identify sentiment in customer support, find anomalies in sensor telemetry. The engineer builds something that meets the technical objective. Nobody has done the analysis of whether meeting that objective is worth the cost at actual production volume. The second-order question doesn’t surface until the infrastructure bill does. Andrew described a financial services organization that needed one new capability: sharing data with customers in a specific compliant format. The off-the-shelf path would have delivered it in two to three weeks. The response was to assign four engineers and give them a year to build a custom solution. The result was an order of magnitude more in engineering investment and a feature that reached customers ten times later than it needed to.
The second is the pressure to demonstrate AI progress. At many large organizations, completing an AI initiative has become a success metric independent of whether the initiative produces value that outweighs its cost. Prototypes get built, demos get presented, and production decisions get driven by enthusiasm. The economic analysis gets deferred.
The third is familiar to anyone who has worked in engineering. Engineers are drawn to technically interesting work. Building a complex LLM-powered pipeline is more compelling than instrumenting a rule-based classifier, and the more novel the approach, the more organizational momentum it gathers regardless of whether the complexity is necessary. By the time the infrastructure cost becomes apparent, significant time and budget are already committed.
Where the math breaks
The infrastructure cost problem plays out differently depending on how easily you can define the return on investment. Andrew and Kevin walked me through three versions. The outcome is similar across all three.
When ROI is indirect and hard to measure, a budget downturn exposes the problem immediately. A large multinational conglomerate built supply chain applications covering risk management, inventory management, and supply visibility, aggregating data from dozens of sources into dashboards that gave teams genuinely faster access to what they needed. Data that had previously taken weeks to pull together was available in seconds. Users valued the tools. The problem was that the value was indirect: better decisions, easier workflows, faster access to information. When business conditions tightened, indirect value is difficult to defend against a concrete infrastructure line item. The organization rebuilt the capability in-house to reduce costs, which helped partially but didn’t resolve it. The infrastructure required to normalize, enrich, and cache data from that many sources couldn’t be justified, and the program was significantly scaled back.
When ROI is directly measurable, the math still often fails. County governments have a clear case for property appraisal tooling: counties routinely lag market rates on valuations by years, and more accurate appraisals translate directly into tax revenue. The value is quantifiable before a line of code is written. Accuracy requires continuous investment, though: a large volume of localized data, regular model retraining as market conditions shift, inference pipelines running frequently enough to stay current. That compute cost only makes economic sense for large counties with substantial technology budgets already in place. Across state and local government programs, dozens of these projects have been started, tested, and cancelled at the cost analysis stage.
When ROI is nearly impossible to define, the infrastructure requirement still has to be justified against something. Law enforcement intelligence applications aim to make it easier to investigate crimes and identify connections across data from multiple agencies. The value of improving investigative capacity is real and nearly impossible to convert into a defensible number. The infrastructure requirement is extreme: data from multiple sources that is frequently incomplete or incorrect requires heavy ETL pipelines to correct and normalize, hard latency requirements force large datasets into expensive graph caches, and those graphs are costly to build and maintain. Across the programs Haevek has encountered in this space, cost analysis has cancelled more projects than any technical obstacle has.
What the economics actually need
Before building, the question worth asking is whether the project is economically viable at the scale and user base the business actually has. Most teams focus on technical feasibility and defer the economic question. By the time it gets answered, reversing course is expensive.
When the infrastructure cost of running distributed compute workloads drops substantially, more projects cross the viability threshold. The supply chain dashboards users genuinely valued become fundable. The property appraisal models that worked in testing become deployable for mid-sized counties. The law enforcement intelligence applications abandoned after successful pilots get a second chance at production. Andrew described it directly: innovation programs that consistently produce promising prototypes with no path to production have the ideas right. The production economics are what break them.
Haevek built Falcon to address that problem. Falcon is a distributed compute engine designed to run workloads at substantially lower infrastructure cost than Apache Spark, which changes the calculation at the production step where most data projects currently stall. If you have workloads that proved themselves in testing and failed the cost analysis, haevek.com is where the conversation starts.
Comments
Loading comments…