Insights
Data infrastructure before AI: why most AI projects fail
Most artificial intelligence (AI) projects fail because the data underneath them is undefined, untrusted or unreachable, not because the model is weak. Vantage Systems builds the data foundation first: agreed definitions, pipelines that run without anyone touching them, named owners and quality checks, so the AI on top has something solid to stand on.
Mohamed El Askary, Founder, Vantage Systems
Updated · 8 minute read
Why do most AI projects fail?
They fail mostly for organizational and data reasons that show up long before anyone tunes a model. The Rand Corporation's 2024 study The Root Causes of Failure for Artificial Intelligence Projects notes that by some estimates more than 80 percent of AI projects fail, twice the rate of information technology projects that do not involve AI.
Rand's researchers interviewed 65 experienced data scientists and engineers and found five root causes. Two of them are about data directly: organizations lacking the data needed to train an effective model, and lacking the infrastructure to manage that data and deploy what they build. A third, misunderstanding the problem to be solved, usually shows up as arguments over what a number means.
Analysts say the same thing in different words. Gartner predicted that at least 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025, naming poor data quality first among the causes.
What does AI-ready data actually mean?
It means data that is fit for one specific use, not data that is perfect everywhere. Gartner's February 2025 research warns that through 2026, organizations will abandon 60 percent of AI projects unsupported by AI-ready data, and that 63 percent of organizations either lack or are unsure they have the right data management practices for AI.
In practice, data is ready for a use case when four things are true: the people who will act on the output agree on what each field means, the data arrives on a schedule without manual steps, someone is accountable when it breaks, and you can trace any number back to its source. Readiness is use-case specific. Sensor data can be ready for a maintenance model while the same plant's cost data is nowhere near ready for a margin forecast.
Gartner also stresses that readiness is ongoing work rather than a one-time cleanup, covering metadata, monitoring and governance. That matches what we see: the pipeline that was correct at launch drifts the first time someone renames a product line.
Which data problems sink AI projects most often?
Four problems account for most of the damage we see: definitions nobody agreed on, quality nobody measures, pipelines that depend on a person, and data that nobody owns. Each looks like a technical issue and turns out to be a management one.
Definitions
Ask three departments what "revenue", "active customer" or "downtime" means and you will often get three answers. An AI system trained or prompted on mixed definitions will produce confident numbers that each department rejects. Write the definitions down, get one owner to sign off, and put them where the pipeline can enforce them.
Quality
Bad data is expensive before AI ever touches it: Gartner's data quality research puts the average cost of poor data quality at no less than 12.9 million dollars a year per organization, from its 2020 research. The fix is boring and effective: automated checks for missing values, duplicates, impossible readings and late arrivals, with alerts to a named person.
Pipelines
If a monthly report depends on someone exporting a file from the enterprise resource planning (ERP) system and pasting it into a workbook, any AI built on it inherits that fragility. Automated pipelines that extract, clean and load on a schedule are the difference between a demo and a system that runs on a holiday weekend.
Ownership
Every data set needs a business owner who decides what correct means and an engineer who keeps it flowing. When ownership defaults to the information technology team, quality questions go unanswered because nobody there knows what a plausible kiln temperature or grant amount looks like.
What is the minimum viable data foundation for a first AI use case?
It is the smallest set of trusted, automatically refreshed tables that one use case needs, plus the rules that keep them trustworthy. You do not need an enterprise data platform before your first model; you need one narrow slice done properly, which our data infrastructure work is scoped around.
- One use case with a named business owner and a decision it will change.
- Source systems identified and connected, whether that is the ERP, a manufacturing execution system (MES) or a research grants database.
- A written data dictionary for every field the use case touches, signed off by the owner.
- Scheduled pipelines into a central store that can be queried with SQL (Structured Query Language), with no manual steps.
- Automated quality checks with alerts, and a log of every failed run.
- Access rules that decide who sees what, and data kept in the region agreed with the business.
- A first screen people actually use, so problems in the data are seen and reported early.
What did our university and cement plant projects teach us?
They taught us to start with the decision, accept partial data honestly, and make definitions the first deliverable. For a university in Egypt we built a research impact dashboard relating research spending to research output by faculty, department and researcher. The question was simple to ask and hard to answer, because no single system held both sides.
Where real cost figures were not yet available, we used an illustrative cost model, labeled as such on screen, rather than waiting or quietly inventing precision. The dashboard went live and was presented to university leadership, with every estimated figure marked so nobody mistook a placeholder for a measurement.
At a cement plant in Pakistan, our operational intelligence work is in progress. The task is connecting plant and business systems into one view of production and operations, and most of the early effort is exactly what this article describes: finding where each number lives, agreeing what it means across departments, and automating its path to a single screen. It will feed operational dashboards first; any AI comes after the foundation holds.
How can you tell whether your data is ready? A checklist
Run your first use case through the checks below. If more than one row lands on the right-hand side, fix the foundation before you buy or build any model; the same logic shapes our AI implementation plan.
| Check | Ready | Not ready yet |
|---|---|---|
| Definitions | Each metric has one written definition and an owner who signed it | Departments argue over numbers in meetings |
| Access | Source systems connect through an API (application programming interface) or database | Data arrives as emailed exports |
| Freshness | Tables refresh on a schedule nobody has to remember | Someone runs the update by hand each week or month |
| Quality | Automated checks flag gaps, duplicates and impossible values | Errors are found when a manager spots them in a report |
| History | Enough past records to show the patterns the use case depends on | Records start last quarter or were overwritten |
| Ownership | A business owner and an engineer are named for each data set | It is assumed to be the information technology team's problem |
| Lineage | Any number on a screen can be traced to its source rows | Nobody can explain how a figure was calculated |
Frequently asked questions
Do we need a data warehouse before we start with AI?
Not a full one. You need a reliable, automatically refreshed store for the data your first use case touches. That store can grow into a warehouse as more use cases arrive, which is cheaper and faster than designing everything up front.
How long does it take to make data ready for AI?
For one narrow use case, a first phase typically covers connecting sources, writing definitions and putting quality checks in place, and it ships working software every week. Organization-wide readiness is ongoing work, not a project with an end date.
Can AI clean our data for us?
Partly. Models are useful for matching messy names, classifying free text and flagging anomalies. They cannot decide what a field should mean or who is accountable for it; those are business decisions that have to be made by people first.
Should we fix all our data before any AI project?
No. Fix the data that one valuable use case needs, ship it, and repeat. Trying to clean everything first is how data programs run for years without producing anything leadership can use.
Who should own data quality, the business or the technology team?
Both, with distinct roles. The business owner decides what correct means and accepts or rejects the data; engineers build and run the pipelines and checks. When either role is missing, quality problems stay unresolved.
What if some of the data we need does not exist yet?
Say so on screen and keep going. Use a clearly labeled estimate or illustrative model where real figures are missing, and track the gap as a task with an owner. Hiding missing data behind confident numbers is what destroys trust in the whole system.
Sources
- Rand Corporation: The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (August 2024)
- Gartner: 30 percent of generative AI projects will be abandoned after proof of concept by end of 2025 (July 2024)
- Gartner: lack of AI-ready data puts AI projects at risk (February 2025)
- Gartner: data quality, why it matters and how to achieve it
About the author
Mohamed El Askary, Founder
Mohamed builds the systems Vantage ships, from data pipelines and models to the screens people use. He is a United States Air Force veteran and a Chapman University graduate in data science and business finance.