The Compounding Cost of Early Decisions
In technology, some decisions are easily reversed and some are not. Cloud provider, framework choice, database engine — these are expensive to change but technically possible. Data architecture, security model, and vendor lock-in structure are much harder to unwind.
AI program architecture decisions fall into the second category. The decisions you make in the first six months — about where your data lives, how you structure your model access, what level of vendor dependency you accept, how you organize your AI team — will shape and constrain your program for years.
I have seen enterprise AI programs spend 18 months rebuilding infrastructure that was designed wrong early. I have also seen programs that made careful early decisions scale cleanly from two use cases to thirty. The difference usually comes down to these ten decisions.
Decision 1: Model Access Strategy
The foundational question: do you build on hosted APIs, self-hosted open-weight models, or a hybrid?
Hosted APIs (OpenAI, Anthropic, Google, Azure OpenAI) are the right start for most organizations. Fast integration, no infrastructure overhead, access to frontier capability. The risks: cost at scale, data egress concerns, and vendor dependency.
Self-hosted models (Llama, Mistral, Qwen, etc.) require GPU infrastructure and operational expertise. They are the right choice when: (1) you have regulatory data residency requirements, (2) your inference volume makes hosted API costs prohibitive, or (3) you need complete control over model behavior.
Hybrid architecture — which most mature programs converge on — routes requests to the appropriate model based on task complexity, data sensitivity, and cost constraints. This is architecturally more complex but gives you the best of both worlds.
Make this decision explicitly and early. It drives your infrastructure choices, cost model, and security architecture.
Decision 2: Data Foundation First or Features First
This is the decision that most AI programs get wrong.
"Features first" teams build AI capabilities on top of whatever data infrastructure they have. They ship fast, demonstrate value, and then hit a wall when the AI cannot improve because the data is fragmented, inconsistent, or missing the signals the model needs.
"Data foundation first" teams invest in data quality, unified data models, and data pipelines before building AI features. They are slower to ship the first thing, but they build capabilities that compound — each AI feature makes the data better, which makes the next AI feature more capable.
If your organization's data infrastructure is in poor condition — multiple source-of-truth systems, significant data quality problems, no real-time event streaming — the AI program should begin with a data infrastructure investment, not an AI model investment.
Decision 3: Build vs. Buy for AI Infrastructure
There is a growing market of AI infrastructure tools: observability platforms (LangSmith, Braintrust), prompt management (Humanloop), vector databases (Pinecone, Weaviate, pgvector), agent frameworks (LangChain, LlamaIndex).
The build vs. buy decision for each layer should be made with a clear principle: buy infrastructure that is not differentiated, build where your intelligence is.
Observability, logging, and vector storage are infrastructure — buy them. The prompt engineering, the retrieval architecture, the evaluation criteria, and the domain knowledge encoded in your AI system are potentially differentiated — build and own them.
Decision 4: AI Team Structure
Three common models, each with different tradeoffs:
Centralized AI team: A dedicated AI team that builds and maintains all AI capabilities across the organization. High expertise concentration, slower delivery to individual business units.
Federated AI: AI engineers embedded in each product or business unit team. Fast delivery, risk of fragmentation and duplicated infrastructure.
Center of Excellence (CoE) model: A central team that sets standards, provides shared infrastructure, and advises on complex problems. Product teams own delivery. This is the model that scales best.
The CoE model is the right answer for most organizations beyond the startup scale, but it requires investment in platform thinking and documentation that many organizations underestimate.
Decision 5: Evaluation Infrastructure
You cannot improve what you cannot measure. Building AI capabilities without building evaluation infrastructure is building without a compass.
Evaluation infrastructure for LLM systems includes:
- A test dataset representative of production inputs
- Automated evaluation metrics (LLM-as-judge, task-specific metrics)
- Baseline benchmarks for all production models
- A process for running evaluations on every model or prompt change before deployment
- Human evaluation for qualitative assessment on a regular sample
This infrastructure should be built before the first production AI system ships. It is never convenient to build it afterward.
Decision 6: Governance Threshold by Risk Level
Not all AI use cases carry the same risk. A system that helps engineers write documentation carries different risk than a system that recommends clinical treatments or approves credit applications.
Define your risk taxonomy early:
- Low risk: AI that assists with information retrieval, content generation for internal use, or decision support for low-stakes decisions
- Medium risk: AI that generates customer-facing content, makes recommendations that influence significant decisions, or handles personally identifiable information
- High risk: AI that makes or heavily influences consequential decisions, operates in regulated domains, or handles sensitive protected data
For each tier, define the governance requirements: what review is needed before deployment, what monitoring is required, what documentation is mandatory. Apply these consistently.
Decision 7: Security Model
Design the security architecture before building the first AI system, not after. The questions to answer:
- What data can the AI access? Through what mechanisms? With what authorization controls?
- How do you prevent prompt injection? What guardrails are mandatory?
- Who can see the AI's inputs and outputs? What is the audit log retention policy?
- What data can go to hosted model APIs? What stays on-premises?
- How do you handle AI security incidents?
These questions are much cheaper to answer before you have 10 AI systems in production than after.
Decision 8: Observability and Cost Governance
AI systems have unique observability requirements — you need to monitor not just infrastructure health but output quality, user satisfaction, and cost. Build observability from the first system.
Cost governance is often neglected until the bill arrives. LLM costs at scale are significant and can grow rapidly. Implement:
- Cost tracking by team, product, and user segment from day one
- Budget alerts and hard limits on development and staging environments
- A cost review process as part of the AI change management workflow
Decision 9: Vendor Lock-in Management
AI vendor landscapes are evolving rapidly. The model you depend on today may be superseded in 12 months. The infrastructure vendor you choose may change pricing or discontinue a product.
Manage vendor dependency by:
- Abstracting model access behind an interface that can route to different providers
- Avoiding deep proprietary integration with any single vendor's ecosystem
- Evaluating new model options on a regular cadence
- Maintaining the capability to self-host models as a fallback for critical systems
Decision 10: The Make-vs-Wait Decision
The final decision — and the one CTOs often find hardest — is when to move and when to wait.
AI capabilities are improving at a pace that makes some problems worth waiting on. A capability that requires complex, expensive engineering today may be solved by a model update in six months.
But the organizational learning, the data infrastructure, the governance processes, and the team capabilities that make AI programs successful do not get built from watching. They get built from doing.
The right answer for most organizations: move on the use cases where the capability exists today and the ROI is clear. Build the organizational and data infrastructure that will let you capitalize on future capability improvements. Do not let perfect be the enemy of compounding.
