The Square Peg Problem
Most engineering organizations are structured around a simple principle: teams own services. The payments team owns the payments service. The search team owns the search service. Clear ownership, clear accountability, well-understood boundaries.
AI does not fit this model. An AI capability might improve search, payments, customer support, and fraud detection simultaneously. It requires data from across the organization. It needs evaluation expertise that does not exist in any single team. And it evolves on a timeline dictated by external model improvements, not internal sprint cycles.
Companies that try to squeeze AI into existing team structures get one of two bad outcomes: a centralized AI team that becomes a bottleneck, or distributed AI efforts that reinvent the wheel in every team.
The Three Models We See
Model 1: Central AI Team. A dedicated team that handles all AI development and supports other teams through an internal consulting model. Pros: consistency, shared infrastructure, concentrated expertise. Cons: becomes a bottleneck, disconnected from product context, prioritization conflicts between consuming teams.
Model 2: Embedded AI Engineers. AI engineers sit on product teams and build AI features within their team's domain. Pros: deep product context, fast iteration, aligned incentives. Cons: inconsistent practices across teams, duplicated infrastructure, limited knowledge sharing.
Model 3: Platform plus Embedded (the one that works). A small central platform team builds shared infrastructure: evaluation tools, model orchestration, prompt management, monitoring, and cost optimization. Embedded AI engineers on product teams use this platform to build product-specific AI features. The platform team does not build product features. The embedded engineers do not build infrastructure.
Why Model 3 Wins
- The platform team creates leverage. Instead of every team building their own evaluation framework, one team builds it well. This is the same logic that led to platform engineering for DevOps, now applied to AI.
- Embedded engineers maintain context. They understand users, workflows, and product constraints intimately. This context is essential for building AI features that actually work in production.
- The boundary is clear. Infrastructure is centralized. Product intelligence is distributed. Nobody fights over ownership because the domains are distinct.
The right AI org structure separates the platform from the product. Centralize the tools. Distribute the intelligence. The platform team builds the kitchen. The product teams cook the meals.
How to Transition
Start by identifying the infrastructure that multiple teams are building independently: evaluation datasets, prompt management tools, model routing logic. Consolidate that into a platform team. Then ensure every product team with AI ambitions has at least one embedded AI engineer who can move fast without waiting for a central team. This transition takes 6 to 12 months but pays dividends for years.