The Three Bad Defaults
When companies decide to "do AI," they typically organize in one of three ways, all of which are wrong:
The centralized AI team. A dedicated AI/ML group that serves the entire organization. This sounds efficient but creates a bottleneck. The AI team becomes a service bureau, overwhelmed with requests from business units, and disconnected from the product context that determines whether AI features actually succeed. We have seen centralized AI teams with 18-month backlogs while the business units they serve grow increasingly frustrated.
The embedded model. Data scientists and ML engineers are distributed across product teams. This solves the context problem but creates fragmentation. Each team builds its own infrastructure, makes its own vendor choices, and develops its own evaluation practices. You end up with 10 different ways to deploy a model, none of them well-maintained.
The research lab. A separate AI research group focused on advancing the state of the art. This is appropriate for companies like Google DeepMind and Anthropic, where AI research is the product. For everyone else, it creates a group of talented people doing interesting work that never ships.
The Structure That Works
The model we have seen succeed at scale has three layers:
Layer 1: AI Platform Team (centralized). This team owns the shared infrastructure: model serving, evaluation frameworks, monitoring, data pipelines, and vendor management. They do not build products. They build the platform that product teams use to build AI-powered features. Think of them as the AI equivalent of a developer platform team.
Typical composition: 3-5 ML infrastructure engineers, 1-2 data engineers, 1 platform product manager.
Layer 2: AI Product Engineers (embedded). These are engineers who sit within product teams and build AI features using the platform. They are not ML researchers. They are product engineers who are fluent in AI. They know how to prompt models effectively, build evaluation datasets, design user experiences for probabilistic outputs, and integrate AI capabilities into existing products.
This is the fastest-growing role in AI and the hardest to hire for. The best AI product engineers come from product engineering backgrounds with strong product intuition, not from ML research.
Layer 3: Applied Research (small, focused). A small team of 2-3 senior ML engineers who investigate techniques that could give the company a meaningful edge: custom fine-tuning, novel evaluation approaches, or proprietary model architectures for specific use cases. This team has a clear charter and a requirement to demonstrate production impact within each quarter.
Key Principles
- The platform team is a force multiplier, not a gatekeeper. If product teams are waiting for the platform team to build features, the structure is failing. The platform should enable self-service with guardrails.
- AI product engineers report to product team leads, not the AI organization. Their success is measured by product outcomes, not model metrics. This keeps AI work grounded in user value.
- The applied research team has a production mandate. Interesting research that does not ship is not valued. Every research initiative must have a clear path to product impact.
- Invest in internal mobility. Product engineers who want to learn AI should have a clear path to becoming AI product engineers. The best talent for these roles often exists inside your organization already.
This structure scales from 50-person startups to 10,000-person enterprises. The ratios change, but the principles hold. Get the team structure right, and you will ship more AI in six months than most companies ship in two years with twice the headcount.