10 Interview Questions That Reveal Whether an AI Developer Can Ship to Production (Not Just a Notebook)

A 2024 Gartner survey found that 85 percent of AI projects fail to move from proof of concept to production deployment. The gap is rarely the algorithm. It is almost always the engineer. Developers who can tune a model inside a Jupyter notebook are plentiful; developers who can wire that model into a monitored, versioned, latency-aware production system are not.

These 10 questions are designed for founders and technical buyers who need to distinguish the two. Each question targets a specific production competency: data contracts, serving infrastructure, drift monitoring, rollback discipline, and cost awareness. Skip the whiteboard puzzle and ask these instead. For a broader set of technical screening questions, the ML interview questions guide covers the ML-specific layer in more depth.

This is the single most diagnostic question on the list. A production-ready developer will describe a pipeline: training run, evaluation gate, packaging (Docker or a model registry such as MLflow or BentoML), staging environment, canary or blue-green rollout, and a monitoring check before full traffic promotion. A notebook-only developer will describe a Jupyter file they emailed to a data team.

Listen for specifics. Named tools (FastAPI, Kubernetes, AWS SageMaker, Azure ML, Vertex AI) are positive signals. Vague language ("I pushed it to a server") is a flag. If they cannot name the serving framework, probe: "What handled inference?" The answer should include something real, not "Python script on an EC2".

The strongest answers mention what they monitored after deployment and what trigger would cause a rollback. That last detail separates engineers who shipped once from engineers who own what they shipped.

Data contracts, the formal agreement between the team producing data and the team consuming it, are a 2023 to 2026 industry standard in production ML organisations. Companies including Airbnb, Spotify, and Monzo have published their approaches publicly. A developer who has never encountered the concept is almost certainly working at a prototype level.

A strong answer will describe a schema definition (JSON Schema, Protobuf, or Great Expectations), the enforcement point (ingestion pipeline, CI check, or a data quality monitor), and what happened when a contract violation surfaced. The failure story is more valuable than the success story. It reveals whether the developer treats data integrity as a runtime concern or an afterthought.

Production AI systems have SLAs. A recommendation model on an e-commerce site may need to return results in under 100 milliseconds. A document extraction pipeline for a back-office workflow may tolerate five seconds. The key is that the developer should know the number and have shaped their architecture around it.

Developers who have only worked in notebooks answer this question with confusion. Developers who have shipped production systems answer immediately, often followed by an explanation of how they optimised: model quantisation, ONNX export, batching strategy, caching, or asynchronous serving. A developer who has had to hit a latency target knows what trade-offs they made to reach it.

This question surfaces engineering depth beyond the model. In a mature production deployment, the model is often the easiest piece. The hard parts are feature pipelines, schema evolution, authentication, access controls, cost management, and integration with existing software. A developer who answers "the model was always the main challenge" has likely not shipped into a real organisational context.

Strong answers point to a specific non-model challenge: integrating with a legacy ERP system, building a feedback loop that labels production data, or managing GPU cost overruns during a fine-tuning run. When evaluating the candidate's portfolio for this pattern, the Gen AI portfolio red flags post identifies the portfolio signals that distinguish demo builders from production engineers.

Model monitoring covers three distinct concerns: infrastructure health (latency, error rate, memory), data drift (the input distribution shifting from what the model was trained on), and prediction quality (the model's outputs degrading over time). A production-capable developer can speak to all three, even if they have relied on managed tooling such as Arize AI, WhyLabs, or AWS Model Monitor.

The question to ask as a follow-up: "What is your alerting threshold for data drift and what action does it trigger?" Notebook developers have no answer. Production engineers describe a specific metric (Population Stability Index, Jensen-Shannon divergence, or a custom statistical test), a threshold, and a runbook: retrain, roll back, or flag for human review. The existence of a runbook is the tell.

Agentic systems, those that chain LLM calls with tool use, retrieval, and decision logic, introduce production challenges that single-model deployments do not: non-determinism, cost per session, latency stacking, and error propagation across steps. The 2025 Sequoia AI report noted that orchestration failures (not model failures) were the primary source of production incidents in LLM-powered applications.

A developer with real agentic production experience will describe how they handled tool failure (retry logic, fallback paths), how they capped token spend per session, and how they logged intermediate reasoning steps for debugging. If they have only built agents in LangChain tutorials, the architecture story will be shallow. For more patterns that expose weak agentic experience, the AI agent hiring red flags guide is worth reading before the interview. Organisations hiring for this capability specifically should engage AI agent development services from a developer who has shipped these systems.

Continuous integration for ML projects requires more than unit tests. It requires data validation, model evaluation against a holdout set, and a gate that prevents a regressing model from reaching production. Tools used in mature ML CI include DVC for data versioning, MLflow for experiment tracking, and GitHub Actions or CircleCI for pipeline orchestration.

Ask: "What would cause your pipeline to fail and block a release?" A developer working at production maturity will list multiple gates: data schema validation failure, evaluation metric below threshold, latency regression on the staging benchmark, and security scan. A developer who describes only "unit tests passing" has not shipped ML systems that carry real business risk.

This question tests whether the developer treats a model as a living system or a static artefact. Production conditions change: user behaviour shifts, upstream data pipelines change schema, seasonal patterns emerge, regulatory requirements alter what the model can use. A developer who has only trained models in controlled settings will struggle to describe a reactive retraining cycle.

Strong answers describe a specific trigger (a drift alert, a business metric decline, a data pipeline schema change), the decision to retrain vs. fine-tune vs. prompt-engineer around the gap, and the validation process before the new version went live. Developers working in custom AI model training at production scale treat this cycle as routine, not exceptional.

This question is not a trap. The developers who cannot answer it are the ones to be concerned about, because the likely explanation is that they have never shipped anything expensive enough to break. Production AI systems accumulate cost in GPU compute, API calls, vector database queries, and human review queues. A developer who has never had a cost incident has not owned a production system under load.

Common honest answers: a GPU training job left running over a weekend, an embedding model called per-request instead of cached, an LLM prompt that triggered a token spike on a viral traffic event, or a retraining pipeline that processed duplicate data and produced a worse model. The quality of the answer is less important than the specificity. Vague answers ("I always tried to be careful") indicate limited production exposure.

This question reveals documentation discipline, system thinking, and professional maturity. A developer who cannot describe their handover process has likely never been in an engagement where continuity mattered. Production AI systems need runbooks, data flow diagrams, retraining procedures, alert escalation paths, and dependency lists.

The strongest answers reference specific artefacts: a README that describes environment setup, a model card that documents training data, intended use, and known failure modes, and a monitoring dashboard the receiving team can read without explanation. This question is also directly relevant to engagements where a freelancer builds a system that an in-house team will inherit, a pattern that carries its own distinct set of handover considerations.

Production AI is not harder than notebook AI because the maths changes. It is harder because the responsibilities multiply: data integrity, serving reliability, cost governance, monitoring, and handover documentation all belong to the engineer who owns the system. The 10 questions above are designed to surface those responsibilities quickly, before a contract is signed.

If the interview answers are strong across deployment, monitoring, and handover, you are talking to someone who ships. If the answers are vague, notebook-centric, or unable to name real tools, the gap between demo and production is yours to carry. To work with an engineer who has shipped production AI across NLP, generative AI, and agentic systems, explore what it means to hire an AI developer with a verified production track record.