MLOps
Put AI in production and keep it there — through drift, incidents and invoices.
Most AI projects do not fail at the model. They fail three months after launch, quietly, when nobody can say which version is running, why the numbers moved, or who to call. MLOps is the discipline that prevents that, and it is unglamorous on purpose. This path treats a model as what it really is in production: a dependency with a data contract, a cost, a latency budget and a decay rate. You start with the boring foundation — reproducibility. If you cannot rebuild last month’s model from the repository alone, you do not have a system, you have a story. Then deployment: packaging, rollout, rollback, and the uncomfortable truth that a shadow deployment will teach you more in a week than any offline metric. Then monitoring, where this path takes its strongest position: model accuracy is the last signal to alert you and the first thing everyone watches. Input distribution and data quality move first, so that is where your alerts belong. You will handle drift, retraining decisions, and the question people avoid — when to retire a model. And you will run a real incident, because writing a postmortem for a system whose correctness is statistical teaches more in one afternoon than a month of reading. Cost is a first-class metric throughout, not a finance surprise discovered at renewal.
What you will learn
Prerequisites
- Python and Git in daily use
- Some ML or LLM application experience
- Containers and the command line
- Basic exposure to a cloud provider
Where it leads
- MLOps Engineer
- ML Platform Engineer
- AI Reliability Engineer
- Data Engineer with production ML scope
Phases
Phase 1 — Reproducibility & packaging
Be able to rebuild, identify and roll back anything you ship — from the repository alone.
Estimated duration · 8-10 weeksIf you cannot rebuild it, you do not own it
Ask a team which exact data produced the model currently serving customers. The silence that follows is the problem this step solves. Code versioning is solved; data versioning is where teams cheat, and it is the half that decides whether a debugging session takes an hour or a fortnight. Lineage from a single prediction back to the rows that trained it sounds excessive until a regulator or an angry customer asks.
Topics covered
What you will build
- Rebuild a model from a commit hash alone and show the metrics match the original within a stated tolerance
- Trace one production prediction back to the exact training snapshot and dataset version that produced it
- Write the data contract for one input feed, then break it on purpose in staging and prove the pipeline refuses to train
Packaging and serving
The serving decision shapes everything downstream, and most teams default to a real-time endpoint they never needed — a nightly batch job is cheaper, simpler and easier to fix, and a surprising share of use cases tolerate it. The classic killer here is training/serving skew: the same feature computed two different ways in two different codebases. It never shows up in your offline metrics and always shows up in production.
Topics covered
What you will build
- Serve one model three ways — batch, online, shadow — and write the one-page memo choosing between them with numbers
- Build a skew detector that compares training-time and serving-time feature values on live traffic and alerts on divergence
- Take an over-provisioned endpoint and halve its cost by right-sizing, with a latency percentile table proving nothing broke
Phase 2 — Deployment & CI/CD
Make releases boring: automated, reversible, and observed before they reach everyone.
Estimated duration · 8-10 weeksReleasing without holding your breath
Shadow deployment is the most underused technique in this field: run the new model on real traffic, serve nothing, compare. It costs one extra inference and buys you evidence no offline benchmark can. And rollback deserves to be a rehearsed path, not an improvisation at midnight — if reverting a model requires a person who is asleep, you do not have rollback, you have hope.
Topics covered
What you will build
- Run a new model in shadow for a week and produce the comparison report that decides go/no-go
- Rehearse a rollback with a stopwatch until any team member can revert production in under five minutes
- Ship a canary at a small traffic share, define the automatic abort rule in advance, and trigger it deliberately
Pipelines and CI/CD
A test suite that checks your code but never your data is testing the easy half. The transformations are where the bugs hide, and a silently corrupted column will happily train a model that passes every unit test. The other decision is retraining: scheduled retraining is a habit, triggered retraining is a strategy. Most teams retrain on a calendar because it is easy to justify, then retrain on nothing when the calendar becomes noise.
Topics covered
What you will build
- Add data quality tests to one pipeline, then corrupt a column in staging and prove the build fails before training starts
- Put an evaluation gate in CI and open a pull request that legitimately fails it
- Replace one scheduled retrain with a trigger tied to a measured signal, and defend the threshold you chose in writing
Phase 3 — Monitoring, drift & incidents
Notice degradation before your users do, and know exactly what to do at 3am.
Estimated duration · 8-12 weeksMonitoring what actually moves first
Accuracy dashboards are comforting and nearly useless as an alarm, because ground truth arrives weeks late — by the time the curve bends, you have been wrong for a month. Inputs move first. Watch distributions, null rates and cardinality, and treat an upstream schema change as the incident it is. Also: every drift alert you cannot act on trains your team to ignore the next one.
Topics covered
What you will build
- Instrument input monitoring on one production model and catch a real upstream change before any accuracy metric moves
- Break the average on purpose: find a segment where your model is far worse than the headline number and add a per-segment alert
- Audit your alerts, delete every one nobody acted on last quarter, and defend each deletion
Cost, incidents and retirement
Statistical systems fail differently: there is no exception, no stack trace, just answers that are quietly worse. Your runbook must therefore answer a question ordinary software never asks — how do you decide the system is broken? And retirement is the taboo: models outlive their usefulness because nobody is rewarded for switching one off, and every one you keep is permanent cost and permanent risk.
Topics covered
What you will build
- Publish cost per prediction for every model you run, then cut the most expensive one by a third without a quality regression
- Write a runbook whose first section is "how to tell if this is actually broken", then hand it to someone who has never seen the system and have them run a drill
- Retire one model: prove it is no longer earning its cost, migrate its consumers, and switch it off
Questions
Is MLOps just DevOps with extra steps?
The tooling rhymes; the failure modes do not. Ordinary software is deterministic and breaks loudly — an exception, a stack trace, a red dashboard. A model has a dependency you do not control and cannot version by asking nicely: the world. It degrades silently, correctness is statistical rather than binary, and the ground truth that would tell you it is wrong arrives weeks after the damage. That changes what you build. You need data tests alongside code tests, shadow deployments because staging cannot reproduce real inputs, drift monitoring because nothing throws an error, and a runbook whose first question is "is this actually broken?". DevOps skills transfer. The instincts do not.
Do I need MLOps if I only use LLM APIs and never train anything?
More than you think, and often in a worse position. You inherit every operational burden without owning the model. The provider updates something under you and your behaviour changes without a single commit on your side — that is a dependency with no changelog you control. Your prompts are artifacts that need versions, diffs and rollback, and most teams keep them in a string literal edited on a Friday. Your cost is variable per request rather than fixed per server, which finance has never had to model before. Your evaluation suite is your only regression test, because there are no unit tests for tone. The vocabulary shifts; the discipline is identical.
Which tools and platforms should I learn?
Learn the concepts and build the minimal version by hand first. A model registry can start as a folder with a naming convention and a manifest; an eval gate can start as a script in CI; lineage can start as a column in a table. Doing it manually for one real system teaches you what the problem actually is, so that when you evaluate a platform you can tell whether it solves your problem or merely renames it. Engineers who learn a platform first end up fluent in one vendor and helpless when the company picks another — and the vendor landscape here churns faster than almost anywhere in software. The concepts are stable: reproducibility, gates, observation, reversibility. Nothing in that list has changed in a decade.
Related paths
Want to walk this path with a coach and a cohort?
Discover the 212AY Academy →