← Back to the learning paths
Foundations & researchAdvanced

Computer Vision

From pixels to a camera that works in a real, badly lit room.

Every computer vision demo works. That is the problem. The notebook shows crisp boxes around pedestrians in a bright test image, everybody applauds, and then the same model is pointed at the actual camera in the actual warehouse — mounted too high, aimed into a doorway that blows out at noon, watching people who are mostly occluded by pallets — and it collapses. Nothing in the tutorial prepared anyone for that, because the tutorial trained and tested on the same distribution, which is the one distribution you will never deploy into. This path is built the other way round. You start with the image itself, because a shocking number of production bugs are a colour channel in the wrong order or a resize that erased the object you were trying to find. Then labels, which are your real ceiling: two annotators disagree on a fifth of the boxes, and no architecture recovers information the labels never contained. Then the three tasks that make up the field — classification, detection, segmentation — plus tracking, which is most of what clients actually buy and almost none of what gets taught. You finish at the edge: quantisation, frame budgets, and the unglamorous truth that a model at fifteen frames per second on the device beats a better model that runs at three.

LevelAdvanced
Estimated duration6-9 months
Phases3

What you will learn

Image fundamentals & colour spacesDataset building and annotation qualityClassification & object detectionSemantic and instance segmentationMulti-object trackingFine-tuning pretrained backbonesQuantisation & edge inferenceCamera pipelines and frame budgets

Prerequisites

  • Solid Python and NumPy
  • Deep learning fundamentals: training loops, CNNs, fine-tuning
  • Access to a GPU for training and, ideally, one small edge device
  • Patience with data labelling — you will do more of it than you expect

Where it leads

  • Computer Vision Engineer
  • Perception Engineer
  • Edge AI Engineer
  • Applied Scientist (vision)

Phases

Phase 1 — The image and its labels

Master what you are actually feeding the model, and accept that your labels are your ceiling.

Estimated duration · 6-8 weeks

What an image actually is

The single most common production bug in this field is a preprocessing mismatch: you trained on RGB and the camera library hands you BGR, or you normalised with one set of statistics at training and another at inference. The model does not crash. It just gets quietly worse, and you spend two weeks blaming the architecture. Learn the pixel, the colour space, and above all what resizing destroys — the small distant object your client cares about most is exactly the one that a naive resize turns into four grey pixels.

Topics covered

Pixels, channels, bit depth and dynamic rangeRGB, BGR, HSV, grayscale — and why the order bitesResizing, interpolation, aspect ratio and letterboxingEXIF rotation: the metadata that silently flips your training setNormalisation statistics and keeping them identical at inferenceClassical operations still worth knowing: thresholding, morphology, edges

What you will build

  • Take a working model and break it three ways — channel order, wrong normalisation stats, EXIF rotation — and record the exact accuracy drop for each
  • Write a preprocessing module used identically by training and inference, and a test that fails if the two ever diverge
  • Measure how far your detector degrades as you shrink the input resolution, and pick the smallest size that still meets your requirement

Datasets, annotation and your real ceiling

Give two competent annotators the same thousand images and they will disagree on a meaningful share of the boxes — where exactly the truck ends, whether that shadow is a person, whether a half-visible object counts at all. That disagreement is your ceiling; the model cannot be more consistent than its labels. So the highest-leverage work in this entire path is not architectural, it is writing an annotation guide precise enough that two strangers produce the same file. Almost no team does this, and almost every team wonders why their metrics stalled.

Topics covered

Writing an annotation guide that resolves the ambiguous cases in advanceInter-annotator agreement, and measuring it before you train anythingLabel noise and how it caps your metricClass imbalance and the rare class that is the whole pointAugmentation that respects your task — and the flips that do notSplitting by scene, camera or day, never at random

What you will build

  • Label 200 images yourself, have someone else label the same 200, then compute and report your agreement score
  • Rewrite your annotation guide after that measurement and prove agreement improved on a fresh batch
  • Build a split by camera and by day, and show how much your reported score drops compared with a random split

Phase 2 — Detection, segmentation, tracking

Cover the three real tasks, and the fourth one that pays: following an object across time.

Estimated duration · 10-12 weeks

Detection, and the metric that hides the failure

Detection is where most vision products live, and mAP is where most vision products lie to themselves. It is a research metric that averages over thresholds and classes, which is a fine way to rank papers and a terrible way to answer the only question your client has: did it miss a person. Learn how a detector is assembled — backbone, head, non-maximum suppression — but spend at least as long learning to report per-class recall at the operating point you will actually ship, because that is the number that ends the argument in a meeting.

Topics covered

Bounding boxes, IoU and what "correct" meansOne-stage versus two-stage, anchors versus anchor-freeNon-maximum suppression and the crowded-scene failuremAP: how it is computed and what it concealsConfidence thresholds chosen from cost, not conventionSmall objects: the hardest common problem in the field

What you will build

  • Fine-tune a detector on your own labelled set and report per-class recall at a fixed operating point, not just mAP
  • Construct a crowded scene where non-maximum suppression deletes a true positive, then show the trade-off curve as you tune its threshold
  • Split your test set by object size and prove where your detector actually fails

Segmentation and tracking across time

A box tells you roughly where; a mask tells you exactly what, which matters the moment you need to measure an area or separate touching objects. But the thing clients actually buy is tracking: not "there is a person" but "this is the same person as three seconds ago". That identity is where the money and the difficulty both live — through occlusion, through people crossing, through someone leaving the frame and returning. Tutorials stop at detection because tracking is where it gets hard, which is exactly why knowing it makes you employable.

Topics covered

Semantic versus instance versus panoptic segmentationMasks, boundaries and why the edges are always wrong firstPromptable and zero-shot segmentation as a labelling acceleratorTracking by detection: association, motion models, identityOcclusion, ID switches and re-identificationMetrics for tracking, and why detection metrics do not transfer

What you will build

  • Use a promptable segmentation model to pre-label a dataset, then measure how much annotation time it actually saved after human correction
  • Build a tracker on top of your detector and count ID switches over a two-minute clip with real occlusions
  • Take the ten worst ID switches in your log, look at the frames, and write down the cause of each one

Phase 3 — Onto the device

Get the model out of the notebook and onto hardware that has a fixed frame budget and no excuses.

Estimated duration · 8-10 weeks

Pretrained backbones and the domain you did not train on

You will never train a backbone from scratch and you should stop feeling guilty about it — the compute and the data are not coming. Fine-tune, and put your effort where the leverage is: the gap between the internet photographs your backbone learned from and the specific camera, angle, and lighting you are deploying into. That gap, not the architecture, is why your model drops fifteen points on site. The fix is almost always more of the right data rather than a cleverer model, and accepting this early saves a quarter of wasted work.

Topics covered

Choosing a backbone by frame budget, not by leaderboard rankFreezing, unfreezing and how much data each strategy needsDomain shift: lighting, angle, sensor, weather, dustCollecting a small on-site set that outperforms a large generic oneZero-shot vision models: real uses and honest limitsHard-negative mining from the deployment footage itself

What you will build

  • Evaluate a model trained on public data against footage from one real camera, and quantify the domain gap in points
  • Label 300 frames from that camera, fine-tune, and show how much of the gap that closes
  • Mine hard negatives from a week of deployment footage and demonstrate the reduction in false positives

Edge deployment and the frame budget

Decide your frames per second before you choose an architecture, not after — teams that pick the model first spend the following month trying to make an impossible thing fit. On real hardware the model is often not even the bottleneck: decoding video, copying to the accelerator and drawing the output can eat more milliseconds than inference. Learn to profile the whole pipeline, learn what quantisation actually costs you in accuracy, and learn to say out loud that a model at fifteen frames per second beats a better one at three.

Topics covered

Setting the frame budget first, and letting it choose the modelExporting to a portable graph format and the operators that refuse to convertQuantisation: post-training versus quantisation-aware, and the accuracy you payPruning and distillation as alternatives to a smaller backboneProfiling the whole pipeline: decode, copy, infer, drawBatching, streaming and dropped frames as a design decision

What you will build

  • Write down a frame budget for a real use case, then justify your architecture choice against it in one page before writing any code
  • Quantise a trained detector, measure the exact accuracy you lost and the exact latency you gained, and state whether the trade was worth it
  • Profile an end-to-end camera pipeline on a device and produce a millisecond breakdown showing where the time really goes

Questions

Do I still need classical image processing, or has deep learning replaced it?

You need a few weeks of it, not a semester, and the reason is practical rather than nostalgic. Every pipeline you build will do classical operations whether you understand them or not — a resize, a colour conversion, a normalisation — and those are precisely where the silent bugs live. Beyond that, there are still problems where a threshold and a morphological operation solve in twenty lines what a neural network would need ten thousand labels to approximate: a fixed camera, controlled lighting, high contrast parts on a conveyor. Reaching for a model there is not sophistication, it is waste. The engineers worth hiring know which tool the problem deserves.

How much labelled data do I actually need?

Far less than you fear, and of far better quality than you were planning. Fine-tuning a pretrained backbone for a narrow task on a fixed camera often works with a few hundred well-chosen frames, and a few hundred images from the actual deployment camera will beat several thousand scraped from the internet every single time. The mistake is optimising for volume: teams label ten thousand easy frames of the obvious case and none of the doorway at noon, then wonder why the model fails exactly where it matters. Label the hard cases, label the failures your model produces in the field, and re-label anything two annotators disagreed on. Two hundred difficult frames are worth more than five thousand easy ones.

My detector scores well on my test set but fails on site. What happened?

Almost certainly one of three things, in descending order of likelihood. Your test set is not a test set: if you split randomly, frames from the same clip landed on both sides, and the model is being graded on images it has effectively already seen. Or your preprocessing differs between training and inference — a channel order, a resize, a normalisation constant — which costs you accuracy without ever raising an error. Or the site simply does not look like your data: different angle, different light, different lens, and the model was never given a chance. Check them in that order. It is nearly never the architecture, and the month spent trying a bigger model is the most common way to waste a quarter in this field.

Related paths

Deep LearningMachine LearningMLOps

Want to walk this path with a coach and a cohort?

Discover the 212AY Academy →