Deep Learning
Understand neural networks well enough to debug them at 2am.
Deep learning is taught backwards almost everywhere. Courses spend eight weeks on architecture diagrams and twenty minutes on the training loop, then graduates discover that architecture is the part you copy and the loop is the part that ruins your month. The truth is unromantic: most neural network bugs are shape errors, silent broadcasts, a forgotten evaluation mode, or a learning rate off by a factor of ten. Your model does not fail with an exception; it fails by converging to something plausible and wrong. This path is organised around that. You begin with tensors and autograd, and you implement backpropagation by hand exactly once — not to reinvent it, but so that a gradient stops being magic and starts being arithmetic you could check on paper. Then you cover the architectures that earned their place: convolutions and the inductive bias that makes them work on images, and the road from recurrence to attention, because you cannot understand why Transformers won without understanding what they replaced. The last phase is the one professionals actually live in: making training converge, regularising honestly, and fine-tuning a pretrained model instead of burning a fortune training from scratch. Almost nobody trains from scratch. Almost everybody should stop pretending they will.
What you will learn
Prerequisites
- Comfortable Python; you can debug someone else’s code
- Machine learning fundamentals: validation, overfitting, metrics
- Linear algebra basics — matrix multiplication must feel routine
- Access to a GPU, even a rented one, for the later phases
Where it leads
- Deep Learning Engineer
- Research Engineer
- Computer Vision / NLP Engineer
- Applied Scientist
Phases
Phase 1 — Tensors and gradients
Make gradients boring. Once backprop is arithmetic, the rest of the field opens up.
Estimated duration · 10-12 weeksThe tensor, the shape, and the silent bug
Ninety percent of the time you spend stuck in your first year will be shape errors, and the cruel ones do not raise an exception at all: broadcasting quietly turns a (32, 1) against a (32,) into a (32, 32), your loss becomes an average over nonsense, and training proceeds smoothly to a useless model. You will learn to annotate shapes as you write, to distrust any operation you did not think about, and to treat the device and dtype of every tensor as facts you check rather than assume.
Topics covered
What you will build
- Write ten tensor operations whose shapes you predict on paper first; log every case where you were wrong
- Build a deliberate broadcasting bug that trains without error, then write the assertion that catches it
- Profile GPU memory on one training step and account for where each allocation came from
Backpropagation, by hand, once
You should implement a small network and its backward pass from scratch exactly once in your life, and then never again. The point is not the code, which will be worse than the library. The point is that afterwards, a vanishing gradient stops being a phrase you repeat and becomes a chain of multiplications you can picture shrinking. Everyone who skips this ends up cargo-culting fixes off forums; everyone who does it can reason about a new architecture in an afternoon.
Topics covered
What you will build
- Implement a two-layer network with manual forward and backward passes in NumPy, and verify every gradient numerically
- Reproduce your NumPy network in a framework and confirm the losses match to four decimals on the same seed
- Break initialisation deliberately (all zeros, then far too large) and document exactly how each failure looks in the loss curve
Phase 2 — Architectures that earned their place
Understand what each architecture assumes about its data — that assumption is the whole design.
Estimated duration · 12-16 weeksConvolutions and the bias that makes them work
A convolutional network is not magic; it is a hard-coded belief that nearby pixels are related and that a cat is a cat wherever it sits in the frame. That belief is why a CNN needs far less data than a fully connected network on the same images. Learn to see architecture as encoded assumptions and you can predict where a model will fail: a CNN has no idea that the top of an image is the sky, which is precisely why it will confidently classify an upside-down photograph.
Topics covered
What you will build
- Train a small CNN and a fully connected net of equal parameter count on the same images; report the gap and explain it in terms of inductive bias
- Visualise the receptive field of a chosen neuron and confirm it matches your arithmetic
- Forget to call eval mode on purpose, measure the accuracy drop caused by batch norm, and write down the symptom so you recognise it later
From recurrence to attention
You will not build RNNs professionally, and you still need to understand them, because the Transformer is an answer to a question that recurrence asked badly. Sequential processing means a long dependency must survive hundreds of multiplications; it does not. Attention discards recurrence entirely and lets every position look at every other in one step — which is why it parallelises, why it scales, and why it costs quadratic memory. Understand that trade and the entire modern landscape becomes legible rather than a list of names.
Topics covered
What you will build
- Implement single-head attention from scratch and verify your output against a framework implementation on identical inputs
- Train an RNN and a small Transformer on the same sequence task, then plot accuracy against sequence length to show where recurrence collapses
- Extract and visualise the attention weights on one sentence, and state honestly which of your interpretations the data actually supports
Phase 3 — Training and fine-tuning
Make training converge on purpose, then stop training from scratch and start adapting.
Estimated duration · 12-16 weeksThe training loop is the job
Before you touch a real dataset, overfit a single batch to near-zero loss. If you cannot, your loop is broken and no architecture will save you — the label is misaligned, the loss is wrong, or the gradient is not reaching your weights. This one test catches more bugs than any amount of staring at code. After that, learning rate is the hyperparameter that matters, and most of the rest is folklore repeated by people who never ran the ablation.
Topics covered
What you will build
- Overfit a single batch to near-zero loss on three different tasks; document what was broken the times it failed
- Run a learning-rate range test and produce the curve that justifies the value you chose
- Build a training script that resumes correctly from a checkpoint mid-epoch and prove it by killing the process
Regularisation and fine-tuning what already exists
More data beats a better architecture, and augmentation is the cheapest data you will ever get — but only augmentation that preserves the label, which is why flipping a photograph of a road sign can quietly destroy your model. Then accept the economics: you will not train a foundation model, you will adapt one. Fine-tuning is its own discipline, with its own failure — catastrophic forgetting, where your model gets excellent at your five thousand examples and forgets everything that made it worth starting from.
Topics covered
What you will build
- Fine-tune a pretrained model on a domain of your own and beat a from-scratch model trained ten times longer
- Find one augmentation that damages your task, prove the damage with numbers, and explain why the label did not survive it
- Measure catastrophic forgetting: score your fine-tuned model on the original task before and after, and report the loss
Questions
Do I really need a GPU, or can I learn this on a laptop?
Phase one runs comfortably on a laptop, and you should keep it that way: a slow machine forces you to think before you launch, which is a habit worth acquiring. From phase two onward you need acceleration, but you almost certainly do not need to own it. Rented compute by the hour is the sane default, and the discipline it imposes is a feature — when the clock is running you stop launching hopeful jobs and start reading loss curves. Debug on a tiny subset locally, always, and only send a job to a GPU once it has already trained correctly on ten examples on your own machine.
Is it still worth learning RNNs when Transformers won?
You should spend days on them, not months, and the purpose is diagnostic rather than practical. The Transformer is an answer, and an answer you cannot trace back to its question is a piece of trivia. Recurrence teaches you viscerally what a long-range dependency costs, why gradients die across many steps, and why LSTMs bolted on gates to keep a signal alive. Once you have felt that, attention stops being an arbitrary block from a diagram and becomes an obvious move. Skip it and you will be able to use a Transformer but not reason about one, which is precisely the gap that shows up in an interview and in production.
My model trains, the loss goes down, but the results are bad. Where do I even start?
In order, and without skipping: look at the actual inputs your model receives, after every transform, with your own eyes — most of the time the bug is visible there and nowhere else. Then confirm your labels are aligned with those inputs; an off-by-one in a data loader trains perfectly and predicts nothing. Then try to overfit a single batch: if the loss will not reach near zero, the problem is mechanical, not statistical, and no amount of tuning will fix it. Only after all three do you consider architecture, and by then you usually will not need to. This ordering is unglamorous and it is what separates people who ship models from people who tweak them for six weeks.
Related paths
Want to walk this path with a coach and a cohort?
Discover the 212AY Academy →