← Back to the learning paths
Foundations & researchIntermediate → Advanced

NLP & Transformers

Understand how machines read language — and why they read Arabic badly.

The uncomfortable truth of this field is that almost everything you read about natural language processing was measured on English, on clean text, on benchmarks whose test sets have probably leaked into training data. Then you point the same pipeline at a corpus of Moroccan customer messages — half darija, a third French, a scattering of Modern Standard Arabic, phone numbers written in Latin digits and Arabic ones in the same sentence — and the accuracy you were promised evaporates. This path exists because that gap is not a footnote. It is the work. You start where every serious NLP course should start and almost none does: with tokenisation, and specifically with what tokenisation destroys. An Arabic word carries a preposition, a definite article, a root, a pattern, a possessive suffix, all fused; a subword tokeniser trained mostly on English shreds it into meaningless fragments and charges you three times the tokens for the privilege. From there you build up honestly — representations, attention, the Transformer, fine-tuning — but always with the question of what breaks when the language is not English. The final phase is evaluation, because this is the discipline where self-deception is easiest: a good score on a contaminated benchmark, an F1 computed on annotations two people disagreed about, a metric that rewards fluent nonsense. If you finish this path you will be slower to claim victory than most people in the industry. That is the point.

LevelIntermediate → Advanced
Estimated duration6-9 months
Phases3

What you will learn

Tokenisation and subword algorithmsMorphology-aware preprocessing for ArabicEmbeddings and vector semanticsText classification and sequence labelling (NER)Transformer architecture and attentionFine-tuning, adapters and parameter-efficient methodsAnnotation design and inter-annotator agreementEvaluation, contamination and metric criticism

Prerequisites

  • Solid Python and comfort with arrays and tensors
  • Linear algebra and probability at working level, not proof level
  • Prior exposure to training a neural network end to end
  • Access to a GPU, rented or borrowed, for the fine-tuning phase

Where it leads

  • NLP Engineer
  • Applied Research Scientist
  • Machine Learning Engineer (language)
  • Computational linguist / low-resource language specialist

Phases

Phase 1 — Text is not data yet

Understand what happens between a sentence and a tensor, and everything the conversion silently throws away.

Estimated duration · 8-10 weeks

Tokenisation and what it destroys

Subword tokenisation is a compression scheme fitted to whatever corpus trained it, and that corpus was mostly English. Arabic pays twice: its morphology fuses meaning that gets shredded across fragments, and its script eats more tokens per word, so the same paragraph costs more and fits less. Darija pays a third time, because it has no standard orthography — the same word appears in five spellings, and to a tokeniser those are five different words.

Topics covered

Character, word and subword tokenisationBPE, WordPiece, SentencePiece and their biasesFertility: tokens per word across languagesArabic morphology: clitics, roots, patterns, diacriticsDarija orthographic variation and ArabiziCode-switching and mixed-script documentsNormalisation choices you can never undo

What you will build

  • Measure token fertility of the same translated paragraph in English, French, MSA and darija, and publish the ratio
  • Take a hundred darija messages and count how many distinct spellings you find for ten common words
  • Break a tokeniser: find real sentences where its segmentation makes a downstream task impossible

Representations: from counts to embeddings

An embedding is a claim that meaning has geometry, and it is a useful claim that is also partly false. You will build up from bag-of-words to contextual vectors and see exactly what each representation can and cannot encode — including the fact that a multilingual embedding space is usually an English space with other languages politely accommodated near the edges.

Topics covered

Bag-of-words, TF-IDF and why they are still competitive baselinesStatic embeddings and the analogy mythContextual embeddings and polysemyCosine similarity and what it actually measuresCross-lingual alignment and its failure casesBias encoded in the vector spaceDimensionality, pooling and sentence vectors

What you will build

  • Beat a fine-tuned model with TF-IDF on one real classification task, then explain honestly why it was possible
  • Probe a multilingual embedding space: show a pair of sentences that mean the same thing in French and darija but sit far apart
  • Build a semantic search over a mixed French–Arabic document set and document where it retrieves nonsense

Phase 2 — Transformers and adaptation

Understand the architecture well enough to debug it, and adapt a pretrained model without wasting a month of compute.

Estimated duration · 10-12 weeks

Attention and the architecture

Attention is a soft lookup table, and the Transformer is mostly that idea repeated with residual connections and normalisation holding it together. You will implement it once from scratch — not because you will ever write your own in production, but because you cannot diagnose a training collapse or a context-length problem from a library you have only ever called. The encoder–decoder distinction still matters, whatever the current fashion says.

Topics covered

Self-attention, queries, keys and valuesMulti-head attention and what heads actually learnPositional information and long-context limitsEncoder, decoder and encoder-decoder — and when each winsResidual streams, layer norm and training stabilityPretraining objectives: masked vs autoregressiveQuadratic cost and why context is not free

What you will build

  • Implement single-head attention from scratch and reproduce a library’s output on the same input, to numerical agreement
  • Visualise attention on an Arabic sentence and show where the tokeniser’s segmentation misleads the heads
  • Profile inference cost as context grows and plot the curve against your own prediction

Fine-tuning, and when not to

Fine-tuning is the answer to a narrower question than most people think. It teaches format, style and a specialised label space; it is a poor and expensive way to teach facts. The real constraint is never the GPU, it is the data: three hundred well-annotated examples beat thirty thousand scraped ones, and nobody wants to hear it because annotation is boring and renting a GPU feels like progress.

Topics covered

Task heads: classification, sequence labelling, span extractionFull fine-tuning vs adapters and low-rank methodsCatastrophic forgetting and how to notice itData quantity vs data quality, honestly measuredContinued pretraining for a low-resource dialectPrompting or retrieval as the cheaper answerReproducibility: seeds, splits and leakage between them

What you will build

  • Fine-tune a compact model for NER on Moroccan entities — brands, cities, street addresses — and beat a general model on your own held-out set
  • Run the same task with prompting, retrieval and fine-tuning, then report cost, latency and score side by side
  • Deliberately induce catastrophic forgetting, detect it with a control set, and mitigate it

Phase 3 — Evaluation and the multilingual reality

Learn to distrust your own numbers, and to build systems for languages the benchmarks ignore.

Estimated duration · 8-10 weeks

Metrics that lie

Every popular metric was designed for a purpose it has since outgrown, and the gap between what it measures and what you care about is where careers go quietly wrong. Accuracy on an imbalanced set, overlap metrics that reward fluent paraphrase, benchmark scores from test sets that leaked into pretraining. Underneath all of it sits annotation: if two competent annotators disagree a third of the time, no metric computed on their labels means anything.

Topics covered

Precision, recall, F1 and the imbalance trapOverlap metrics and fluent nonsenseBenchmark contamination and test-set leakageAnnotation guidelines and inter-annotator agreementError analysis by category, not by aggregateStatistical significance on small evaluation setsLLM-as-judge: where it helps, where it flatters

What you will build

  • Write an annotation guideline, have two people label two hundred items, and report the agreement score before any model result
  • Take a published benchmark claim and reproduce it on genuinely fresh data you collected yourself
  • Produce an error taxonomy for one model with counts per category and one fix per category

Low-resource languages and dialects

Darija is spoken by tens of millions of people and has almost no clean annotated corpus, no orthographic standard, and heavy borrowing from French, Amazigh and Spanish in the same breath. That is not a deficiency of the language, it is a deficiency of our datasets — and it is also an opportunity, because whoever builds the resource owns the ground. This step is about collecting data ethically, handling code-switching, and being honest that a model can be state of the art and still useless in Casablanca.

Topics covered

Building a corpus when none existsConsent, licensing and privacy in scraped textCode-switching and mixed-script normalisationTransfer from MSA and from French — and where it misleadsData augmentation and synthetic data’s limitsDialect identification as a task in itselfDeployment, drift and the cost of small models

What you will build

  • Collect and publish a small, cleanly licensed darija dataset for one task, with a documented annotation guideline
  • Ship a classifier that handles French–darija code-switching and report its score separately on monolingual and mixed inputs
  • Show a case where a high-scoring multilingual model fails on real Moroccan user text, and explain the mechanism, not just the symptom

Questions

Do large language models make classical NLP obsolete?

They made a lot of it unnecessary and a little of it more valuable than ever. Nobody should hand-build a sentiment classifier from scratch today when a prompt gets eighty percent of the way in an afternoon. But the last twenty percent is where the money is, and getting there requires exactly the classical skills: knowing what your tokeniser did to the input, designing an annotation scheme two people can agree on, computing a metric that is not lying to you, and recognising that your benchmark leaked. A large model is a very good default and a very bad excuse for not understanding your data. Also, at real volume, a small fine-tuned classifier is often cheaper by an order of magnitude and runs where a large model cannot — which matters more in practice than it does on a leaderboard.

What exactly makes Arabic and darija hard for these models?

Four things stack up. First, morphology: one Arabic word can carry what English spreads over five, so a tokeniser trained on English cuts through meaning rather than around it. Second, the script costs more tokens per unit of meaning, so you pay more and fit less context for the same paragraph. Third, diacritics are usually absent, which makes some words genuinely ambiguous without them. Fourth, and worst for darija specifically, there is no orthographic standard and no large clean annotated corpus — the same word appears in Arabic script and in Latin script with digits, in half a dozen spellings, all in one conversation. None of this is exotic or unsolvable. It just means published numbers do not transfer, and you have to measure on your own data. Most people skip that step and are then surprised in production.

How much mathematics do I actually need?

Enough to read a matrix multiplication and know why a softmax saturates — not enough to prove a convergence theorem. In practice: linear algebra to the level of comfort with shapes and projections, probability to the level of understanding a likelihood, and calculus only far enough to know what a gradient is and why it vanishes. That is a working level, and it is reachable in a couple of months of honest effort. The people who struggle in this path are almost never the ones with weak maths; they are the ones who cannot resist skipping the data work. You can survive fuzzy intuitions about attention. You cannot survive a training set you never looked at.

Related paths

Deep LearningLLM ApplicationsMachine Learning

Want to walk this path with a coach and a cohort?

Discover the 212AY Academy →