LLM Applications
Close the gap between a demo that impresses and a product that holds.
A working LLM demo takes an afternoon. A working LLM product takes months, and the gap between the two is where most teams stall. This path is about that gap. You start with the API surface properly — not just chat completion, but structured output, function calling, streaming, and the retry semantics that keep a request alive when a provider hiccups. Then retrieval, where the honest lesson is that RAG is an information retrieval problem wearing a machine learning costume: your chunking, your metadata and your ranking decide answer quality, while the vector database you agonised over decides almost nothing. You will build evaluation before you build features, because an LLM app without a test set degrades silently and nobody notices until a customer does. And you will treat cost and latency as design constraints from day one rather than a cleanup task, because they determine which architectures are even possible — a pattern that is elegant at ten requests a day is bankruptcy at ten thousand. What you will not get is a tour of frameworks. Frameworks churn every quarter and abstract away exactly the parts you need to see when something breaks. Knowing why a call failed, what is safe to cache, and when to stop retrying does not expire.
What you will learn
Prerequisites
- Comfortable writing Python or TypeScript
- Basic HTTP APIs and JSON
- Git and a real code editor
- Prompt engineering fundamentals
Where it leads
- LLM Application Engineer
- AI Product Engineer
- RAG / Search Engineer
- Full-stack developer with an AI specialisation
Phases
Phase 1 — The API surface
Stop treating the model as a chatbot. Treat it as an unreliable remote function you must design around.
Estimated duration · 6-8 weeksCalling the model like an engineer
Every LLM call is a network request to a service that is slow, rate-limited, occasionally wrong and billed by the token. Once you internalise that, half of your architecture decisions make themselves. You will learn request anatomy, token accounting, backoff that does not amplify an outage, and the timeouts that stop one slow call from taking down a page.
Topics covered
What you will build
- Write a client wrapper with timeout, jittered backoff and a hard retry ceiling — prove it with a fault-injection test
- Instrument every call with tokens in, tokens out and latency, then produce a one-page cost report for a week of traffic
- Take a live endpoint down deliberately and show your app degrades to a useful message instead of a stack trace
Structured output and function calling
The moment your app needs to do something with the answer, prose becomes a liability. Structured output turns the model into a component you can compose. The subtlety is that a schema is a prompt: field names, enums and descriptions change the output as much as any instruction, and a model that picks the wrong tool is usually reading a bad tool description, not thinking badly.
Topics covered
What you will build
- Convert one free-text feature to a validated schema and drive a real UI component from it
- Build a validation layer that logs every rejected payload, then fix the top failure by editing the schema, not the prompt
- Give the model four tools with deliberately vague names, measure wrong-tool rate, rename them and measure again
Phase 2 — Retrieval & context
Make the model answer from your data — and make it say so when it cannot.
Estimated duration · 8-10 weeksRAG that actually retrieves
Almost every bad RAG system is bad at retrieval, not at generation, and almost every team debugs the generation. Evaluate the retriever on its own — if the right passage is not in the top results, no prompt will save the answer. Hybrid search and a reranker beat clever prompting nearly every time, and cost a fraction of the effort.
Topics covered
What you will build
- Build a 50-query retrieval test set with known correct passages and report recall@5 before touching any prompt
- Ship hybrid search on one corpus and show the recall delta against vector-only
- Break your own chunker: find three documents where it splits mid-argument, then fix the strategy and re-measure
Grounding, citations and refusal
A confident wrong answer costs more than no answer. The engineering goal is a system that points at the sentence it used and declines when the context does not support a claim. Refusal is a feature, and it is the one product managers resist and users trust. You will also handle the case nobody plans for: two documents that disagree.
Topics covered
What you will build
- Ship an answer UI where every sentence links to the exact source span that supports it
- Add 15 unanswerable questions to your eval set and get the system to refuse all of them without becoming useless on the rest
- Enforce per-user permissions at retrieval time and write the test that proves a user cannot surface a document they do not own
Phase 3 — Product
Ship something people use, then keep it good and affordable while they use it.
Estimated duration · 8-10 weeksStreaming, latency and the feel of the app
Users do not experience total latency, they experience waiting. Streaming is a product decision disguised as a transport detail: it makes a slow system feel fast, and it makes a wrong answer arrive faster. Know when not to stream — nobody wants to watch a database write appear token by token.
Topics covered
What you will build
- Cut time-to-first-token by half on your slowest screen and prove it with before/after percentiles, not averages
- Stream a structured object and render each field the instant it is complete, without breaking on a truncated stream
- Implement cancellation end-to-end and measure the tokens you stop paying for over a week
Evaluation, cost and shipping
This is the step that decides whether your app is alive in a year. An eval suite in CI is the only thing standing between you and a silent regression shipped on a Friday. Cost per successful task — not cost per call — is the metric that tells the truth, because a cheap model that needs three attempts is not cheap.
Topics covered
What you will build
- Wire an eval suite into CI so a pull request that drops quality below a threshold cannot merge
- Cut cost per successful task by 40% using caching and routing, with your eval score unchanged — publish both numbers
- Run a real rollback: ship a deliberately worse version behind a flag, detect it from your dashboards, revert in under five minutes
Questions
Do I need a vector database?
Almost certainly not on day one. A vector extension on the database you already run will carry you further than most teams expect, and keeping your documents and their permissions in one place removes an entire category of sync bugs. The vector store is rarely the bottleneck: chunking, metadata and ranking are. Teams that migrate to a dedicated store before measuring recall usually discover their retrieval quality did not move, because the problem was never the index. Add one when your own measurements — scale, latency, or a filtering pattern you cannot express — tell you to.
Is RAG obsolete now that context windows are enormous?
No, and the argument confuses "it fits" with "it works". Stuffing a whole corpus into context is slower, far more expensive per request, and less precise — accuracy degrades when the relevant passage is buried among thousands of irrelevant ones. It also breaks the things a real product needs: per-user permissions, freshness, and the ability to say which document an answer came from. Large windows are genuinely useful, but they mostly change the shape of the problem: you retrieve fewer, larger chunks instead of many small ones. Long context hides a weak retriever, it does not repair one.
Which framework should I learn first?
None. Build your first application with plain HTTP calls and your own hundred lines of glue. It is genuinely not hard, and it teaches you where the seams are: what a retry costs, what the model actually receives, where the latency goes. Developers who start inside a framework can ship a demo in a day and then cannot explain a production bug, because the interesting part is behind an abstraction they never opened. Once you have built it raw, adopt a framework deliberately — you will evaluate it on what it hides rather than what it advertises, and you will keep the option of walking away when it stops fitting.
Related paths
Want to walk this path with a coach and a cohort?
Discover the 212AY Academy →