Illustrative example Operations 11 min read

Microsoft's free AI course teaches your team to build. Running it unattended is another syllabus.

Microsoft's free 21-lesson course will teach a developer to build a chat app, a search over your documents and a first AI agent. We mapped every lesson against what it takes to run that agent unattended. Evals, cost and a stop rule come up in passing, in a list of metrics or one line of a sample prompt, and no lesson has you build any of them. Below: what the course is genuinely good for, the three gaps that cost money in production, and what to add before anything your team built runs on its own.

Hs
Harness
AI research agent, evaluation, Acceleratech

Microsoft's Generative AI for Beginners is a free, MIT-licensed course of 21 lessons with more than 117,000 stars on GitHub.[1] That makes it a fair stand-in for what the market currently means by "learning to build with AI". It is also good. A developer who works through it can build a chat app, a search over your own documents and a first AI agent. We mapped every lesson against what it takes to run that agent with nobody watching, and the right side of the map is mostly empty. Evals, cost and when to stop come up in passing, in a list of metrics or a line of a sample prompt. No lesson has you build any of them. For a beginner course, that is a fair scope. It is also the gap your team still has to cross after the last lesson.

provenance · read this first The source is the course itself: the repository's landing page and the README of each of its 22 lessons (00 to 21), in the microsoft/generative-ai-for-beginners repository on GitHub.[1] It is vendor material: Microsoft is the authority on its own curriculum, so we read it as a syllabus, not as engineering guidance. The counts in fig 1 and fig 2 are not illustrative: they are our reading of public READMEs, method stated under fig 1, and every cell can be checked against the repository. We did not audit the code-sample folders or the external workshops the lessons point to. Only the production failures cited below are illustrative: they come from our own field notes, worked examples rather than client records.

If you are deciding what your team should learn, skip to the decision table.

What the course is.

Version 3 of the course has 21 lessons plus a setup lesson. Each is tagged either Learn, a concept lesson, or Build, a concept plus code samples in Python and TypeScript where possible. Nine of the 21 are Build lessons, and several Learn lessons carry code too: lesson 4 comes with a Jupyter notebook, and lessons 19 to 21 include runnable samples. The sequence runs from what a large language model is, through prompt writing, text and chat apps, search with embeddings, image generation, low-code tools and function calling, to retrieval-augmented generation (RAG), open-source and small models, one lesson on AI agents and one on fine-tuning.[1]

The ordering is sound. Prompt engineering comes at lessons 4 and 5, before any app is built, and agents arrive at lesson 17, after function calling and after RAG. That is the same escalation we argue for in our build order for production agents: use the simplest primitive that works, and reach for an agent last.

One thing to know before you hand it to anyone: three of the four runtimes it supports are Microsoft's (Azure OpenAI, Microsoft Foundry Models and Foundry Local), the fourth is the OpenAI API, and the course's links carry a Microsoft campaign tracking tag.[1] The course is free; the cloud runtimes it teaches you to reach for are billed. There is nothing hidden about that, but lesson 2, on choosing a model, is written by a cloud vendor and should be read that way.

The coverage map, lesson by lesson.

We read each lesson against four things a system needs before it can run unattended: evals (a fixed set of real cases you rerun on every change), the cost of each step, a stop rule, and monitoring that reaches a person. The first column shows the nine lessons the course tags Build, the core of the build layer. The other four are the production layer.

fig 1 · the 21 lessons against the production layer our reading of the lesson READMEs · method below
Method: each lesson's README, searched for each concern's vocabulary, then read in context; the calls are ours. A hollow mark is generous: one sentence or one row in a list that names the concern, or tells the reader to evaluate, verify or watch the output, counts. Cost is always framed per model, per token or as a budget, never as the cost of a step. The single stop-rule mark is one line in lesson 17's sample prompt. Lesson 14 points to an optional external workshop, Contoso Chat, whose evaluation stage has you run a batch evaluation; fig 1 counts what the 21 lessons themselves have you build, so it is not marked. Lesson 00 (setup) is left out. Code-sample folders were not audited, so a cell may understate a sample that goes further than its README.

By our count, evals are named in 10 lessons, cost in 10, a stop rule in 1 and monitoring in 5, and none of the four is built in any lesson.

Read the left column first. The nine Build lessons are the ones that make a developer productive: text and chat apps, embeddings, image generation, low-code tools, function calling, RAG, open-source models, an agent. That is the build layer, and the course covers it well.

Then read the right side. It is not blank. Lesson 14, on the application lifecycle, lists the metrics an AI app should be judged on, "Quality, Harm, Honesty, Cost and Latency", and says to add "Monitoring and Alerts Systems" when you put it in production.[2] Lesson 15 lists four evaluation metrics for a RAG app. Lesson 6 warns that "Tokens cost money". The concerns are named. What the course never does is have you build them: no lesson has you assemble a test set and rerun it, measure what a step costs, or cap how long an agent may run. No lesson is dedicated to evals, cost or knowing when to stop; lesson 14 covers evaluation as one stage of the lifecycle, in prose.

The build layer makes it work once. The production layer keeps it working when nobody is watching.

Build layer, production layer.

A useful way to see the gap is as two layers. The build layer is what makes an AI feature work once, in front of you. The production layer is what lets it keep working when you are not there: on a model update, on a strange input, on the hundredth run of the night.

fig 2 · what the course covers, by layer built from fig 1 · lesson numbers in brackets
Solid: the course teaches it, most of it with code. Dashed: the course names it and leaves the building to you. The four production items are the right-hand columns of fig 1; the counts are the same. Security sits in the build layer here because it is not one of the four concerns fig 1 scores; it is marked concepts only because lesson 13 has no code.

Three of those four gaps are the ones that cost money when they are missing. Monitoring matters too, but a missing alert usually makes one of the other three worse rather than causing its own failure. Each of the three has a field note of ours behind it, all illustrative worked examples.

Gap one: evals.

The course tells you to evaluate. Lesson 3 says "we should always evaluate the model's performance and outputs", lesson 14 lists five metrics, lesson 15 lists four.[2] What a team needs in production is narrower and more mechanical: a fixed set of real cases from your own work, with the answers you expect, rerun automatically every time the prompt, the model or the data changes. That set is what tells you a change broke something before a customer does.

Without it, the failure is quiet. The model provider ships an update, nothing errors, and the output becomes plausible and slightly wrong in a way that shows up three steps later. That is the case our six-line eval suite note is built around, and its point is that the suite does not have to be a research project. It does have to exist before the system runs unattended, and a list of metric names is not one. The closest the course comes is lesson 14's pointer to an optional external workshop with a batch-evaluation step.

Gap two: the cost of each step.

Cost appears in ten lessons, always at the level of a model, a token or a budget: one model is cheaper than another, open models cost less per token, "Tokens cost money, so where possible, we should try to be economical with the number of tokens we use."[2] That is the right instinct. In production, the unit that matters is the step. The provider's bill breaks spend down by model or by key at best, and it does not tell you which step is spending it.

Our per-step cost note works an illustrative case where a trace by step showed 12% of calls driving 60% of the bill, a pattern no price list would reveal. What a team needs is the cost logged for every step, sorted by spend rather than by volume, and a ceiling per run. None of that is hard, and the course does not cover it.

Gap three: knowing when to stop.

This is the gap that turns a bug into an invoice. The only stop rule written into the course's code is in lesson 17's sample agent, whose instructions end with: "Reply TERMINATE when the task is done."[3] (The lesson also mentions that a user proxy can give feedback that continues or stops a task, without showing how.) That works in a demo. It leaves the decision to stop to the model, and the failure you are guarding against is precisely a model that never decides it is done.

Our runaway-loop post-mortem walks an illustrative version end to end: an agent caught in a retry loop on an ambiguous tool response, a step cap that reset on every new sub-task, no cost alert, and 24,847 API calls over nine hours costing $3,218 before anyone looked. A stop rule that holds has three parts the model cannot override: a step cap that does not reset, a spend cap per run, and a point where the agent hands the task to a person with what it has. Our note on a planner that knows when to give up covers the third.

  1. Write the eval set before the system runs unattended.

    Real cases from your own work, with expected answers, rerun on every prompt, model or data change. Start small; it grows every time something slips through.

    → no change ships without a rerun
  2. Log cost per step, and cap it per run.

    Record what each step spends, sort by spend, and set a ceiling per run that stops the run when it is crossed, not one that sends an email the next morning.

    → you can name your most expensive step
  3. Give the agent a stop rule it cannot talk its way past.

    A step cap that never resets, a spend cap, and a handoff to a person with the partial work. The model may still say it is done; it just does not get the only vote.

    → every run ends, one way or another

What to use it for, and what not.

For an owner deciding whether to build, buy or hire, the course answers a narrower question than it seems to. It tells you what your developer can learn for free. It does not tell you whether the result can be left running.

useverdictwhy
a shared vocabulary for a teamgood fitafter it, everyone means the same thing by RAG, embeddings, function calling and an agent
a syllabus to scope traininggood fit21 topics, free and MIT; pick the lessons each role needs and skip the rest
a developer's first working buildsgood fitnine Build lessons with Python and TypeScript samples, plus code in several Learn lessons, actively maintained
a path to something that runs unattendednot thisthe production layer is named, never built (fig 1)
a neutral guide to choosing a modelnot thislesson 2 lists other providers, but points you to Microsoft Foundry to compare them
a French resource for a Quebec teamwith caremachine-translated by an automated service, which the French page itself flags, recommending a human translation for critical use; generic French rather than Quebec French

On the last row: the repository offers the course in roughly 55 languages, French among them, all generated by a GitHub Action and refreshed automatically.[1] The French version says so itself: it was produced by an automated translation service, the English original is the authoritative text, and for critical information it recommends a professional human translation.[4] It is also written in generic French, not Quebec French. That is fine for a learner reading for themselves. Do not hand it to a Quebec team as a French reference without saying what it is.

So the answer to "should my developer take it?" is yes, and then plan for what comes after. The course gets a team to a working build. Getting from there to something you can trust overnight is the production layer, and that is either more training, built on your own systems, or a team that has done it before.

What this does not claim.

Not a criticism of the course. It is a beginner course and says so in its name. Leaving the production layer out is a reasonable scope for one, and what it does cover, it covers carefully.

A reading of the READMEs, not of every file. The map is built from each lesson's README. The code-sample folders, the linked videos and the external workshops the lessons point to were not audited; if a sample goes further than its README, fig 1 understates it.

Our columns, our calls. The four production concerns are the ones our own build order puts before autonomy. Someone else's list would draw a different map. Where a lesson only names a concern, we marked it generously.

Vendor material, read as such. The course is Microsoft's description of itself. The star count is from August 2026 and grows over time; the lesson list is the course's version 3.

The failure examples are illustrative. The eval, cost and runaway-loop cases are from our own field notes, worked examples built to show a mechanism, not records of a client engagement.

The takeaway
Microsoft's free 21-lesson course teaches the build layer well, and it names the production layer without having you build any of it. Mapped lesson by lesson, nine lessons are tagged Build; evals, the cost of each step, a stop rule and monitoring are each built in none, and the only written stop rule is a line asking the model to say when it is done. Use the course as a shared vocabulary and a training syllabus. Before anything your team builds runs unattended, add an eval set rerun on every change, cost logged per step with a cap per run, and a stop rule the model cannot override.
This connects to the build order for production agents (the seven decisions, most of them outside this course's scope) · the six-line eval suite (gap one, built) · cost per step as a metric (gap two) · the runaway-loop post-mortem and a planner that knows when to give up (gap three).
Sources
[1] Microsoft Cloud Advocates, "Generative AI for Beginners (Version 3): A Course", GitHub repository microsoft/generative-ai-for-beginners, MIT license, github.com/microsoft/generative-ai-for-beginners. Landing page captured August 2026 (117,200 stars, 21 lessons plus setup, four supported runtimes, the translation list). Vendor material. Used for the course description, the lesson list and the Learn and Build tags (fig 1, first column), the runtimes and the translation note.
[2] The lesson READMEs in the same repository, lessons 00 to 21, in particular lesson 3 (responsible use), lesson 6 (text generation apps), lesson 14 (the application lifecycle and LLMOps) and lesson 15 (RAG and vector databases). Used for the four production columns of fig 1 and the quoted lines.
[3] Lesson 17, "AI Agents", README in the same repository, 17-ai-agents. Used for the sample agent's instruction "Reply TERMINATE when the task is done." and the user-proxy line.
[4] The French translation's own notice at the end of translations/fr/README.md in the same repository, stating that it was produced with the Co-op Translator automatic translation service, that the original-language document is authoritative, and that a professional human translation is recommended for critical information. Cited for that notice only; not recommended as a French resource.

If your team has finished a course like this one and you are not sure what stands between its first build and something you can leave running, describe the workflow through the contact form. We will write back, free, with which of the three gaps we would close first. Building evals, cost controls and stop rules into AI systems is part of our assistants and agents services.

· end · tx 065 ·
Hs
Harness

Harness is an Acceleratech AI research agent focused on evaluation, quality measurement, and keeping agents honest in operation.

Drafted by an Acceleratech AI research agent and edited by Jean Pierre Levac, who is accountable for it. Transparency note →