Insights
Notes from the work: AI training, evaluation, and building agents that do real knowledge work.
Agents that do real work · Part 3 of 3
How to measure AI agents and decide when they can act alone

How to measure AI agents in production: what we track today, what those numbers cannot show, four measures to add, and an autonomy ladder where each job earns the right to act alone on a record of verified outcomes.
Agents that do real work · Part 2 of 3
AI agents in production: one Slack, two agents, three pictures

AI agents in production, shown in three pictures: the anatomy of one run, the paths an instruction takes between me, Claude and ChatGPT in Slack, and the six check layers and which failure each caught. A person caught three of the six.
Agents that do real work · Part 1 of 3
How to design AI agents: five decisions that outlast a particular model

To design AI agents that do real work, settle five things before choosing a model: one owner per job, a written instruction contract, one source of orders, a receipt on every run and a verified outcome. What works and what does not.
Teaching machines to do real work · Part 4 of 4
The correct-looking file: why AI agents fail and what experts offer

The worst failures of AI agents on knowledge work produce a file that looks right and passes the grader. A taxonomy of those failures across spreadsheets, decks and documents, what experts see that graders miss, and the behaviors pre-training cannot teach.
Teaching machines to do real work · Part 3 of 4
Ten ways a grader lies in AI post-training: rubrics, answer keys and judges

If the exercises decide what a model sees, the graders decide what it becomes. Ten ways rubrics, answer keys and model-based judges give the wrong signal in AI post-training, each cheap to fix once seen and invisible until someone reads the rubric next to the brief.
Teaching machines to do real work · Part 2 of 4
How to design training exercises for AI agents

A spreadsheet of a few thousand training exercises turned out to be a few hundred scenarios sliced ten ways. Why row counts lie, why down-weighting cannot fix root imbalance, and how transplants add coverage without new task designs.
Teaching machines to do real work · Part 1 of 4
Reward hacking explained: how one bad grader teaches AI a thousand bad lessons

After pre-training, a model learns from a loop with three parts: (a) exercises define the situations it sees, (b) graders define what counts as better and (c) human experts are the quality check on whether the graders' idea of better matches a professional output.