How an expert‑graded task is made

An evaluation set is only as good as its hardest task. A model that scores well on easy questions tells you very little about whether it can do the job a professional is paid to do. So the unit we care about is the single task: one realistic piece of work, one clear standard for doing it well. This note describes how we make one.

1. An expert writes it from real work

Every task starts with a practising professional, not a template. They write a request of the kind they handle in a normal week, with the context a colleague would have: the documents, the numbers, the constraint that makes it awkward. The best tasks contain the detail a hurried reader would miss, because that is where real work goes wrong.

2. A second expert reviews it

A different expert in the same field then works the task independently. Is the request clear? Is there one defensible answer, or a defensible range? Would a senior person in their firm accept it as realistic? If the reviewer disagrees with the author on any of these, the task goes back. It is not delivered until both agree.

3. The rubric is written

Next comes the rubric: the criteria a good answer must meet, each with a weight that reflects how much it matters. Getting a figure right may be worth more than formatting; missing a regulatory point may matter most of all. We also write penalty criteria for shortcuts, the moves an expert would never make even when they produce a plausible answer. A model that takes a reported number at face value when the notes say otherwise should lose points, even if its conclusion happens to be right.

4. It is tested on real models

Before delivery we run the task against current models and grade the outputs with the rubric. This does two jobs. It shows whether the task actually separates good work from weak work. And it catches rubric gaps: if a model earns credit without doing the work, the criteria were too loose, and we tighten them and test again.

5. It is delivered whole

A client receives more than a prompt and an answer key. Each task arrives with its rubric and weights, the reviewer's notes, and the model results we observed, so your team can see why a criterion exists and re-run the same task on every release.

Why the process is the product

None of these stages is novel on its own. What matters is that none is skipped. A task written by one expert carries that expert's blind spots. A rubric never tested on a model is a guess. A score with no reasoning behind it cannot be trusted or improved. The process exists so that when a task says a model failed, you can see exactly where, and believe it.