Measure before you build

Everyone I talk to who is building a brain for their agents can tell me what they plan to build. Very few can tell me how they'll know it worked. So this series starts with the test, not the brain.

By the end of this part you'll have four things: about 20 questions from your real work, what's true for each one today, a score for how your agents answer them right now, and a script that scores them again after every change you make.

This method works with MD files, a database, a knowledge graph, built-in memory or Team0. It scores the answers your agent produces, with no bonus for a particular storage format. The script reads saved answers; it does not connect to memory products for you.

Why measure first

Once you start building, every change feels like progress. Without a starting score, you can't tell a better brain from a busier one.

The most common trap: you connect another source, answers get longer and sound smarter, and the share of answers that are correct and current goes down. You only see that if you measured before.

Step 1: Collect real questions

Don't invent them. Go through the last two weeks and look for moments when:

• you pasted context into an agent because it didn't know;
• you corrected an agent;
• an agent was confident and wrong;
• you asked a coworker something an agent should have known.

Cover five kinds, a few of each:

1. A value that changed. You emailed a customer a pilot start date of November 1. On a call you moved it to November 15. Which one does your agent use?
2. One person, seen in different places. The person who emailed you on Monday is the person you met on Tuesday. Does your agent know that?
3. A decision this agent never heard. Made by email, in another tool, or by a coworker.
4. A promise. You told a customer something would happen. Did it, and does your agent know?
5. Something that looks like a fact but nobody said it. A newsletter, an automated email, a tentative calendar hold.

Step 2: Write what's true now

For each question, write what is true today, most important first, and the outdated values that must not be presented as current. Each question is one line in a .jsonl file. Spread out so you can read it:

{"id": "q01", "split": "practice", "asked_on": "2026-10-20",
 "question": "Draft a follow-up email to Jordan at Northpeak about the pilot start date.",
 "true_now": ["The pilot starts November 15, agreed with Jordan on the call on Oct 14",
              "Jordan asked for the contract by the end of the month"],
 "stale": ["The pilot starts November 1 (the emailed proposal from Oct 2)"]}

In the file, keep each question on a single line, or the script can't read it.

This is the hardest step, and the most useful one. You'll find questions where you aren't sure what's true yourself. Keep them. Those are exactly the ones your brain will have to handle. Resolve the expected truth with the person who knows before including the case in the numerical score; keep unresolved cases separately.

Step 3: Hide a third of them

Mark about a third of the questions "split": "held_out", and don't look at them while you build.

When you tune your prompts and rules on the questions you test with, your scores go up partly because you fitted them to those questions. The held-out questions tell you what real use will look like. If practice scores climb and held-out scores don't, you're fitting the test, not improving the brain.

Step 4: Ask your agents today

Ask every question to the agents you use today, as they are. Start a new session each time and add no context of your own. Save each answer and how many tool calls it took:

{"id": "q01", "tool_calls": 6, "answer": "Hi Jordan, following up on the pilot starting November 1..."}

This file is your "before". Keep it forever.

Step 5: Score it

Each answer is scored from 0 to 1 on five measures:

Run the script. It's available under the MIT license: score.py. The example questions, before answers, after answers and human labels show the file formats.

pip install -U 'anthropic>=2.3,<3' 'pydantic>=2,<3'
python3 score.py questions.jsonl before.jsonl

It needs an Anthropic API key in ANTHROPIC_API_KEY, and it calls a model once per answer, so each run costs money. It sends the question, expected truth, outdated values and answer to Anthropic. Use synthetic or redacted data, or obtain the necessary permission before sending work information. Check the selected model's current pricing.

Validate the files and coverage without a model call first:

python3 score.py questions.jsonl before.jsonl --dry-run

After each change, score the new answers next to the old ones:

python3 score.py questions.jsonl before.jsonl after.jsonl --out scores.json

You get one table for practice questions and one for held-out, with the change on each measure and in average tool calls. Tool calls matter: a brain that gets the right answer in one read instead of six searches is usually faster and cheaper, and that shows up here. Check your token bill to be sure.

Step 6: Check the judge

A model does the scoring, and a model can be wrong in consistent ways. Score about ten answers yourself, in the same five measures, and run:

python3 score.py questions.jsonl before.jsonl --labels my_labels.jsonl

The script shows the average gap between the judge and you on each measure. With two answers files, it checks the first. A gap above 0.2 is flagged. Fix the judge's instructions for that measure, or score that measure by hand, before trusting it.

Keep the answer-generating sessions separate from the expected-truth files and judge. Do not give an agent the answer key before testing it. Use the same model, task instructions, source snapshot and tool budget when comparing memory setups. Keep source text as data rather than instructions to the judge.

What a good result looks like

There is no magic number. What you want to see, on the held-out questions:

• Correct and current, and Missed, go up together. One without the other is a trap. A brain that says little is rarely wrong, and misses everything.
• Tool calls usually go down. The brain is doing the work, so the agent doesn't have to.
• Relevant doesn't fall as you add sources. If it does, you're letting junk in. That's its own part of this series.

A set of about 20 questions is a useful diagnostic, not a precise estimate of performance for every workload. Keep the rubric and judge configuration with each result; re-check your human agreement when either changes.

Keep it alive

Every time an agent surprises you, good or bad, add that question. New questions make good held-out questions, because you haven't tuned on them. And once you look at a held-out failure and fix for it, it isn't held out anymore: move it to practice and add a fresh one. Re-score after every change, and never delete the "before".

Next, part 1: what stays in instruction files, and a first awareness layer.

---

Build and measure your own setup with the companion skill. See the series. For a maintained company install, talk to Team0.