Part 9
Running it
Operate your prototype with health checks, cost tracking and comparable evaluations.
Yogev Ben-Tov · 3 min read
A useful prototype needs ongoing care. Sources stop syncing, model behavior changes and yesterday's correct answer becomes outdated.
By the end of this part, you will have a short operating checklist, a dated evaluation record and an estimate of what your system costs to keep useful.
Measure learning and usage separately
Track what it costs to bring information in and maintain it, and what it costs for agents to use it. Include model calls, storage, source processing and quality checks alongside read latency and token use.
Define a useful read before calculating cost per answer. A low bill can mean an efficient system or a broken importer. Compare quality and coverage with cost.
Choose health checks that catch your failures
Start with the problems you have seen: source freshness, stalled work, unusual changes in what is retained, agents skipping reads, failed maintenance or a missing backup.
Decide who sees a failure and what they do next. For each check, try a controlled failure on a disposable setup and verify recovery. A backup needs a tested restore, including the privacy consequences from part 8.
Record changes to prompts, code, models and integrations with their dates. If a rule changes, decide whether older retained information needs review. Respect user corrections and the retention limits of the material you still hold.
Keep the evaluation comparable
Version the questions and expected truth. An old answer scored against newly changed truth is a different experiment; preserve the earlier snapshot and collect a fresh baseline where needed.
Record the agent model, memory configuration, source coverage, judge model and rubric. Use the same conditions for comparisons. A model name can point to an updated version, so keep the provider's returned model identifier when available and re-check the judge against human labels.
python3 score.py questions.jsonl before.jsonl part9.jsonl --out part9-scores.json
Small test sets help you discover failures. They do not establish universal accuracy. Rotate held-out questions once you inspect and tune against them, and add new ones from real surprises.
Decide what to keep operating
Compare the result with part 0: current correctness, relevance, omissions, ordering and context completeness, plus latency, tool calls and cost. An MD-based memory can be a good result for a small workload. A growing team may need additional controls and maintenance. Let the measured needs drive the next step.
You can keep building with the companion skill. If you want a maintained company product, Team0 runs in your company's cloud: the product, updates and support are the offer, while the company controls its infrastructure and provider accounts. Book a company-install conversation.
Build and measure your own setup with the companion skill. See the series. For a maintained company install, talk to Team0.