start ugly, write evals anyway.

TLDR; understanding the importance of evals, and how they can make yours and your agent life easier without spending exponential tokens and time to complete a task.

pre -

Agent = Model + Harness.

• Model - the one thats brings the intelligence, capability to think and act.
• Harness - the one that lets the model act by giving tools, state, memory, envs, etc..

imo there are - model evals, and harness evals. and this article talks more about the harness eval, which helps us understand - does this harness support agent to to do what it should be doing.

im assuming you know what is a harness, if not, read this -

the anatomy of an agent harness — viv ( @Vtrivedy10 )

–

what is eval, compared to tests for sdlc #

back when we were building our search agent, who can search across of your entire data estate and point you to right things or possibly get you an answer which you are looking for – we struggled a lot with getting the accurate responses, as the models were not that great to do better tool calling and also writing better markdown and html (this is early 2025).

because this was very new for teams who are now building agentic systems (moving from standard SDLC to ADLC), but assessing/evaluating these systems didnt have a playbook.

why, you may ask! the non determinism enters the request lifecycle:

• the context keeps changing in the background, so does the thinking, context window, exploration (tool calls), and understanding of data results different tokens.

there is a beautiful writing on why is evals not equals tests here –

evals ≠ tests — ankur goyal ( @ankrgyl )

TLDR of above post;

• Tests = does it work?
• Evals = what can it do?
• Mature Evals = what should it be doing?

not all harness are the same, can be managed or open source - which can then be of different structure based on the needs.

before metrics, get the nouns straight. this saved us a lot of arguing in review:

• task - one test case. input + starting state + what counts as success.
• trial - one run of that task. you run many, because the same task passes today and fails tomorrow.
• transcript (trace / trajectory) - the full record of a trial. reasoning, tool calls, intermediate results, tokens.
• outcome - the end state, not the final message. the agent saying “i created the connector” is transcript. the connector actually existing is outcome.
• grader - the logic that scores it. code, llm-as-judge, or a human. one task can have many.
• eval harness - the infra that runs all of this. different from the agent harness. yes, confusing. two harnesses.
• suite - a bunch of tasks that share a goal.

take a pov of junior engineer and is given an engineering task - yes, you being the senior engineer you give him access to laptop, saas tools, notebook, internet, etc.. and ask him to finish the task.

the junior engg has the capability of using these tools + internet can gather information and complete the task in 4 days not 1 day. the harness can help j engg do more work but “more work doesnt mean its getting better.”

in a similar way, harness helps agent do more things but doesnt mean its getting better in any sense.

when you give a new tool/notebook, to help structure things - did it worsen the quality of work, failures, time? how would you track? - evals.

–

how to approach evals #

ive seen most teams struggle with the cold start problem with evals, mainly for - what are good metrics to evaluate/assess an agent, and how can i build them.

the cold start is smaller than people think. you do not need 500 tasks. 20-50 tasks pulled from real failures/hand written is a fine start. early on, every change has a big, obvious effect, so a small sample is enough to see it. the tasks get harder to write the longer you wait - early on your product requirements are your test cases, later you are reverse engineering success criteria out of a live system.

where to get those 20-50: whatever you already check by hand before a release, plus your bug tracker, plus the support queue. real complaints make better tasks than synthetic ones.

two things that quietly decide whether your eval is worth anything:

write tasks two people would grade the same way. if two domain experts read the task and disagree on pass/fail, that ambiguity shows up as noise in your metric forever. also write a reference solution for each task - a known-good output that passes all graders. it proves the task is solvable and that your graders arent broken.

build the negative half. one sided evals create one sided agents. if you only test “did it search when it should”, you ship an agent that searches for everything. we felt this hard with the search agent - undertriggering and overtriggering are the same bug with opposite signs, and you only catch it if the suite has both kinds of task.

if im starting on evals today, i will only build these 2 metrics to gain confidence in the agent:

1. task completion
2. correctness

was agent able to complete the task? yes! is that the right answer? yes!

and you are good for internal testing, not yet ready for the production.

–

grade what it produced, not the path it took #

the instinct is to assert the exact sequence of tool calls. dont. agents keep finding valid approaches you didnt think of, and every brittle path assertion punishes that. path assertions also couple your eval to internal implementation details of the harness, so you cant refactor without breaking your own tests.

outcome grading is your gate. transcript reading is your debugger.

one more: build in partial credit. an agent that identified the problem and verified the user but failed the last step is meaningfully better than one that died at turn 2. a single binary hides that entire continuum, and the continuum is where your improvement is.

–

then watch the trajectories #

you will now see the traces flowing from internal testing, and you start monitoring the trajectories - which will show you the real insights into thinking, tool calls and can help you understand the behaviour of the agent for variety of tasks.

the signal you might want to capture and evaluate are:

• distraction - when agent is off track, does it align itself on the task or goes haywire.
• tokens spent - does it even need to spend millions of tokens to complete the task or can be done in lesser as well.
• latency - the time agent takes to complete a task. mainly, when its customer facing, there is not a lot of attention and time customer spends waiting for the agent to say i cant help!
• context bloat - how much unnecessary information is flowing into the context window.
• consistency - same task, same conditions, five runs. same answer? this one surprised me the most.

that last one deserves its own section.

–

your eval is also a thing that can be broken #

this is the part i underrated for the longest time.

a low score is not automatically the agent’s fault. it can be the grader, the task spec, or the harness.

a good rule: 0% pass across many trials usually means a broken task, not an incapable agent. and failures should feel fair - you should be able to look at a failed transcript and say yeah, that was on the agent.

so, read the transcripts. not a sample after a big release, weekly. it is the single highest leverage habit in this whole thing and the one everyone skips because it doesnt produce a number.

same applies to llm-as-judge. it needs its own calibration set against human grading, its own version history, and a disagreement analysis. give the judge an explicit way out (“return unknown if you dont have enough information”) so it stops hallucinating a verdict. and grade one dimension per judge instead of one judge grading six dimensions - much cleaner signal.

worth budgeting for too - judging in production tends to run somewhere around 30-50% of your inference cost.

–

evals need to evolve #

one interesting learning i had over the time building and maintaining both the agent and evals is, todays criteria of metrics doesnt makes sense tomorrow for a better model/harness. evals need to evolve.

we started with search agent, but as both models and harness are becoming more capable, it turned out to be a generic agent which can perform actions, setup data connectors, generate presentation ready artifacts, and more - the evals which were evaluating the search tasks doesnt do a good job anymore. so we evolved.

all new and successful patterns needs to feed back into evals. can give you regression and can-be-improved insights.

then hill climb.

once you crack all these for you agent, you are ready to hill climb with these evals, and @Vtrivedy10 did a great work explaning how you can do it –

better harness: a recipe for harness hill-climbing with evals — viv ( @Vtrivedy10 )

as you go deeper you will realise you need an eval to evaluate your evals. thanks for reading!