A reflection on non-deterministic evals

The era of evaluating LLMs has largely been defined by coding tests, math tests, and other deterministic outputs. While these are the obvious first steps for understanding whether an LLM is being “correct,” when working on problems that are not deterministic, evals are a whole other problem.

The fundamental question goes from “Is it correct?” to “What is correct?”

After building out a set of evals for Phoebe, then rebuilding them, and then building them once more, I feel like I have a couple of things to share that would be beneficial.

Do not define success in a vacuum. Start with your customers.

Fundamentally, defining success for a nondeterministic, subjective output is impossible.

The age of neural networks is built on the back of one concept: that traditional machine learning systems are hindered by the fact that humans create the rules they operate on. The world is full of subtle associations that an engineer cannot understand, so how do we then understand whether the LLMs are right?

For that, we must turn to our customers. There are inevitably power users, users who love the product, and users who hate the product. Certain features of the product are used more, and certain parts are used less. Study that. Study how customers interact with your LLM, what requires reprompting or multiple tries, and what gets generated and then thrown out. Also look at the ways that are successful. Who is using the product over and over, and how are they using it?

For us, that meant looking at how each of our customers interacted with our agents. We looked at success rates and error rates, specific traces that failed, and really dug into why they failed.

Using actual product signals, we built out a set of “correct” outcomes. Note that a lot of these “correct” outcomes look arbitrary and often go against what we, as engineers, thought.

Simulations are just simulations.

It’s tempting at first (at least it was for me) to recreate a small part of your product offline to run tests against. As we’ve already determined, though, AI is nondeterministic, and the same things that make it great—well, make it hard to evaluate.

Any tool call, data input, or really anything that is ever given in context to an agent or surfaced for the agent to interact with needs to have as much product data parity as possible. Without that product data parity, even one small variable that is seemingly unrelated can make a big difference when testing, and the worst part is that you will never know why.

Account for user error

Sometimes users are wrong. You should know that. It’s unrealistic to assume there is no user error in any product, especially in a product that uses agents that have natural-language directions. Users can contradict themselves. You shouldn’t treat these user errors as failure cases, but instead as eval cases to see how your agent deals with these contradictions.

There should be eval cases, yes, but also deep thinking about what the behavior should actually be in response to these contradictions—or, certainly, you could just look at the traces (as mentioned above).

If you aren’t confident in your eval cases when you make them, how will you be confident in them when you use them?

Coding agents are great. Not having to write code is great. But not understanding what an eval actually tests—not great.

After you have distilled “correctness” from your customers, it will be tempting to tell a coding agent to go ham, build up the cases from 5 to 50, and make no mistakes. But if you don’t fundamentally read and check every iteration, or at least validate that these cases have some sort of validity, there’s no world in which you would ever trust them enough to let your eval score define your feature choices.

So how do you write good evals for subjective agents?

Here’s a step-by-step process:

1. Create visibility into your current agents and how they are being used.
2. Sort traces, either deterministically or with an LLM.
3. Create as close to a one-to-one environment with prod as you can.
4. Recreate those traces within this environment as close to one-to-one as you can.
5. Define scoring based on the correct or incorrect outcomes of traces, as well as traditional scoring methods.
6. Repeat.

Note that the nice part of this method is that, as your agent improves over time, the types of issues that arise in production will inherently become new eval cases that 1) you would have never thought of before and 2) are ceiling-raising, such that you will never end up with an agent that passes all evals but still fails in production.

How do you score good evals?

Scoring any output has been an issue (look at SWE-bench—that is supposed to be deterministic!), and subjective outputs are even harder. That being said, while a raw number might not tell you just how good your agent is in production, using the same scoring system, and checking it against production outcomes, gives you relative scores between different agent configs and can at least give you insight into the difference between two stacks.

For us, we had four main criteria we wanted to check:

• Correctness (including NOT surfacing false positives, NOT surfacing clear negatives, and correctly surfacing positives)
• Evidence (It’s hard to evaluate confidence, but using evidence as a vessel for confidence helps us be confident that the outcomes are “correct”)
• Process (This is mostly behavioral, including the correct ordering of tool calls and not getting stuck in reasoning loops)
• Safety (These are mostly hard cases that lead to catastrophic issues, for example, deleting integral data)

Once we defined our criteria, we created a rubric of binary questions. Using binary questions allowed us to easily identify which part of each case an agent might be failing and also allowed us to easily add or delete criteria in our rubric on a run-by-run basis while still keeping some level of determinism within our LLM judge.

Finally, we used a team of three LLM judges from different providers. For certain criteria that were more subjective, we averaged the answers; for others, we used majority rule. Again, the point of using three LLM judges here is to correctly ride the line between determinism and flexibility in scoring.

A method for turning production failures into eval cases

Now, you should start with handwritten, or at least human-directed, cases, but once you have a good base, you can close the loop on evals.

Some notable parts of this generation

A big part of generating an eval case is porting the failure case into one of the seeded orgs, or worlds, that we have. Note that because creating new worlds is relatively expensive, we will not have a direct representation of every world in the organization set, so we need to first pick the most representative org and then make small adjustments to make it more representative of the original trace.

When we surface a failure mode and create an eval case, it is important that the same failure mode arises in the eval case itself. It would be irrelevant to make a case that doesn’t actually show the failure case.

If the failure mode doesn’t surface in the first iteration, we enter the re-edit loop, where the repair agent receives:

• Sentinel’s required failure signal
• Cartographer’s reconstruction plan and invariants
• The latest Writer output
• Checker blocking issues and required fixes
• All prior replay summaries
• The replay-loop decision explaining why another repair is needed

It then reasons through and edits the eval case to attempt to surface the correct failure case.

A case study in using eval cases to make decisions

After creating a good set of evals, we finally had the chance to use them.

We evaluated Phoebe V1, Phoebe Lily, and Phoebe V2.

To see whether our changes were actually helping, we ran the same 28 production-derived cases three times against each stack. Phoebe V2 scored 62.1% on average compared with 51.8% for Phoebe V1. It also completed the suite in almost half the time.

This is useful evidence, but it is not enough on its own. Averages can make a system look stable even when its behavior changes significantly from run to run.

Looking at repeatability made that variance much clearer. Phoebe V2 passed 37.8% of scored cases on an average run, compared with 27.5% for Phoebe V1. It also passed more cases at least once across the three runs.

The more important result, though, was consistency. Phoebe V2’s pass^3 was 14.3%, compared with 3.6% for Phoebe V1. Better, but still not good enough. In production, the average doesn’t really matter to the person using the product. They get one run. If the agent does the right thing once and fails the next two times, it is not reliable. This is why I like pass^3: it tells us whether the behavior keeps showing up, not just whether the agent can get there once.

The aggregate score also hid where the improvement came from. Safety increased by 8.9 percentage points and recall increased by 5.7 points, while process was effectively flat. Coverage actually declined by 1.5 points.

I think this is an important way to read eval results: “better” is rarely uniform. A stack can improve meaningfully overall while still introducing a regression in a dimension that matters. The score should start an investigation, not finish one.

Even those dimensions are still abstractions. When we broke the averages down by case, the changes became much more concrete. Some production behaviors improved dramatically, including recovery from inactive-caregiver rejection and handling minimum-suggestion requirements.

At the same time, several cases regressed. Those failures were not noise to average away; they showed us exactly where the new stack had traded one behavior for another.

This is ultimately how I think evals are most useful. The headline number helps compare stacks, the dimensions show the shape of the change, and the individual cases explain what actually happened.

None of these numbers tells us that an agent is simply “good” or “correct.” They give us a repeatable way to ask whether a specific change improved the production behaviors we care about—and where it did not. That confidence depends on the cases themselves. Because these evals were reconstructed from real production traces, we can be more confident that changes in the scores reflect behaviors users will actually encounter, rather than artifacts of synthetic prompts or simplified simulations. The closer our eval cases remain to real-world scenarios, the more meaning we can assign to the comparison.