Testing LLMs: You’re Probably Doing It Wrong

When most folks start testing LLMs, they reach for the familiar: regex, hard-coded assertions, or golden outputs.

That’s a great way to get started — but it’s not ideal.

Traditional software testing assumes determinism — that the same input will produce the same output every time. But LLMs aren’t deterministic. They’re probabilistic. Ask the same question 100 times and you might get 100 subtly different answers. So the question becomes: what does “passing” even mean?

Three Ways to Think About Testing LLMs

Run it 100 times. Then ask: what percentage of those outputs meet your bar for success? In probabilistic systems like LLMs, there’s no single “correct” answer — just ranges of acceptable outcomes. That threshold is entirely use-case dependent. For some applications, 95% consistency is fine. For others, anything less than 99.9% breaks trust. Testing here isn’t pass/fail — it’s statistical sampling, grounded in the real-world tolerance of your downstream system or user.
Use an LLM to evaluate another LLM. This feels weird at first — using one model to grade another — but it works surprisingly well. The catch? Judgment varies. If your evaluation model isn’t tuned to the task, you’ll get false positives or negatives. You might need a human in the loop. Or at least a good prompt engineer to guide it.
Let the model help you write better tests. One of the most overlooked moves: ask the LLM to evaluate its own response and suggest improvements. If your prompt is ambiguous, the outputs will be too. Sometimes, just the act of writing a clear test prompt will make you realize: “Damn, I could’ve done this task myself faster.”

What Testing Does Tell You

Testing LLMs doesn’t tell you if something works once. It tells you how it performs at scale. Over thousands of runs. Across edge cases. It also reveals how brittle your prompt is, how vague your intent was, or how tight your expected structure needs to be.

Testing structured inputs (e.g. forms, IDs, known entities) is straightforward.

But for unstructured tasks like chatbots, creative writing, or open-ended QA? You’re not validating truth — you’re validating acceptable variation.

Bottom line?
Don’t test GenAI the same way you test APIs. Test for behavior, not just correctness. And when in doubt, have a model or human double-check the output — because “good enough” is contextual.