EvalHawk¶

Trustworthy LLM evaluation. Bring your own model, judge and data. EvalHawk makes the numbers trustworthy.

The problem¶

Teams grade LLM outputs with LLM judges, then trust the scores. But judges make mistakes, test sets are small, and related questions aren't independent. "v2 scored 85% vs v1's 82%" is often just noise.

What EvalHawk does¶

1. Calibrates your judge against humans. A fast, blind labeling UI measures how often the judge agrees with people.
2. Corrects for the judge's bias. Pass rates are bias-corrected (Rogan-Gladen, prediction-powered inference) and always come with confidence intervals.
3. Compares versions honestly. Paired tests say whether v2 is really better, worse, or "not enough data yet: you need about N more examples".
4. Works with anything. Any OpenAI-compatible model, cheap judges like JEV, your own HTTP API, or pre-recorded outputs.

A result looks like this:

no_fabrication   0.781 [0.742, 0.816]   (ppi, n=412, unknown=1.2%)

Where to go next¶