Chatbots and Q&A¶

What it covers¶

Any system that takes one message and returns one answer: support bots, internal assistants, FAQ bots, "ask the model" features. For bots that retrieve documents first, also read RAG systems. For whole conversations, see Conversations.

What can go wrong¶

Each failure you care about becomes one criterion.

The evals¶

Tips for custom questions:

• One failure per question. "Is it accurate and polite?" is two criteria.
• Phrase it so PASS means good: "Is the answer free of made-up facts?"
• Avoid rewarding length. "Is it helpful?" tends to favour long answers.

What you provide¶

A JSONL file with one test input per line:

{"input": "What's your refund policy?", "reference": "Refunds within 30 days of purchase."}
{"input": "Do you ship to Canada?", "reference": "Yes, 5–7 business days.", "group_id": "shipping"}

• reference is optional (only correctness needs it).
• group_id is optional: use it for related questions, so the error ranges stay honest.

Your chatbot is called through a target: a Python function, an HTTP endpoint, or an OpenAI-compatible model. If you already have the answers, add an "output" field and use the recorded target.

How it's tested¶

1. EvalHawk checks the file (every row has an input; reference exists if correctness is used).
2. It assigns each input to train, dev or test, and locks the assignment.
3. It sends each input to your chatbot and stores the answer.
4. Each answer is checked against each criterion. Code checks run instantly; AI-judged criteria are sent to the judge, and results are cached.
5. You label about 100 answers per AI-judged criterion (evalhawk label).
6. The report shows corrected pass rates with error ranges (The trust layer).

Example config¶

[target]
type    = "http"
url     = "https://bot.internal/api/chat"
body    = '{"message": "{{input}}"}'
answer_path = "$.reply"

[[criteria]]
id       = "answer_relevance"
template = "answer_relevance"

[[criteria]]
id       = "correctness"
template = "correctness"

[[criteria]]
id       = "empathetic"
question = "If the customer sounds upset, does the answer acknowledge it?"

[[criteria]]
id    = "concise"
check = { type = "max_words", value = 150 }

[evaluations.support]
dataset  = "data/support.jsonl"
kind     = "single_turn"
criteria = { answer_relevance = "local", correctness = "local", empathetic = "local", concise = "code" }

What you get¶

Illustrative output:

evaluation: support    run: v2    n = 500    judge: local

criterion          pass rate (corrected)    judge says   judge status      unknown
answer_relevance   91%  [88% – 94%]         93%          ✅ ok (n=100)      0.4%
correctness        78%  [73% – 83%]         84%          ✅ ok (n=120)      1.2%
empathetic         64%  [55% – 72%]         70%          ⚠ need ~40 more    2.0%
concise            97%  [95% – 98%]         code check   —                  —

Compared with DeepEval¶