> For the complete documentation index, see [llms.txt](https://doc.aissist.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doc.aissist.io/capabilities/evaluation.md).

# Evaluation

Test your Aissist agent against realistic simulated conversations before launch and after every change, and score how it did.

*Last updated: November 6, 2026*

Evaluations test your AI agent against realistic conversations and score how it did. Use them before you go live, and again after every change, so you know a change helped before your customers find out.

### Why it matters

* **Confidence before launch.** See how the agent handles a refund, a late delivery or a difficult customer before it meets real ones.
* **Safe changes.** Re-run the same test after editing an instruction, asset or action, and see whether the score really moved.
* **Real scenarios, not guesses.** Sets can be written from conversations that actually happened.

### How it works

An **evaluation set** describes one test:

* **Scenario** - the situation a simulated customer arrives with and pursues, for example "the customer asks for a refund on a damaged item but does not remember their order number".
* **Finish** - what counts as finished. By default: the conversation ended, or was escalated to a person. Change it when the scenario ends on something specific, such as "a plan was recommended, or the customer was passed to sales".
* **Measurements** - what is scored, each with guidance and a weight. Weights should add up to 100%. An optional **Measurement Instruction** sets how strictly to score.

A **run** generates simulated conversations from the scenario against your current workspace, then scores each one against every measurement. When you start a run you choose:

* **Number of Sessions** - 1 to 30
* **Strategy** - how varied the simulated customers behave: **Default**, **Creative** or **Crazy** (adversarial)

Each run shows a score, a comment, the result per measurement with its reason, and the simulated conversations with their own scores.

Evaluation sets are in **Insight → Evaluation**.

### Score against your own documents

If your project has [Insight Assets](https://doc.aissist.io/capabilities/insight-assets.md), such as QA guidelines or internal policies, every run uses them. For each simulated conversation, the grader looks up what the measurements need to know and checks the agent's answers against the passages it finds, so a confident but wrong answer is marked wrong. Insight Assets are never given to the agent being tested, only to the grader.

### Compare against an earlier run

A score on its own moves a few points even when nothing changed. To see whether a change helped, use **Compare against** on a completed run. The new run replays exactly what the simulated customers said in that run, so only the agent's side differs. The **vs. baseline** column shows the difference and whether it is a real change or within noise.

### Build sets from real conversations

* **Describe it and let Ask Engine write it.** In **Insight → Evaluation**, click **Add Evaluation**, describe the scenario, what to measure and when to finish, and click **Confirm**. Ask Engine searches your past conversations for ones like it, checks your existing sets so it does not repeat one, and drafts the set with the conversations it found. Nothing is created until you click **Create evaluation set**. Prefer a blank form? Choose **Write it myself instead**.
* **Evaluate around one conversation.** On a session in **Deploy → Sessions**, click **Evaluate around this session**. Aissist drafts a set from that conversation, replays the original against today's agent, and simulates variations of it (**Close**, **Varied** or **Adversarial**). This is the quickest way to check a fix for a conversation that went wrong.
* **Score your people too.** Sets that score human agents on their real conversations drive [Agent Insight](https://doc.aissist.io/capabilities/agent-insight.md).

### When to evaluate

* **Before launch:** one set per main sub-agent, plus an adversarial set for anything sensitive.
* **After changes:** re-run with **Compare against** after editing instructions, assets, actions or sub-agents.
* **After a bad conversation:** evaluate around it, fix, and run again.

### With Ask Engine

[Ask Engine](https://doc.aissist.io/capabilities/ask.md) reads your evaluation sets, runs and results:

> Which evaluation runs failed this week, and what went wrong in them?

> Create an evaluation set for customers asking to cancel their subscription, measuring whether we offer only the options we really have.

For the second, Ask Engine drafts the set from real conversations and tells you it is not created yet; you create it in **Insight → Evaluation** after reviewing it.

### Best practice

* Write the scenario the way a customer would arrive with it. Real-sounding scenarios find real conversations to run against.
* Keep each set to one situation and a few clear measurements.
* Run enough sessions to see a pattern, and use **Compare against** before calling a change a win.
* Re-run your key sets every time you change something important.

### Related

* [Insight Assets](https://doc.aissist.io/capabilities/insight-assets.md)
* [Try](https://doc.aissist.io/tutorial/try.md)
* [Continuous Improvement](https://doc.aissist.io/capabilities/continuous-improvement.md)
* [Reliability](https://doc.aissist.io/capabilities/reliability.md)
