DeepEval LLM evaluation for testing AI assistants
What DeepEval is, what businesses use it for, and how MILabs sets it up to test AI assistants.
Updated 2026-07-06
What DeepEval is
DeepEval is an open-source Python framework for testing applications built on large language models. It is published by Confident AI and runs inside a normal Python project, so developers can keep evaluation code alongside the rest of their software. A test case passes an input and a model's output through one or more metrics, such as whether an answer stays faithful to supplied source text, whether it addresses the question asked, or whether it follows a set of instructions. Results show which cases passed and which did not, so a change to a prompt, a model or a document set can be checked before it reaches customers. DeepEval can point at different model providers, which lets a team compare options on the same test set. For a business, the practical value is a repeatable check on an AI assistant's behaviour over time. Assistants that answer from company documents, policies or price lists are where such checks matter most, because a small wording change can alter what a customer is told.
DeepEval in plain language
An AI assistant can give a convincing answer in a demonstration and a wrong one on a busy afternoon. Language models produce text that varies between runs, and a change to a prompt, a document or a model version can shift many replies at once. Evaluation is the work of deciding in advance what a good answer looks like, then checking outputs against that standard each time something changes.
DeepEval treats this as a form of software testing. Each test case holds an input, the output an assistant produced, and where relevant a source passage or reference answer. One or more metrics then score the case, each measuring a single property such as whether a reply sticks to its source or follows a format the business specified. Because the cases sit in a code project, the same checks can run again after every edit. Each failure points to a specific exchange, so a person can read it and decide whether the assistant, the test or the expectation needs attention. Metric scores are most useful when a team reads them alongside the failing examples themselves.
What businesses use it for
- Checking that a customer support assistant answers from current policy documents
- Testing a quote or booking assistant against a fixed set of typical enquiries
- Comparing two model providers on the same questions before switching
- Watching for drift in answers after a prompt or document update
- Confirming an assistant declines out-of-scope requests such as legal or medical advice
- Keeping a written evaluation set that staff can read and extend
Who it suits
- Teams with a developer who can maintain Python code
- Owners whose assistant answers from policies, price lists or contracts
- Businesses changing models or documents often enough to need repeatable checks
How MILabs sets it up
The easiest first step is to say hello through the enquiry form, and we'll take it from there. The conversation then turns to the owner, or the team lead who will answer for the assistant, and which questions and documents matter most. From there we draft the evaluation set, with example questions, the answers the business expects, and the cases where the assistant should decline or hand over to a person. DeepEval is installed in a project the business controls, and we connect it to the document store, model accounts and any helpdesk or CRM already in use. Permissions limit changes to named staff, and the owner records who reviews failed runs and who updates expected answers when policy changes.
Builds it fits
Industries it fits
DeepEval in practice
In everyday work, the evaluation set usually starts as a document that the owner, a team lead and a developer write together. It lists the questions customers ask most often, the answers that would be acceptable, and the requests the assistant must decline or pass to a person. A developer turns that list into test cases and runs them against the assistant before any change goes live.
When a price list, a policy or a product specification changes, the document owner updates the expected answers, and the test run shows which replies now disagree with the new wording. When a model provider releases a new version, the same cases run against both versions, so the team can see where the newer model improves and where it introduces a regression.
Results usually land in a shared place where the responsible person reads each failure. Some failures are real defects in the assistant, which lead to a prompt or retrieval change. Others reveal a test that was written too narrowly, and those get corrected rather than left to fail. Over time the file becomes a readable account of what the assistant has been asked to do and how it has been checked.
Frequently asked questions
- How does MILabs set up DeepEval?
- MILabs starts by agreeing with the owner or team lead which assistant is being tested, which questions matter most, and who answers for the results. From those conversations a starter set of test cases is drafted, and DeepEval is installed in a project the business controls. The team then links the document store and model accounts, so every run reads current content. Permissions and review ownership are recorded before the first run.
- Where does DeepEval store data?
- DeepEval runs as a Python library inside the project where it is installed, so test cases and results stay in files and environments the business controls. The optional Confident AI platform can receive results if the team connects it, which makes that link a choice the owner signs off on. MILabs confirms the storage location with the owner during setup, so everyone knows exactly where results live from day one.
- When is DeepEval the wrong choice?
- DeepEval fits least when a business only needs a chat interface and has no one to maintain code, or when the main question is cost tracking or production monitoring. It is a lighter fit for a single, short-lived assistant that will not change after launch, because its value comes from re-running the same checks over time. In those cases MILabs often suggests a simpler review routine, with a staff member reading sample answers regularly.
- Can my team use DeepEval without training?
- Yes, staff who write test cases in plain language can work with the evaluation set without learning the library. Running and changing the tests does require someone comfortable with Python, though. MILabs includes a short handover during setup, so the owner and one or two staff members can read a failed run, see which metric flagged it, and add an example question.
- Could MILabs build on top of DeepEval?
- Yes, MILabs can build an evaluation layer around DeepEval for an assistant the business already runs or one being built now. That layer might include a scheduled test run, a shared folder of expected answers, and a summary the owner reads regularly. Whether those pieces are worth building depends on how often the assistant changes and how much of its output customers see directly.
