Promptfoo Prompt Testing for Small Business Assistants
Promptfoo is an open-source tool for testing AI prompts and models. Learn how MILabs sets it up for small businesses.
Updated 2026-02-24
What Promptfoo is
Promptfoo is an open-source tool for testing how language model applications behave. A team writes a set of test cases, defines what a good answer looks like and runs those cases against one or more prompts and models. The results show which versions pass, which fail and where the differences lie, so changes can be compared on evidence rather than impression. Promptfoo runs from the command line, and its configuration lives in files that can be stored alongside the rest of a project. It can also be self-hosted, which means test data and results can stay on infrastructure the business controls. For a small business the tool is most useful once an AI assistant already answers customers or staff, because each change to a prompt or model can be checked before it reaches people. Promptfoo does not decide what the assistant should say. That judgement belongs to the owner or team lead, who writes the expectations the tests check against.
Promptfoo in plain language
A language model rarely answers the same way twice, and a small edit to a prompt can shift the tone, accuracy or length of a reply without anyone noticing until a customer does. Testing is the habit of writing down what a good answer should contain and checking each output against that list whenever something changes. Promptfoo gives that habit a clear structure. Each test case pairs an input, such as a question a customer might type, with the conditions a reply must meet, such as mentioning a refund window or declining to quote a price. Those conditions can be plain text checks, or judgements made by another model that reads the answer.
Because the cases are stored as files, the same set can be run again after each edit, which turns a memory of what used to work into a repeatable record. Teams usually begin with a short list of real questions and extend it as new failures appear. The value depends on the cases chosen. A set built only from easy questions will report that everything passes, and that result says little about the replies that matter most.
What businesses use it for
- Checking that a customer support assistant answers correctly before a prompt change goes live.
- Comparing two models on the same enquiries to see which handles the business's own wording better.
- Confirming that a document question tool cites its source material and admits when it cannot answer.
- Keeping a saved set of test cases so each update to a staff assistant can be rechecked.
- Catching a drop in quality when a model provider changes a version the business relies on.
Who it suits
- Teams that already run an AI assistant and want to test changes before customers see them.
- Owners with a developer or technical contact who can maintain the test files.
- Businesses that want assistants checked against their own examples rather than generic benchmarks.
How MILabs sets it up
MILabs starts with the owner or team lead, who agrees the testing plan, picks the first assistant to check and sets out which answers count as acceptable. We then connect Promptfoo to the models and tools the business uses, through access the business approves, and write the first test cases from real examples the team provides. Permissions are agreed early, covering who can edit test files, who can run them and where results are stored. Because the tool can be self-hosted, the business decides whether results stay on its own infrastructure. Before handover, the owner and MILabs agree who looks after the test set when prompts or models change. We are based in Sydney, work remotely and run sessions by video call.
Builds it fits
Industries it fits
Promptfoo in practice
In a business, testing usually settles into a small set of recurring checks rather than a single launch event. The owner, or a staff member who handles enquiries, collects questions that customers and colleagues really ask, along with replies they consider right. Those pairs become the first test cases. When the assistant changes, whether through new instructions or a different model, the cases run again and the outputs are compared side by side. A reviewer reads the failures and decides whether the test or the assistant is at fault.
That decision matters, because a test that encodes an outdated policy will punish a correct answer. Over time the set grows as unexpected replies surface in daily use, and each new failure is added so it cannot quietly return. A technical contact keeps the files tidy and runs the suite, while the people who know the business keep the expectations honest. Sensitive replies, such as those about money, health or legal obligations, still go to a person, because the test set only checks what was written into it.
Frequently asked questions
- When is Promptfoo the wrong choice?
- Promptfoo is less suited to a business with no AI output to test yet, or one whose real need is a finished customer product rather than a way to check one. It also fits poorly where nobody will look after a set of test cases. In those cases a manual review or a simple checklist may serve better, and MILabs will say so early.
- Can my team use Promptfoo without training?
- Yes, for simple use. A team member who writes clear test cases can usually work with Promptfoo after a short walkthrough. Running the tool well still calls for comfort with the command line and configuration files. MILabs covers this during setup by video call, using the team's own prompts as examples. Staff who only read results need less support than those who edit the test files.
- Can MILabs build on top of Promptfoo?
- Yes. MILabs can build test sets, evaluation runs and reporting around Promptfoo for an assistant the business already has or plans to build. The owner scopes the work, and the business keeps control of its test data and results. Progress is shared as files and reviewed on video calls. We are based in Sydney, work remotely and run sessions by video call.
- What does a Promptfoo setup need from me?
- You name the assistant or workflow to test, provide real examples of questions and good answers, and say who will review results. You also need access to the models and systems you already use, granted through accounts you control. Agree who maintains the test set when prompts or models change. MILabs handles the configuration and explains each step during video call sessions.
- What does Promptfoo not do?
- Promptfoo does not decide what a good answer is for your business, and it does not replace human review of sensitive replies. It cannot guarantee that an assistant behaves the same after every update. It checks outputs against the tests you define, so results are only as useful as those tests. A test that misses a failure will also miss it in the results.
