MILabs
Tools

OpenLLMetry for tracing AI applications in a small business

What OpenLLMetry does for AI applications, what businesses use it for, and how MILabs sets it up.

Updated 2026-10-07

What OpenLLMetry is

OpenLLMetry is an open-source set of extensions from Traceloop that adds tracing to applications built on large language models. It is built on OpenTelemetry, a widely used standard for collecting traces from software. Once an application is instrumented, each model call and each supporting step can be recorded as a span, and those records can be sent to an observability backend the business already uses. The library runs inside the application, so the team can see how an AI feature behaves alongside the rest of its software. It helps answer practical questions: which step was slow, which call failed, and where a response went wrong. Because it follows an open standard, it can be paired with different backends, and it can be self-hosted when a business wants its telemetry kept on infrastructure it controls. For a small business, the main value is a record to review. Owners and team leads can ask a plain question about an AI feature and get an answer from actual traces instead of memory.

Traceloop: official site

OpenLLMetry in plain language

Software that calls a language model does not run as a fixed sequence of steps. A call that usually returns quickly can stall during a busy period, and the final reply often hides what happened before it. Tracing records each step of a request so the path can be reconstructed afterwards.

A trace covers one request from start to finish. Each operation inside it is stored as a span, with a name, timings and attributes. A span for a model call might record the model used, its duration and its token counts, alongside spans for a document search or database lookup, so the sequence reads in one view.

OpenTelemetry is the open standard that defines how spans are created, how context passes between operations, and how data is exported. Because the format is shared, many monitoring products can read the records. OpenLLMetry supplies instrumentation that understands common model and framework calls, which saves developers from writing that code by hand.

Recording prompts and responses is a separate decision. These fields are often the most useful when something fails, and also the most sensitive, since they can contain customer names or health details. Setting capture per feature keeps that choice visible.

What businesses use it for

  • Finding which step in an AI workflow is slow or failing, such as a retrieval call that times out.
  • Reviewing how a customer-facing assistant answered a particular enquiry when a complaint comes in.
  • Checking that a document question-answering tool is pulling from the right sources before staff rely on it.
  • Keeping AI telemetry inside the business's own infrastructure when privacy rules or client contracts require it.
  • Giving a technical contractor or internal team a shared view of how an AI feature runs in production.
  • Watching for changes in behaviour after a model or prompt is updated.

Who it suits

  • Teams running a custom AI feature in production who need to see how it behaves.
  • Businesses with data-handling rules that require telemetry to stay on their own infrastructure.
  • Owners who already use OpenTelemetry-compatible monitoring and want AI calls in the same view.

How MILabs sets it up

MILabs starts with the owner or team lead, agreeing which AI features need tracing and what the business needs to be able to see when something goes wrong. We then review the observability tools and hosting the business already uses, so the traces land somewhere the team can open. Where a self-hosted backend is the right choice, we set it up inside the business's own environment. Access is set by role, so people who review traces can read them, and sensitive inputs and outputs are captured only where the owner has agreed. Before handover, we agree in writing who looks after the instrumentation, who reviews the traces, and who is told when a model or prompt changes. The aim is a setup the team can keep running without us.

Builds it fits

Industries it fits

OpenLLMetry in practice

In a business running an AI feature in production, tracing usually appears as a shared view rather than a new tool for staff to learn. A customer support assistant may answer enquiries all day, and nobody examines how it reached each reply. When a customer disputes an answer, the owner opens the trace for that conversation and sees the sequence: the question, the documents searched, the model call and the reply. The same view shows a contractor where time went, so a slow search can be fixed quickly.

Most of the work lies in deciding which steps get recorded. A document question-answering tool for a law firm needs its retrieval step visible, so the team can confirm the right passages were used. A booking assistant for a salon may need only timing and outcome fields, with the wording of messages left out. These choices are made with the people who will read the traces, so the view answers their questions.

Routine upkeep is quieter. When a model or prompt changes, the team compares traces from before and after to see whether behaviour moved. Access stays limited to the people who review them, and one named person owns the instrumentation so that library updates do not go unnoticed.

good questions, straight answers

Frequently asked questions

Can I self-host OpenLLMetry?
Yes. The library runs inside your own application, and it is open source, so you can review the code before deciding to use it. Self-hosting usually refers to the backend that stores and displays traces. If the business wants that backend on infrastructure it controls, MILabs can set it up as part of the build. Which backends are available depends on what the business chooses to run, settled in the setup conversation.
Which builds work well with OpenLLMetry?
Builds that call a language model several times, or that pull from documents or a knowledge base, benefit most. A customer support assistant, a knowledge base chatbot and a document question-answering tool all have steps that can fail or slow down in ways that are hard to see from outside. A simple single-call feature gains less, so the decision depends on how many steps the build has.
What is OpenLLMetry?
OpenLLMetry is an open-source tracing project from Traceloop. It adds instrumentation to applications built on large language models and builds on OpenTelemetry, so traces can be sent to the monitoring tools a business already uses. It records the model calls and surrounding steps of an AI feature, which makes it easier to review how that feature behaved on a given request.
How does MILabs set up OpenLLMetry?
MILabs begins with the owner or team lead to agree which AI features to trace and what the team needs to see. We connect traces to the observability tools the business already has, set access by role, and agree who looks after the instrumentation once it runs. If a self-hosted backend is needed, it is set up inside the business's environment. The result is a setup the team can run on its own.
Where does OpenLLMetry store data?
The library itself stores nothing. It sends telemetry from your application to an OpenTelemetry-compatible backend that you choose, and that backend keeps and displays the traces. That may be a hosted service or a system the business runs itself. Which data is captured is set in the instrumentation, so the owner decides what is recorded before anything leaves the application.
let's make something happen

Workflow automation: how we can help

Choose how you want to work on this with MILabs. Both options start with an enquiry.

Discovery is free, no obligation from the first chat.

Let us build it for you

We build it and look after it.

Build it yourself, we'll help

You build it with our direction, teaching and a weekly check-in.