LangWatch

What is LangWatch?

LangWatch is an open-source LLMOps platform designed for large language model applications and AI agents, operated by Reasoning Engine B.V. It integrates capabilities such as monitoring, evaluation, agent simulation, prompt management, AI Gateway, and governance, thereby assisting teams throughout the entire process from development and testing to operational deployment.

The platform utilizes OpenTelemetry to receive and organize tracking data, and it supports hosted cloud environments, fully self-hosted setups, enterprise-specific clouds, as well as hybrid deployment models. Developers can identify issues by examining model calls, tool usage, and user sessions, and they can then convert real-world scenarios into evaluations and testing cases.

Overview of key capabilities

LLM and agent observability

LangWatch automatically records calls to large language models, tool usage, searches, errors, and user interactions, and presents the complete execution pathway in its tracking reports. Teams can view input and output data, tokens, costs, delays, information related to the models, users, as well as custom metadata.

• Filter tracking by project, user, model, prompt, or version.
• View the execution order of nested Spans and proxy multi-step processes.
• Positioning tool failures, slow calls, and abnormal costs.
• Set up dashboards for production quality and business metrics.
• Feedback is linked to specific tracking through user events.
• Export the analyzed data to an external system.
• The quality of alerts and automated responses declines.

OpenTelemetry and integration

The tracking layer is built on OpenTelemetry; therefore, it allows the use of the LangWatch SDK as well as the reception of data generated by other compatible libraries. It offers direct integration with common proxy frameworks, model providers, and automation platforms.

Offline experiments

Experiments is used to run model, prompt, or proxy versions on fixed datasets before going live, while recording the output and score for each sample. Teams can conduct experiments through the interface, Python, or TypeScript, and then compare the quality, cost, and latency of various configurations.

1. Create datasets that represent real tasks.
2. Select the current production version as the baseline.
3. Define the prompt, model, or agent configuration that needs to be compared.
4. Configure built-in or custom evaluators for output.
5. Run all samples in parallel and record the tracking.
6. Compare scores, errors, delays, and costs.
7. Include the failed samples in the regression dataset.
8. Integrate the minimum quality requirements into the continuous integration gateway.

Online evaluation

Online Evaluation runs the designated evaluators automatically once production tracking data is available, in order to continuously monitor aspects such as authenticity, relevance, security, PII, or custom business metrics. It is useful for detecting any changes in quality that may arise as a result of changes in users, models, or data.

Guardrails real-time protective barriers

The evaluator can be executed synchronously within the application code as part of Guardrail, and it decides whether to allow or block execution based on whether certain conditions are met. It is suitable for detecting attempts at unauthorized access at the input side, performing security checks at the output side, and verifying critical business rules; however, it increases call latency and the amount of events generated.

• Detect jailbreaking and prompt injection.
• Identify PII and sensitive information.
• Check for toxic, harmful, or illegal content.
• Verify the fidelity and relevance of RAG responses.
• Execute LLM-as-a-judge or custom code scoring.
• Return a security message in case of failure, or redirect to manual handling.

Evaluators and workflows

LangWatch offers built-in evaluators such as RAGAS, Hallucination, Toxicity, PII, and Model Judging; it also allows teams to create reusable organization-level evaluators. Complex evaluations can be carried out by combining multiple steps and judgments within workflows, with custom scores being assigned to tracks or Spans.

Datasets and annotations

Datasets can be managed through SDKs and APIs, or they can be extracted from production tracking or generated with the assistance of AI. Annotation Inbox is used to assign samples to human reviewers, allowing business experts to add labels, target answers, and subjective quality assessments.

• Create a set of failure cases based on real-world tracking.
• Save evaluation inputs such as text and images.
• Create and query datasets in bulk through code.
• Use AI to quickly generate initial test samples.
• Record manual labels and automatic evaluations separately.
• Use the audit results for subsequent regression testing.

Agent Simulations

Agent Simulations verifies whether an agent can achieve its goals, adhere to rules, and use tools correctly in real conversations by simulating multiple rounds of interaction between the user and the agent under test. It is more suitable for assessing aspects such as state management, memory, tool utilization, and behavior in long conversations compared to evaluation based on a single round of input and output.

Scene structure

Supported proxy types

• Text chat agent.
• Remote agent that requires login or a token.
• Proxy functions in local development.
• Real-time proxies such as OpenAI Realtime.
• Voice agents such as ElevenLabs, Twilio, and Pipecat.
• Real-time multimodal agents such as Gemini Live.

Red team and security scenarios

Scenario testing can be used to create adversarial users, make unauthorized requests, and carry out prompt injection attacks, in order to observe the agent’s defensive capabilities across multiple rounds of interaction. The results of red-team testing should serve as input for improvements; they cannot be interpreted as indicating permanent security after just one test.

Scenario open-source library

The standalone Scenario repository provides proxy testing tools for Python and TypeScript, and is licensed under Apache Version 2. It can run locally as well as in continuous integration environments; it is not required that all tests be carried out in the LangWatch cloud.

Prompt management

Prompt Management allows prompt templates, variables, versions, and release status to be stored in one place, preventing these prompts from being scattered across code, tables, and interfaces. Developers can synchronize prompts using the SDK or CLI, and they can replace models and parameters in the Playground to conduct comparisons.

AI Gateway

AI Gateway enables unified access to multiple model providers, and it allows for the management of virtual keys, routing, budgets, caching, and Guardrails within an organization’s boundaries. When using a self-hosted setup, it is possible to have model traffic terminate within one’s own network boundaries.

• Isolate applications and teams using virtual keys.
• Route between multiple models or suppliers.
• Set hierarchical budgets for organizations, projects, or users.
• Use hint caching to reduce redundant reasoning.
• Implement real-time guardrails at the unified entry point.
• Centralize the logging of costs, failures, and model usage.

MCP, CLI, and Skills

LangWatch offers MCP, command-line tools, and skills designed for programming assistants, enabling AI assistants to help with access tracking, creating evaluations, running simulations, and managing prompts. Automated modifications still require human review, especially in the case of production keys, safeguards, and deployment settings.

Cloud pricing

The price information was verified on August 23, 2026; the actual amounts, taxes, exchange rates, and discounts may vary, and the final figures will be those displayed on the settlement page.

LangWatch Cloud is billed based on core seats and the amount of events processed; Growth charges a storage fee for data that is retained for more than 30 days. Lite users can view and collaborate on content, but the extent of their editing rights is determined by their current permissions.

Events and storage billing

Each model invocation, tool usage, retrieval, evaluation, or simulation step is counted as an event; therefore, a single user conversation typically involves multiple billing events. The first 200,000 events per month are included in the subscription fee, while an additional 5 euros is charged for every 100,000 subsequent events.

Bill estimation method

1. Count the number of interactions with end users on a monthly basis.
2. Measure how many models, tools, and retrieval events are involved in a typical interaction.
3. Include events generated by online evaluations and Guardrail.
4. Estimate the additional runtime for offline experiments and proxy simulations.
5. Determine the number of core seats that require full access rights.
6. Calculate the GB of data that still needs to be accessed after 30 days.
7. There are additional costs for models, cloud storage, and self-hosted infrastructure.
8. Set aside a budget for growth, backfilling, and abnormal traffic.

Open source and licenses

LangWatch adopts an open-core architecture; the core platform code is licensed under Apache License Version 2, while the Python and TypeScript SDKs as well as the MCP services are licensed under the MIT license. Enterprise-related modules such as SCIM, audit logs, licensing, and billing management are located in separate directories, and a commercial license is required for their use in production environments.

Self-hosted version

Without a corporate license, self-hosted LangWatch still allows an unlimited number of members, teams, projects, and custom content, and it runs the same core software as the cloud version. A corporate license enables SSO, RBAC, SCIM, audit logging, and support within the same deployment, without the need to use a separate version of the product.

Comparison of deployment modes

Self-hosted infrastructure

The current v3 production approach relies primarily on Kubernetes Helm; ClickHouse is used as the core storage for analysis, while object storage is employed to store cold data in a hierarchical manner. The Docker Compose-based solution is still mainly intended for v2, and old tutorials should not be followed when preparing to deploy v3.

• Use ClickHouse to save and query tracking and evaluation events.
• Event tracing ensures the tracking and evaluation of the processing order.
• Layer the old data into compatible object storage.
• Use ClickHouse’s native backup and recovery mechanisms.
• Adjust deployments of different scales using Helm Overlay.
• The AI Gateway sub-chart can be enabled optionally.
• It is necessary to manage the keys, network, backups, upgrades, and monitoring on one’s own.

Safety and compliance

LangWatch states explicitly that it complies with GDPR and holds ISO 27001 certification; enterprise clients can request relevant reports as well as data processing agreements. Since the platform may come into contact with prompts, outputs, tool parameters, and user data, the team still needs to carry out its own data classification and access control procedures.

• Remove unnecessary personal data and keys before embedding tracking points.
• Restrict who can view the original tracking and datasets.
• Set minimum permissions for export, MCP, and API.
• Review the cloud region, sub-processor, and retention period.
• Transmission and static encryption are enabled in self-hosted environments.
• Establish processes for auditing, deletion, backup, and incident response.
• For regulated activities, compliance responsibilities must be established through corporate contracts.

Which users are it suitable for

• Engineering teams that develop large language model applications and AI agents.
• The team responsible for model quality, regression, and security testing.
• Product teams that need to simulate user testing and multiple rounds of agent behavior.
• Organizations that wish to involve business experts in providing suggestions and performing manual annotation.
• Platform teams that require unified model gateways, budgets, and governance.
• Companies that wish to have open-source, self-hosted solutions and control over tracking data.
• Developers who use low-code tools such as n8n to create AI workflows.

Product advantages

• Observation, evaluation, simulation, and prompt management form a complete closed loop.
• Agent Simulations is suitable for verifying multi-round and tool-based agents.
• It supports offline experiments, online monitoring, and synchronized Guardrails.
• OpenTelemetry reduces reliance on a single SDK.
• The cloud-based pricing offers both a free option and low-cost upgrades for additional features.
• The main advantage of self-hosting is that there are no limits on the number of members or projects.
• The boundaries of the open-core license are clearly defined.

Restrictions and Precautions

• A single user interaction can generate multiple billing events.
• Simulation, online evaluation, and Guardrail increase the costs associated with models and events.
• LLM scorers and simulated users may make mistakes in their judgments.
• A complete quality system requires the maintenance of datasets, standards, and manual review.
• V3 self-hosting is based primarily on Kubernetes, and its operational complexity is higher than that of single-container tools.
• Although the enterprise module is located in the same warehouse, it is not subject to the Apache open-source license.
• The tracked content may contain sensitive business data and model prompts.
• The Developer version offers a limited number of scenarios, simulations, and custom evaluations.

Implementation process

1. Choose LangWatch Cloud or plan for a self-hosted environment.
2. Create a project and integrate OpenTelemetry or the official SDK.
3. Use test requests to verify Span, costs, and user metadata.
4. Establish core quality indicators and failure classifications.
5. The first test dataset is created from real-world tracking data.
6. Offline experiments are set up to indicate model changes.
7. Use scenario simulation to test the behavior of multiple rounds of agents and tools.
8. Add Guardrails for high-risk inputs and outputs.
9. Configure online production evaluation, alerts, and budgets.
10. Review the actual costs by event, seat, storage, and model fees.

Frequently Asked Questions

Is LangWatch free?

The Developer cloud package is available permanently at no cost; it includes 50,000 events per month, 14 days of data access, 2 users, as well as a limited number of scenarios, simulations, and custom evaluations. The self-hosted open-source version imposes no limits on the number of members, teams, or projects, but the user is responsible for covering the infrastructure costs.

What is a billing event?

Model invocation, tool invocation, retrieval, evaluation, and simulation steps can all give rise to events. A complete user session typically contains multiple events; therefore, estimation should be based on actual tracking data rather than simply counting the number of sessions.

Is LangWatch open source?

The main platform is licensed under the Apache Version 2 license, while the SDK and MCP are licensed under the MIT license. The enterprise modules come with a separate commercial license; therefore, it is more accurate to describe it as an open-core platform.

Can it be self-hosted?

Yes, for version 3 it is recommended to use Kubernetes along with the official Helm Charts. The core functionalities of the platform can be utilized without a license; however, features such as SSO, SCIM, RBAC, auditing, and enterprise support require a commercial license.

What is the difference between Agent Simulation and regular evaluation?

Traditional evaluations typically involve checking fixed inputs and outputs, while Agent Simulation allows simulated users to interact with agents over multiple rounds. The latter is more suitable for testing states, tools, voice functionality, handling of rejections, and the entire process of achieving business objectives.

Can it block harmful content in real time?

The evaluator can be used as Guardrail to be executed synchronously before and after model invocation, and based on the results, actions such as blocking or reverting to a fallback mode can be taken. Before going live, it is necessary to test latency, false positives, degradation logic, and exception handling.

Summary

LangWatch not only keeps track of LLM performance but also integrates offline experiments, online evaluations, real-time safeguards, agent scenario simulations, and prompt management into a single AI engineering workflow. It is particularly suitable for teams that need to conduct systematic verification of multiple rounds of agent interactions, rather than merely observing the responses of a single model.

The cloud-based Growth version starts at 29 euros per core, with additional charges applied for extra events and extended storage periods; the open-source self-hosted version offers greater control over the data, but deploying it in production version v3 requires expertise in Kubernetes management. Before adopting it, it is necessary to calculate the number of events, the costs associated with the models, and the storage requirements based on actual traffic levels.