HoneyHive

A one-sentence summary

HoneyHive is an observability and evaluation platform designed for production-grade AI agents; it helps engineering teams monitor models, tools, workflows, and the operation of multiple agents, identifies issues through automated and manual evaluations, and turns actual failures into part of continuous regression tests.

Tool Introduction

HoneyHive integrates monitoring, evaluation, datasets, prompts, and continuous integration into the same development lifecycle. It is not a model provider nor does it create business agents for users; instead, it keeps track of what those agents actually do, assesses the quality of their performance, and helps teams identify the causes of any issues that arise.

The platform relies on OpenTelemetry as its core tracking mechanism, and it offers support for Python, TypeScript, open APIs, the command line, as well as various automatic instrumentation methods. Developers can use the free cloud version, while enterprises can opt for multi-tenant, single-tenant, hybrid, or self-hosted deployment options.

Core data model

Main functions

Distributed link tracing

HoneyHive can record model inferences, external tools, searches, sequential steps, and entire user sessions, while also maintaining the parent-child relationships between them. Engineers can replay a run in chronological order and jump directly to the sections where errors, timeouts, or abnormal behavior occur.

Long-term proxy debugging

The platform displays information on retries, loops, handovers to sub-agents, and tool calls regarding an agent’s trajectory over several hours or days. It helps to determine at which stage the agent deviated from the intended path, rather than merely checking whether the final answer was correct.

Multi-agent observability

Multiple agents or sub-agents can use a unified event model to record decisions, tools, status, and handovers. The team needs to provide consistent tracking identifiers and context; otherwise, operations across different processes may end up as disconnected fragments.

Automatic stake insertion

The Python and TypeScript toolkits are based on OpenTelemetry and can automatically capture data from common model frameworks, proxy frameworks, and external calls. The platform claims to support more than 50 popular libraries, including LangChain, LangGraph, AWS Strands, Google ADK, and OpenAI Agents SDK.

Custom span

For functions that are not recognized automatically, developers can use decorators or manually record input, output, errors, duration, metadata, feedback, and metrics. The finer the level of tracking, the better the debugging capabilities, but it also increases the amount of data processed, leads to higher latency, and raises risks related to sensitive information.

Production quality monitoring

Teams can run evaluations on real-time traffic and associate the resulting scores with user feedback, business outcomes, and specific versions. Production monitoring should include settings for sampling and cost limits, in order to avoid triggering expensive model evaluations every time a complex agent is run.

Custom dashboard

Tracking data, metadata, and evaluation metrics can be grouped by model, version, user group, environment, or function to generate charts. Boolean metrics allow one to view the pass rate, while numerical metrics enable the calculation of averages, quartiles, and extreme values; strings are suitable for sorting and filtering.

Alarm and drift detection

The platform can establish monitoring mechanisms based on errors, delays, costs, quality, and changes in distribution, thereby helping teams to detect agent degradation in a timely manner. The alert thresholds need to be adjusted according to traffic levels and business risks; being too sensitive will result in false alerts, while being too lenient will cause real failures to go unnoticed.

Client evaluation

Applications can calculate metrics related to format, security, personal information, or business rules within their own execution environment, and then write the results to tracking or monitoring systems. Client-side evaluation is suitable for performing sensitive checks before the data leaves the system, and metrics with the same name are not affected by evaluations carried out on the server side.

Evaluation of server-side code

Developers can configure code evaluation in HoneyHive to calculate deterministic metrics for the uploaded logs, time spans, or entire sessions. Code evaluation is suitable for checking format, fields, tool order, and verifiable business rules, and its results are more stable than those obtained through model-based assessment.

LLM judging and evaluation

The LLM Evaluator assigns scores to unstructured responses, search results, or agent trajectories based on custom criteria, and can provide explanations for those scores. The current documentation states that the server-side model used for evaluation is GPT-4o; sensitive data and costs need to be assessed taking into account sampling methods, contracts, and deployment approaches.

Manual evaluation and annotation queue

Domain experts can assign scores, provide comments, and create annotation queues using a custom scale within the interface; this approach is suitable for tasks such as those in healthcare, law, finance, or brand tone management, where it is difficult to make entirely automatic judgments. The human standards must be aligned first, otherwise different annotators will produce results that cannot be compared with one another.

Evaluator verification

The platform allows one to test the evaluator using recent events or manual samples before deciding whether to deploy it in production. Model judgments are not inherently reliable; they should be aligned with expert opinions in order to check for biases, prompt injection, location bias, and scoring stability.

Online evaluation sampling

Server-side evaluation can be carried out on only a certain percentage of production or pre-release events, in order to control costs and throughput. Offline experiments generally do not use the same sampling logic, so as to ensure that different configurations can be compared on the same complete dataset.

Dataset management

The dataset consists of inputs, standard answers, conversation history, and metadata; it can be selected from production tracking or uploaded via an interface or SDK. Teams can continuously add real faults and edge cases to the benchmark set, thereby creating testing assets that are more aligned with business needs.

Importing and exporting datasets

The interface supports JSON, JSONL, and CSV files, and the SDK enables the bulk creation and retrieval of data points. Datasets can be exported for external evaluation, archiving, or fine-tuning, but user identities and confidential information should be removed before exporting.

Offline experiments

The experiment combines the function to be tested, a unified dataset, and a set of evaluators, making it suitable for comparing prompts, models, retrieval strategies, chunking methods, or entire proxy architectures. Metrics and tracking are maintained for each run, facilitating a sample-by-sample comparison of improvements and deteriorations.

Parallel experiments

The evaluation framework can process data points in parallel and control the number of working threads, thereby reducing the time required to run large test sets. However, the level of parallelism is still constrained by limitations imposed by the model provider, budget constraints, shared resources, and the capacity of the target system.

Git context

When running experiments from a code repository, the SDK can keep track of commits, branches, authors, remote addresses, and whether there are any uncommitted changes in the workspace. This information helps to associate the results with specific code versions, but it may also reveal metadata related to the internal repository; therefore, it should be decided whether to retain such information based on the environment and permissions in place.

Barriers to continuous integration

The team can run evaluations whenever the code is modified, set quality thresholds, and prevent releases when the metrics deteriorate. The test set should include cases of critical business failures, rather than only easy-to-handle examples; otherwise, a positive test result does not guarantee reliability in production.

Online experiments and A/B testing

Applications can include experimental identifiers, configuration versions, and user feedback in the tracking system, allowing for a comparison between the control group and the experimental group on the dashboard. HoneyHive is responsible for recording and analyzing this data, while traffic allocation is usually handled by the existing feature switches or the experimental system.

Prompt Playground

Playground is used for quickly testing prompts, models, functions, and external tools; it allows one to open failed events from the tracking process in order to proceed with further iterations. It is suitable for exploring and reproducing issues, but production prompts should not be modified directly, bypassing formal datasets, evaluations, and code reviews.

Prompt version management

Each prompt configuration includes a template, a model, hyperparameters, and tools; it is possible to save historical versions of such configurations and deploy them in applications. Versioning helps to reduce errors resulting from manual copying, but model updates by the providers, changes in external knowledge sources, and modifications to runtime tools can still affect the outcomes.

Production failure transition dataset

Engineers can filter out errors, operations with low scores, or those of high value from the log repository and organize them into datasets. They can then use these same cases to compare the suggestions, models, and code before and after repairs, thus creating a closed loop that goes from observation to improvement.

Comparison of evaluation methods

The workflow from tracking to improvement

1. Create separate projects and API keys for production, preview, and development environments.
2. Integrate models, tools, retrieval, and proxy functions using OpenTelemetry or the official SDK.
3. First, check the span structures, parent-child relationships, fields, and sensitive data in the testing environment.
4. Establish baseline metrics for latency, errors, costs, tool accuracy, and output quality.
5. Filter out failed cases, those with low scores, complaints, and high-value edge cases from the actual traffic.
6. Organize the cases into versioned datasets that include inputs, standard answers, and metadata.
7. Use code and models, together with domain experts, to create complementary evaluators and carry out calibration.
8. Run prompts, model, or architecture experiments to compare improvements and regressions on a sample-by-sample basis.
9. Include key evaluations in continuous integration, and implement sampling and alerts in production settings.
10. Continuously add new failures to the dataset, so that the reliability standards evolve in line with actual business conditions.

Connection steps

1. Create a HoneyHive workspace and project, and generate a separate API key for the project.
2. Install Python or TypeScript toolkits based on the technology stack, or use OpenTelemetry to send traces.
3. Initialize the tracker and set the project, environment, session name, and optional private service address.
4. Run a minimal request to verify that the model, tools, connections, and session layers are correct.
5. Add spans to custom functions to log necessary errors, execution times, feedback, and business metadata.
6. Hide passwords, tokens, personal information, document content, and other data that should not be uploaded.
7. Run it in the log warehouse to verify search, filtering, charts, and team permissions.
8. Create small datasets and deterministic evaluators, then gradually introduce model judges.
9. Set event budgets, request rates, retention policies, sampling, and production alerts.
10. Complete security, DPA, BAA, or private deployment assessments before officially increasing traffic.

Prices and packages

The price information was verified on August 23, 2026; the actual amounts, taxes, exchange rates, and discounts may vary, and the final figures will be those displayed on the settlement page.

HoneyHive currently offers a free Developer version as well as an Enterprise customized solution. The free version does not require a credit card and is suitable for development and small-scale production testing; the Enterprise version provides additional features related to usage limits, user authentication, support services, data retention, compliance requirements, and deployment flexibility.

How events are counted

An event refers to a single tracking span or a set of metric labels; the monthly total is equal to the sum of the number of spans and the number of metrics. A single complex proxy execution may involve numerous models, tools, retry attempts, and metrics, so 10,000 events does not equate to 10,000 user requests.

What is included in the free version?

Developer includes automatic and manual evaluation, comprehensive monitoring capabilities, prompt versioning and deployment, continuous integration, 30 days of data retention, and community support. Social single sign-on and basic role control are available, but there is no enterprise SAML, custom roles, or dedicated support.

What is added to Enterprise?

Enterprise offers the ability to customize usage levels, unlimited users and workspaces, SAML or custom single sign-on solutions, customized retention policies, SLAs and service quotas, DPA and BAA agreements, dedicated technical account managers, as well as quarterly business reviews. For specific prices and contract terms, please contact the sales team.

Comparison of deployment methods

Self-hosting is available only for Enterprise plans, but it does not mean that users can obtain the complete platform code for free. Enterprises are responsible for handling deployment packages, upgrades, telemetry, offline dependencies, backups, disaster recovery, and support issues.

SDK, CLI, and open interfaces

Python SDK

The Python toolkit supports OpenTelemetry, synchronous and asynchronous decorators, manual span creation, context propagation, experiments, and various model spylers. The fact that a code repository is made public does not automatically confer open-source status; in the absence of a clear license, it should be treated as copyrighted material.

TypeScript and MCP

The currently active TypeScript API Client is used for interacting with data interfaces; the old TypeScript SDK repository included MCP capabilities, but it has been archived. New projects should select the active package and fix a version in accordance with the current documentation, in order to avoid relying on generation clients that are no longer maintained.

Open interfaces and other languages

Apart from Python and TypeScript, other languages can directly send OpenTelemetry data, call the open interfaces, or generate typed clients in accordance with the OpenAPI specifications. Clients created manually still need to handle authentication, rate limiting, retries, pagination, and version compatibility.

HoneyHive Daemon

Daemon is used to collect telemetry from coding agents such as Claude Code and export it to HoneyHive; it is useful for monitoring tool calls and agent executions. The coding traces may include repository paths, code, prompts, and commands, and exclusion and masking settings should be configured before enabling it.

Open-source status

The HoneyHive platform is not a fully open-source project; the cloud console, enterprise services, and self-hosted deployments are still subject to commercial plans and contracts. Some SDKs, CLI tools, OpenAPI specifications, and skill components are made available under the MIT license, which simply means that these specific repositories can be used in accordance with that license.

Realign was once released as an independent AI testing and simulation framework, but it has now been set to read-only mode. By separating public tools from closed-source platforms, it is possible to prevent people from assuming that a free clone of the repository will provide access to the full HoneyHive service.

Which users are it suitable for

• Engineering teams that are developing multi-step AI agents, RAG, or tool-call applications.
• Platform teams that require debugging, retries, loops, handover of sub-agents, and long-running operations.
• AI product teams that aim to continuously convert real production failures into regression datasets.
• Organizations that require a combination of code, model evaluation, manual annotation, and user feedback for assessment.
• Development teams that wish to set AI-related quality thresholds for releases within the context of continuous integration.
• Enterprises that require SAML, DPA, BAA, SLA, custom retention, and private deployment.
• Infrastructure teams that are already using OpenTelemetry and wish to standardize AI telemetry.
• Technical leaders responsible for monitoring coding agents and the enterprise’s internal AI platforms are needed.

It is not very suitable for which users

• Ordinary users who only want to use generic chatbots and do not intend to develop AI applications.
• Only an operations team for monitoring traditional server CPUs, memory, and networks is required.
• Organizations that require all platform code to be under an open-source license.
• Sensitive business teams for which it is not possible to mask tracking data or implement access control measures.
• Teams that experience more than 10,000 events per month in terms of agent operations but are unwilling to invest in an enterprise-level solution.
• It is hoped that model judges will automatically take the place of all domain experts and feedback from real users.
• Organizations that cannot accept the fact that cloud data is located in the western United States and that they do not opt for a private deployment option.

Typical use cases

• Replay a failed attempt by a customer service agent to identify the specific steps in the process that led to the omission of the policy lookup.
• Compare the fidelity and tool accuracy of the two prompt versions on the same complaint dataset.
• Monitor the trends in retrieval relevance, answer accuracy, latency, and cost of RAG applications.
• Have domain experts review medical or financial responses in the annotation queue.
• Create regression samples from low-score tracking and integrate them into continuous integration.
• Parallel offline experiments were conducted on different models, chunking strategies, and reorderers.
• Conduct online A/B tests based on agent version and user group, and correlate them with actual results.
• The data plane is deployed in the enterprise’s private environment, while a managed control plane is used.
• Collect information on encoded proxy tool calls and errors to assess the reliability of development automation.

Product advantages

• Connect production tracking, offline experiments, online evaluation, and manual annotation into a closed loop.
• The event model covers models, tools, links, complete sessions, and multi-agent trajectories.
• Based on OpenTelemetry, it can be used in conjunction with existing observability infrastructure.
• It also supports code evaluation, model assessment, manual scoring, and user feedback.
• The dataset can be continuously compiled and versioned directly from actual production failures.
• The prompt version, deployment, as well as experiment and tracking results are all part of the same project.
• The free version does not require a credit card and offers 10,000 events as well as support for up to 5 users.
• Enterprises can choose multi-tenant, single-tenant, hybrid, and fully self-hosted options.
• OpenAPI, CLI, and various SDKs are provided to facilitate integration and automation.

Usage restrictions and precautions

• The free version counts events rather than user requests, and complex proxies can quickly deplete the quota.
• Developers only retain data for 30 days, which is not suitable for long-term compliance audits.
• The free cloud data is located in the AWS US West region; cross-border transfer requirements need to be assessed in advance.
• Detailed tracking may include logs, documents, personal information, keys, and internal code.
• Automatic stub insertion increases the number of events and may affect performance as well as log costs.
• LLM adjudicators can make mistakes, be affected by prompt injection, and incur additional model costs.
• The server-side model used for judgment in the document is GPT-4o; for sensitive tasks, it is necessary to verify the data pathway.
• Self-hosting and single-tenant options are available only in Enterprise, and their prices are not disclosed.
• The platform is not fully open-source; the public SDK cannot replace the commercial console and backend.
• Some warehouses have been archived or lack clear licenses; new projects should verify their maintenance status.
• Dashboards and green ratings cannot replace security testing, red team exercises, or actual business outcomes.
• Production quality standards need to be updated continuously, as fixed test sets may gradually lose their relevance.

Security, Privacy, and Compliance

Tracking data may include user inputs, model responses, retrieved documents, tool parameters, business identifiers, feedback, and code context. The team should perform field-level data masking before sending this data, use separate project keys, and isolate the data for production, testing, and development environments.

Safety and Compliance Statement

HoneyHive claims that both the data at rest and data in transit are encrypted, and that it meets the requirements of SOC 2 Type II, GDPR, and HIPAA; in addition, it undergoes penetration testing by third parties. Regulated clients should still request current reports, details regarding the scope of coverage, any exceptions, DPA agreements, BAA agreements, as well as information on sub-processors.

Enterprise identity and access

The Developer version offers social single sign-on and basic role control; the Enterprise version adds SAML, custom single sign-on, as well as custom roles and permission groups. Project access and workspace access are managed separately, and members still need to be explicitly added to a project.

Prompt provider key

Playground requires connection to model providers; the current documentation states that the provider keys are stored in the browser’s cache in an encrypted form. Companies should ensure that browser management, device sharing, cache cleanup, and key rotation comply with their internal policies.

Basic information

Frequently Asked Questions

What is HoneyHive?

It is a platform for the monitoring and evaluation of AI agents, used for tracking the behavior of models and tools, monitoring production quality, running experiments, managing datasets, organizing manual annotations, and controlling version updates.

Is HoneyHive free?

The Developer version is free and does not require a credit card; it includes 10,000 events per month, up to 5 users, 1 workspace, unlimited projects, and 30 days of data retention.

How many requests are equivalent to 10,000 events?

There is no fixed conversion rate. Each span and metric is counted as an event; a proxy request that includes multiple tools, retries, and metrics can consume dozens or even more events.

Which types of evaluations are supported?

It supports the evaluation of client and server code, LLM judging, manual annotation, and user feedback, and can be used to assess relevance, fidelity, trajectory, tool accuracy, security, and business metrics.

Can it be self-hosted?

Yes, but it is available only for Enterprise users. Enterprises can choose a fully self-hosted approach, a hybrid model with a hosted control plane along with a private data plane, or a single-tenant SaaS solution.

Is Python and TypeScript supported?

Supported. Python is used for tracking and evaluation, while the active TypeScript API Client is utilized for platform interfaces; other languages can send OpenTelemetry data or generate clients based on OpenAPI.

Is HoneyHive open source?

The platform itself is not a fully open-source product. OpenAPI, the CLI, certain TypeScript clients, and various skills are licensed under the MIT license; other repositories require separate verification.

Is Realign still under maintenance?

No further maintenance will be provided. The Realign warehouse was made read-only in December 2025; new projects should prioritize using the current HoneyHive evaluation framework.

Is it possible to monitor multi-agent systems?

Yes. The platform is capable of recreating sub-agents, tools, loops, retries, and handovers, but it requires that the tracking context and identifiers be transmitted correctly within the application.

Are LLM judges reliable?

It cannot be assumed to be reliable outright. A representative sample and expert evaluations should be used for calibration, and attention must be paid to model upgrades, prompt injection, bias, stability, and cost.

Is it suitable for processing medical data?

Enterprise solutions provide a BAA and claim to meet HIPAA compliance requirements, but customers still need to verify that the deployment methods, model evaluation processes, sub-processors, and tracking fields are all within the scope of coverage.

Can it be migrated from Langfuse?

The document outlines the steps for migration tracking, evaluation, and handling of historical data, and offers both private cloud and self-hosted options for enterprises. Before migrating, it is necessary to compare field mappings, attachments, metrics, and retention policies.

Summary

HoneyHive is suitable for teams that have already incorporated AI agents into actual development or production processes and need to understand why those agents fail, how to assess their quality, and how to prevent regressions. Its core value lies in enabling production activities, domain experts, and engineering experiments to be integrated into a single continuous learning cycle.

Small teams can use the free version to test event models and evaluation processes, while manufacturing companies should focus on assessing event costs, 30-day data retention, data anonymization, model decision-making pathways, identity management, and deployment boundaries. Only by combining evaluations with actual business outcomes, as well as red-team testing and human oversight, can we avoid mistaking observable dashboards for a sufficient proof of system reliability.