Maxim AI

What is Maxim AI?

Maxim AI is an end-to-end evaluation and observability platform provided by H3 Labs Inc. for teams responsible for AI products, engineering, and quality. It integrates prompt testing, datasets, agent simulation, automatic and manual evaluation, production tracking, quality monitoring, and alerting into a single iterative process.

Maxim is not responsible for training basic models on behalf of users, nor does it maintain any ranking list of models. It is used to compare different models, prompts, retrieval methods, tools, and agent versions in order to identify areas of regression or failure, and it continuously monitors the quality, cost, latency, and security aspects of actual requests once they are deployed.

Main functions

• Prompt IDE: Allows for comparing different prompts, models, parameters, tools, and contexts without the need to modify the business code. Teams can save the results of their experiments, assess quality, cost, and latency, and then choose the version that is suitable for deployment.
• Prompt versioning and deployment: Prompts are extracted from the code repository, their modification history is retained, and different versions are released according to established rules. Product and engineering team members can collaborate within the same workspace, which helps to avoid version discrepancies that arise from copying and pasting.
• Prompt chains and low-code workflows: They enable the connection of multiple steps related to generation, retrieval, and tools into testable workflows, which is useful for verifying the logic of intelligent agents before they are integrated into actual code. Complex production permissions, retry mechanisms, and transactions still require implementation through engineering approaches.
• Multimodal dataset: It manages test inputs such as text and images along with the desired results, and supports table-like editing. The dataset can be created by organizing production logs, or it is possible to import existing test samples.
• Evaluator Store: It provides pre-built metrics and allows teams to create custom evaluators to measure accuracy, relevance, security, format, retrieval, or business standards. The scores assigned by the models can also be biased; therefore, human-based benchmarks should be used in critical scenarios.
• Agent simulation: Multiple rounds of interactions are generated using different user profiles, objectives, and scenarios, in order to test on a large scale how agents use tools, perform searches, and manage their state. Simulation is useful for expanding the scope of testing, but it cannot fully represent the actual distribution of users.
• Offline evaluation: Run candidate versions on fixed datasets before deployment, and compare input data, tool calls, outputs, metrics, and the reasons for evaluation on a per-item basis. This approach is suitable for regression testing, model migration, and prompt optimization.
• Online evaluation: Production logs are automatically scored based on filtering and sampling rules, and the evaluation can be carried out at the session level, per trace round, or at the specific Span node level. Teams can collect samples with low scores to form new datasets, thus creating a cycle of continuous improvement.
• Distributed tracing: It uses Session, Trace, Span, Generation, Retrieval, and Tool Call to represent multi-agent workflows, allowing for a visual inspection of the inputs and outputs, execution time, and errors of each node.
• Production tuning and alerts: Monitors quality, tokens, costs, latency, user feedback, and fault trends, and issues alerts when metrics decline or security rules are triggered. Alerts should be based on business-specific thresholds and defined workflows, rather than relying solely on default rules.
• Manual evaluation: Domain experts are tasked with annotating, reviewing, and comparing samples, with the results being included in the evaluation report. Enterprise also offers discussion options regarding the human evaluation services managed by Maxim.
• Reports and dashboards: Aggregate experimental and live metrics, compare different versions, and share them with product, engineering, and management teams. Business plans support custom dashboards, while Enterprise offers enhanced audit and service level capabilities.

Typical workflow

1. Define the business objectives of the agent, the levels of failure that are unacceptable, and the user segments; transform accuracy, security, cost, and latency into measurable metrics.
2. Collect authorized real-world questions, edge cases, and historical failures to create evaluation datasets and manual reference answers beyond those used for training.
3. In the Prompt IDE, compare models, prompts, retrieval, and tool configurations; first check with a small sample size, and then run a full offline evaluation.
4. For key metric combination rules, code, model scoring, and manual review, to prevent a single automated evaluator from deciding whether something should be launched or not.
5. Integrate evaluation into CI/CD: prevent the release of versions that fall below the threshold, and save records of the versions, datasets, and scores.
6. Integrate logging and tracking through the SDK to assign consistent names, tags, and user context to sessions, creations, retrievals, and tool calls.
7. After going live, configure online evaluation, sampling, dashboards, and alerts to monitor quality, cost, latency, errors, and user feedback.
8. The actual low scores and abnormal traces are combined to form new regression samples; after making the necessary corrections, offline testing is conducted again to compare the trends.

Evaluation hierarchy

Prices and packages

Professional and Business currently offer a 14-day free trial. The comparison table also shows that the extra charge for log entries in Professional and Business is $1 per 10,000 entries; no such extra charges apply to Developer plans. Specific fees, annual discounts, and prices for other usage levels can be found on the billing page.

Check before purchasing

• Determine whether the logging unit is requests, Traces, or other events, and estimate how many records will be generated by multi-agent calls.
• The data retention period varies significantly, ranging from 3 days to 30 days; separate retention strategies should be established for troubleshooting, auditing, and regression samples.
• Both seat fees and log-overage fees apply, and both team expansion and increased traffic can raise costs.
• The terms available publicly do not provide clear guidelines for self-service refunds; it is necessary to confirm the procedures for cancellation, the date of effectiveness, and what happens to any unused periods before signing a contract or subscribing.

Which users are it suitable for

• It is necessary to establish product and engineering teams capable of carrying out repeatable regression testing for AI agents.
• Developers of LLM applications also compare various models, RAG, tool calls, and prompt-based versions.
• After going live, the operations and platform teams need to monitor the paths, quality, cost, and latency of multiple agents.
• Enterprises that require RBAC, PII management, auditing, VPC, SSO, or compliance support.
• Domain standards are difficult to automate entirely; fields such as healthcare, finance, customer service, and legal support require manual review by experts.

Advantages

• It covers experiments prior to deployment and observations after deployment; offline and online evaluations can share data and metrics.
• It also offers a no-code interface and multilingual SDKs, enabling product managers and engineers to work together within the same workflow.
• The tracking granularity extends to retrieval, tool, and generation nodes, facilitating the identification of specific faults in complex agents.
• It supports a combination of pre-built, custom, automatic, and manual evaluations, thereby avoiding reliance on just one scoring model.
• Companies can opt for VPC deployment and data isolation, which is suitable for teams with specific requirements regarding data boundaries.

Capabilities boundaries and limitations

• The evaluation scores do not reflect the actual business outcomes; insufficient coverage of the dataset or biases in the evaluators can lead to false confidence.
• As an arbiter, a model may favor certain styles or its own family of models, and it must be calibrated using manual labeling.
• Production tracking may include prompts, user content, search snippets, and tool parameters; improper integration can lead to an increased exposure of sensitive data.
• Free and low-cost plans have a short retention period, making them unsuitable for those who need to store audit evidence on the platform over the long term.
• The volume of logs increases with the number of agents, retries, and nodes; therefore, cost estimation cannot be based solely on the number of requests from end users.
• The platform cannot replace security testing, red teaming, business approval processes, and incident response; it is merely part of the infrastructure for quality management.

SDK and Integration

Maxim provides SDKs for Python, JavaScript or TypeScript, Java, and Go, which enable session and trace recording from application code, as well as generation, retrieval, and tool invocation, and facilitate the initiation of evaluations. The platform can also work in conjunction with models or frameworks such as OpenAI, Claude, Gemini, LangChain, LangGraph, CrewAI, and others.

1. Create API keys and log repositories in the workspace, with environments separated for development, testing, and production.
2. Install the SDK for the corresponding language, store the keys in the server’s secret management system, and prevent them from being uploaded to the repository.
3. Initialize the Logger, use stable identifiers for Sessions and Traces, and assign names and tags to key nodes.
4. Log input, output, execution time, tokens, costs, and errors around generation, retrieval, and tool invocation.
5. First, verify data masking and sampling in the testing environment, and then gradually introduce production traffic to prevent sensitive fields from being recorded accidentally.
6. Configure online evaluation and alerts to send samples with low scores to the dataset or to the manual review queue.

Bifrost open-source AI gateway

Bifrost is an open-source AI gateway independently developed by the Maxim team; it is licensed under the Apache 2.0 license. It enables unified connection to multiple model providers and handles tasks such as routing, load balancing, fallback mechanisms, rate limiting, security controls, logging, and cost monitoring. While it can be integrated with the Maxim platform, the fact that Bifrost is open source does not mean that the Maxim SaaS service itself is open source.

• It can be run locally via a container or the command line, and it allows connection to multiple providers through a unified interface that is compatible with OpenAI’s request format.
• It supports adaptive load balancing, clustering, plugins, model routing, MCP, and a monitoring interface; its capabilities are continuously updated with each new version.
• The default configuration requires security enhancements; if the encryption key is not specified, some local data may be stored in plain text. Production environments should not use the settings from the demonstration mode.
• The costs associated with invoking models are still incurred by the respective providers; the open-source gateway does not include any quota for models.

Open-source status

Privacy, Security, and Data

• The platform may handle accounts, Google or GitHub logins, contact information, usage data, as well as the prompts, files, inputs, outputs, and observation logs submitted by customers.
• The terms state that the customer data and outputs belong to the customer or its licensor, with Maxim being granted a limited license to use them for the purpose of providing, operating, and supporting the services.
• The terms state that, without the customer’s prior written consent, the customer’s data will not be used to train, fine-tune, or improve models, nor will functions be developed for other customers.
• The privacy policy outlines the encryption of sensitive data, two-factor access to databases, regular security audits, data minimization, access controls, and procedures for notifying in the event of a data breach.
• The product is certified to meet SOC 2 Type II, ISO 27001, HIPAA, and GDPR standards; for regulated procurement, it is still necessary to obtain the current scope of certifications, a DPA, a BAA, and a list of sub-processors.
• Enterprise offers deployment within a VPC, data isolation, custom retention options, and security audits, making it suitable for organizations that do not allow logs to leave their own environment.
• The platform can be used for detecting and handling security issues and violations; the privacy policy also states that illegal content may be separated from user identifiers in order to be utilized in trust and security systems.
• Before integration, PII should be identified, masked, and minimized for prompt texts, user inputs, search documents, tool parameters, and model outputs.

Terms and purchasing notes

• At the top of the terms page, the old description of the online courses from 2023 remains, but the customer data terms on the same page have been updated to reflect the current AI platform; there are therefore inconsistencies in the text due to historical changes.
• Enterprise clients should request a valid order, a main service agreement, and a data processing agreement that correspond precisely to the current evaluations, observations, SDKs, and Bifrost integrations.
• Define the priority in the contract regarding data and output ownership, the prohibition on model training, deletion, export, retention, service termination, and audit evidence.
• Do not assume that the presence of SOC, ISO, HIPAA, or GDPR implies that your application is automatically compliant; the customer remains responsible for determining the purpose of use, making the necessary configurations, and informing users.

Summary

Maxim AI is suitable for upgrading the quality management of intelligent agents from ad-hoc manual inspections to a continuous engineering process that includes experimentation, simulation, offline evaluation, CI, production tracking, online evaluation, and regression datasets. When selecting a package, it is necessary to take into account the number of seats, the amount of log data, and the retention period; moreover, it is important to clearly distinguish between the proprietary Maxim platform, the public SDK, and the Apache 2.0 open-source Bifrost gateway.