Cleanlab

What is Cleanlab?

Cleanlab is a set of technologies aimed at ensuring data quality and the reliability of AI outputs; it includes open-source Python libraries, the Cleanlab Studio data processing service, Trustworthy Language Models, as well as a control platform for enterprise AI agents. It enables the detection of dirty data before training begins, and it can also identify potentially incorrect responses while generative AI is in operation.

On January 28, 2026, Cleanlab was acquired by Handshake; the team announced that it would focus on researching high-quality data needed for training cutting-edge models. The acquisition announcement promised that the open-source AI platforms would continue to be maintained, but no similar commitment was made regarding the long-term independence of the commercial platforms.

Product composition and open-source boundaries

Making the client code public does not mean that the backend services, scoring technologies, or the enterprise platform as a whole are open source. When offline use is required, it is necessary to first verify whether the desired functionality is indeed included in the Apache-2.0 cleanlab package.

Open-source data quality inspection

• Datalab receives data, labels, model prediction probabilities, or feature vectors, and automatically checks for label errors, outliers, near-duplicates, class imbalance, and other data issues.
• CleanLearning can wrap existing classifiers to identify suspicious samples in the presence of noisy labels and train more robust models.
• The multi-annotator tool can estimate consensus labels and annotator quality, helping to determine which samples need to be re-annotated.
• The ability to learn proactively allows for the identification of the next batch of data that requires labeling or review, thereby reducing waste in human resources.
• The dedicated interfaces cover binary classification, multi-class classification, multi-label classification, entity recognition, regression, image segmentation, object detection, and out-of-distribution detection.

Open-source libraries typically rely on existing models to provide out-of-sample prediction probabilities or feature representations; they are not instant solutions that require no preparation of models or data. The results of detection determine the priority for review, and data cannot be deleted in bulk without human judgment.

Data organization in Cleanlab Studio

• The web interface is suitable for viewing problem samples without coding, understanding the data quality fields generated automatically, and making corrections.
• The Python API supports files, Pandas DataFrames, and PySpark DataFrames, enabling automated uploading, execution of tasks, and downloading of the resulting data.
• The command line is suitable for authentication in data pipelines, uploading data, creating projects, and exporting cleansets.
• It can handle tasks related to text, images, structured tables, documents, multi-label data, multi-annotator data, and synthetic data quality.
• The document outlines the integration processes for Databricks, Snowflake, and BigQuery; the actual permissions and methods of data transfer depend on the deployment configuration.

Common inputs and outputs

TLM reliability score

The core output of TLM is a trustworthiness score ranging from 0 to 1. It can generate responses on its own and assign scores to them, or it can simply evaluate existing responses provided by any large model or written manually.

• RAG: It takes into account the user’s questions, system instructions, and the retrieval context, in order to identify irrelevant information, factual errors, and inadequate responses.
• AI Agent: It can evaluate the final response, as well as check the calls to internal models and tools; it triggers manual approval before actions that would result in a low score are carried out.
• Structured extraction: Scores the fields extracted from documents, databases, or transcriptions, and sends those with low reliability for further review.
• Classification and labeling: Restricting candidate categories, evaluating automatic labels, and identifying samples that show significant conflicts with manual labels.
• Offline evaluation: Batch-scrape production logs or fine-tuning data, focusing on the most unreliable responses rather than relying on random sampling.

The TLM score is neither a proof of fact nor a fixed threshold for compliance. Documents suggest that a score below 0.7 is generally considered low, while a score above 0.9 is considered high; however, the interpretation of scores between 0.7 and 0.9 varies depending on the task, and the thresholds must be adjusted based on actual errors and business consequences.

Closed-loop detection and repair by AI Agent

• Detect adds an independent control layer separate from the existing agents and knowledge base, in order to identify hallucinations, retrieval errors, document gaps, policy violations, and malicious use.
• Low-confidence responses can be hidden, replaced with more conservative replies, requests can be made for additional information, the process can be upgraded to a more advanced model, or the matter can be referred to human handling.
• Remediate forwards the failed cases to business experts, who can correct the answers, add additional knowledge, or adjust the rules, without the need to modify the model code directly.
• The fixes can be reused to address similar issues, thereby creating an iterative process of generating operation logs, expert judgments, and updates to the knowledge base.
• The claim of compatibility with any AI system still requires practical integration testing, including message format, streaming responses, tool calls, and identity permissions.

TLM quick access process

1. Create a Cleanlab account and obtain the TLM API key; place this key in the server’s environment variables.
2. Install the TLM Python client, and use it to invoke the scoring method based on small-scale real samples, or simply use the existing response scoring method.
3. Pass the complete system commands, user queries, search results, and historical messages as they were actually entered in the application to TLM, so as to avoid missing any context that could affect the judgment.
4. Manually label correct and incorrect samples, and compare different base models, quality presets, inference strength, and similarity settings.
5. Select thresholds based on the cost of errors, and design fallback strategies such as switching to manual handling, rejecting requests, performing additional searches, regenerating the model, or upgrading it.
6. After going live, monitor the low-score rate, false positives, missed detections, delays, and token costs; recalibrate these values regularly instead of setting fixed thresholds permanently.

Process for using open-source libraries

1. Install cleanlab, and prepare the raw data, labels, and the model suitable for the task.
2. Out-of-sample prediction probabilities are generated through cross-validation, or feature vectors that can represent sample similarity are extracted.
3. Run Datalab’s issue detection and reporting functions to examine the most suspicious samples based on issue type, score, and category.
4. Have experts in the field identify samples with incorrect labeling, duplicates, abnormalities, or those outside the expected distribution, and document the decisions taken regarding their retention, correction, relabeling, or deletion.
5. Retrain the same model using the organized data, and compare its performance on an independent and verified test set.
6. Integrate data inspection into the continuous pipeline to monitor changes in new batches, drift, and annotator quality.

Suitable for users and scenarios

• Data scientist: Identifying issues with the training set and test set without changing the model framework.
• Machine Learning Engineer: Establishes ongoing data quality controls for text, image, audio, and spreadsheet tasks.
• LLM and RAG teams: Assess the reliability of responses, retrieve context, and perform structured extraction of results.
• Customer service and employee assistance teams: block high-risk responses and pass the failed samples to business experts for correction.
• Regulated enterprises: They manage data areas, networks, and operational responsibilities through single-tenant or self-managed VPCs.

Prices and plans

The current public website no longer provides a complete list of prices, and it is not possible to use the amounts associated with previous packages. The TLM documentation states that free tokens can be obtained; after they are used, charging will take place based on those tokens. However, the number of free tokens available and their specific price are only displayed on the account’s Usage and Billing pages.

Different base models, quality settings, numbers of candidate responses, and reasoning parameters affect the cost and latency of TLM. The available public information does not specify anything regarding refunds, the validity period of free tokens, or long-term price commitments after acquisition; it is necessary to refer to the account settlement page and the contract before making any payments.

Deployment and platform

Capacity boundaries

• The credibility score is a signal of the risk of errors; it may result in low scores for correct answers or fail to detect mistakes. Manual review cannot be bypassed for decisions with a high risk level.
• The score for the same query may not be fixed; caching usually allows repeated requests to yield the same results temporarily, but according to the documentation, the cache is refreshed after about 30 days or when there is an internal update.
• The quality of open-source libraries depends on the model’s prediction probabilities, feature representations, cross-validation methods, and task assumptions; poor quality of the input signals reduces the diagnostic value.
• Automatic label correction may mistake a small number of but genuine edge cases for errors; audit records and recoverable copies should be kept before deletion.
• Following the acquisition by Handshake, the ongoing maintenance of the open-source packages is now guaranteed, but commercial products, account access options, and the enterprise roadmap still need to be confirmed prior to making a purchase.

Privacy, Security, and Terms

• The privacy policy outlines registered information, uploaded data and code, usage records, devices, inferred location, communications, and information related to third-party accounts.
• Payments are processed by Stripe, and site analysis may be carried out using Google Analytics; content uploaded to SaaS, Studio, or TLM is handled by those services.
• The privacy policy permits the processing of information for the purposes of maintenance, improvement, development of new products, analysis, and anonymized aggregation, but it does not specify a fixed retention period for uploaded content nor clear rules regarding what is excluded from model training in the public text.
• The security page states that it has passed SOC 2 audits, as well as independent penetration testing, static analysis, dynamic analysis, and continuous security audits.
• The terms of service prohibit competitive access and commercial benchmarking without written consent; non-commercial benchmarks also require that the experimental code and data be made publicly available.
• Both the terms and the privacy policy came into effect on February 18, 2022, which is prior to the establishment of the current business platform as well as the acquisition in 2026; enterprise customers should request the most up-to-date versions of the order forms, data processing agreements, and lists of subcontractors.

Summary

What sets Cleanlab apart is its ability to integrate data training diagnostics, LLM output evaluation, and manual correction processes, while also offering a data quality database under the Apache-2.0 license that can be used locally. The most prudent approach at present is to differentiate between open-source libraries, TLM, Studio, and enterprise control platforms based on specific needs, and to re-evaluate aspects such as service continuity, pricing, and data terms after making an acquisition.