OpenCompass

What is OpenCompass?

OpenCompass is an open-source platform for evaluating large models, developed by the Shanghai Artificial Intelligence Laboratory. It is used to systematically compare basic models, chat models, and multimodal models. It offers an evaluation framework, dataset configuration, model adaptation, distributed inference capabilities, metric calculation, result aggregation, and ranking functions; it is suitable for research institutions, model development teams, and enterprises to establish reproducible processes for evaluating LLMs.

OpenCompass refers both to the open-source Python tool that can be installed locally, and also to the rankings of LLMs and VLMs available on its official website, as well as to ecosystems such as CompassHub and CompassRank. The results of these rankings are based on specific versions, prompts, datasets, and evaluation methods; they do not indicate that a particular model is superior in all real-world applications.

Core functions of OpenCompass

Multidimensional ability assessment

The ability framework covers general dimensions such as testing, knowledge, language skills, comprehension, reasoning, and security; it also includes specialized skills like handling long texts, working with code, using tools, and enhancing knowledge. Users can choose from existing evaluation sets or combine datasets and metrics that suit their specific needs.

Over a hundred datasets and configurable operation

The project includes a large number of references in both Chinese and English, as well as professional standards. Models, datasets, prompts, inferencers, and evaluators are defined through Python configuration files. These configurations can be inherited and combined, which facilitates evaluating the same model across different datasets or comparing multiple models under identical conditions.

Local model and API model

OpenCompass can directly load local models such as those from Hugging Face and InternLM, and it can also connect to services like OpenAI and DeepSeek that provide compatible APIs. Custom services can see an increase in throughput thanks to tools such as LMDeploy, vLLM, and SGLang.

The closed-source API model does not require a local graphics card, but it incurs token fees and is subject to limitations imposed by the supplier as well as its data policies.

Objective evaluation

For multiple-choice questions, fill-in-the-blank tasks, and those with standard answers, discriminative or generative evaluation methods can be used. The discriminative approach compares the degree of ambiguity among the candidate answers, while the generative approach has the model generate an output directly, which is then processed according to certain rules.

The Prompt template, few-shot examples, and answer extractor all affect the score; when repeating experiments, it is necessary to keep the configuration and random seed constant.

Subjective evaluation and LLM-as-Judge

For open-ended responses, security, expression, and complex reasoning, human judges or large language models can be used as evaluators. GenericLLMEvaluator allows for the customization of the evaluation prompts, reference answers, and the judge model.

CascadeEvaluator allows for evaluation using rules first, and only those samples that fail the evaluation are sent to the LLM, thereby reducing the cost of calls.

LLM reviewers exhibit positional bias, style preferences, self-preferences, and randomness. For high-risk conclusions, multiple reviewers should be used, the order of answers exchanged should be varied, manual checks should be conducted, and the review models and templates should be made public.

Distributed and efficient evaluation

The framework allows for the separation of inference and evaluation tasks, enabling their execution on multiple GPUs or machines. It also facilitates the management of large-scale experiments through task monitoring, re-running failed tasks, and persistent storage of results. While an accelerated inference backend can increase throughput, different backends, quantization methods, and parallelization settings may result in varying outputs, which should be recorded separately.

Multimodal and specialized ecology

The OpenCompass ecosystem includes projects such as VLMEvalKit and AgentCompass, which deal with visual language models and Agent capabilities respectively. Each of these sub-projects has its own method of installation, as well as its own datasets and licenses; it’s not possible to consider all multi-modal capabilities as something that can be handled through a single command within the main repository.

Rankings and reports

The official website provides a list of available models; the framework can generate tables, summaries, and detailed sample results. When examining this list, it is important to pay attention to the date of the model, whether it is closed-source, issues related to contamination in the test set, the prompt used, the inference budget, and the range of samples covered, rather than focusing only on the overall average score.

Comparison of usage methods and costs for OpenCompass

Comparison of evaluation methods

OpenCompass Installation Guide

1. Prepare the environment:Create an independent Python environment; local models install the necessary dependencies based on PyTorch, CUDA, and the graphics card.
2. Installation project:Install the stable or development version from the official repository, and record the commit hash.
3. Prepare the dataset:Download the required benchmarks according to the document, and verify the license, directory, and validation results.
4. Select model configuration:Use existing local or API configurations to set the model path, key, concurrency, and context.
5. Run small-scale tests:First, use a small amount of data to test the prompt, output parsing, video memory usage, and costs, and then run the entire dataset.
6. View results:Check the summary, logs, and detailed sample information; don’t rely solely on the final average score.
7. Save experiment information:Record the model version, dataset, configuration, backend, hardware, and time to facilitate reproduction.

Evaluation of custom datasets

1. Organize JSONL or CSV files by identifying the questions, answer keys, categories, and metadata fields.
2. Define the input and output columns of the Dataset Reader, and separate the development set from the hidden test set.
3. Configure the Prompt Template, Retriever, and Inferencer for the model.
4. Choose rules, professional metrics, or LLM Judge, and write a reliable post-processing function.
5. Use manually labeled small datasets to validate the evaluator, then scale up and analyze the grouping results.

How to interpret evaluation results correctly

The overall score may hide weaknesses in individual aspects; it is therefore necessary to consider the categories of language, domain, difficulty, and security. The context length, sampling parameters, inference tokens, and tool permissions for different models should be as consistent as possible.

Public data may be included in the training corpus; therefore, organization-specific datasets, time-updated data, and real-world business datasets should also be added to avoid optimizing only the rankings.

Which users are it suitable for

• Researchers and algorithm teams that compare the capabilities of open-source and closed-source large models;
• Enterprise technical leaders who need to establish objective criteria for selecting models;
• Teams that develop reasoning, coding, security, long-text, or multimodal models;
• Developers who wish to recreate the public ranking list and analyze specific samples;
• Organizations that need to create private datasets and conduct ongoing regression testing.

Advantages and precautions

• The advantages of OpenCompass are its open-source nature, a well-developed Chinese ecosystem, a rich set of datasets and model configurations, as well as coverage of the entire process from inference and metrics to reporting.
• It is suitable for quick benchmarking as well as can be expanded into an internal evaluation platform within an organization;
• Installation and a thorough evaluation require certain engineering resources and computing power, and the download size of the datasets can also be large.
• The evaluation score is influenced by the prompt, answer extraction, backend system, and the judges, so it cannot be regarded as an absolute ranking of the model.
• Data sets, model weights, and APIs each have their own licenses and privacy policies, and isolated containers should be used when running code for evaluation.

Frequently Asked Questions

Is OpenCompass free?

The open-source framework is free and licensed under the Apache 2.0 license; its features can be viewed for free on the official website. The costs related to local hardware, cloud GPUs, model APIs, and evaluation models are borne by the user.

Can OpenCompass evaluate closed-source models?

Yes, it is possible to connect through the corresponding API, but this will be subject to the supplier’s rate limits, pricing, and data policies.

Does OpenCompass support multimodal input?

The OpenCompass ecosystem offers various multimodal evaluation tools such as the VLM rankings and VLMEvalKit; for details on installation and datasets, please refer to the respective projects.

Is being at the top of the rankings necessarily the best thing?

Not necessarily. The rankings reflect only specific criteria and configurations; the choice of solution should be made by taking into account actual business needs, costs, latency, security aspects, as well as human evaluations.

Is OpenCompass open source?

It is open source; the main repository is licensed under Apache License 2.0. Certain datasets, models, and related sub-projects are governed by their own licensing terms.