LiveBench

What is LiveBench?

LiveBench is a set of continuously updated benchmark tests and public rankings for large language models, designed to address the issue of contamination that occurs when traditional evaluation tasks are included in the model training data. It utilizes up-to-date information, programmatically generated questions, and verifiable standard answers, enabling automatic scoring of complex tasks.

The project paper has been selected for ICLR 2025 Spotlight; the organization has made the website, the paper, the data, and the evaluation code available to the public. It is suitable for studying differences in the capabilities of various models, and it is not a tool for chatting or generating content intended for ordinary users.

Why is a dynamic benchmark needed?

The longer a fixed set of questions remains available, the more likely it is to be used by model developers for training, fine-tuning, or prompt optimization; as a result, the scores on the rankings may reflect memory rather than generalization ability. LiveBench tries to minimize the time during which these questions are exposed by regularly adding new ones and updating old ones.

\"Limited pollution\" does not mean complete absence of pollution; public web pages, competition problems, and code repositories can still end up in the training data. When interpreting the results, it is necessary to take into account the date on which the problems were published, the deadline for model training, and the specifics of each task.

Core design

Continuously update the questions.

LiveBench adds or replaces questions based on newly released datasets, papers, news articles, movie trailers, and competition materials. The update log records the changes to each question along with its date of release.

Objective standard answers

For each question, it is advisable to use verifiable ground truth values and deterministic scoring methods, rather than relying on the subjective judgment of another large model. This helps to reduce biases, order effects, and self-prefrences in the evaluating model.

Multiple capability coverage

The benchmarks cover areas such as mathematics, programming, reasoning, language, instruction following, data analysis, and agent programming. The number of categories, tasks, and questions varies across different versions; therefore, the initial setup of “six categories and eighteen items” should not be considered a permanent standard.

Versioned rankings list

Each update creates a release version with a date assigned to it, and the model’s performance should be linked to the specific version that was released. Different versions have different question sets and scoring mechanisms, so it is not possible to compare scores directly without providing relevant explanations.

Open evaluation code

The official repository provides scripts for generating responses, assigning scores, and displaying results, and it supports API models as well as compatible interfaces. For agent programming tasks, Docker is required to provide an isolated execution environment.

Current evaluation category

List of main functions

• View the scores of different models overall and across various ability categories.
• Compare performance by model, organization, version, and inference settings.
• Understand question updates and ranking changes through date versions.
• Download the public data and view the task structure and standard answers.
• Use the official script to generate model responses and objective scores.
• Evaluate commercial API models or custom compatible interface models.
• Run subsets of tasks such as mathematics, programming, and reasoning independently.
• Submit a request to evaluate the model or report issues through an Issue.
• Read the papers and update logs to understand the methods and limitations.
• Review the specific tasks and model outputs corresponding to the abnormal scores.

Which users are it suitable for

• AI researchers: Analyze model generalization, contamination, and differences in capability.
• Model development team: Conducts regression testing on the questions in the new version.
• Enterprise technology selection specialists: Identify candidate models suitable for specific tasks.
• API platforms and model vendors: Verify new models and submit them to the ranking list.
• Open-source community: Reproducing results, evaluating fixers, and contributing new tasks.
• Development Tools Team: Focuses on comparing coding skills and agent programming capabilities.
• Educators and media researchers: Understanding ranking methods and common misinterpretations.

What situations are not very suitable

• Users who only want to know about the absolutely “best model” without considering differences in tasks.
• Teams that require specialized evaluations regarding Chinese language writing, customer service tone, or industry knowledge.
• It is hoped that a single ranking system can directly replace the procurement decisions made as part of actual business trials.
• Individuals who want to replicate all the tests in their entirety but lack an API budget, GPUs, or a sandbox environment.
• Organizations that require comprehensive certification with regard to security, bias, privacy, and the reliability of facts.
• Ordinary users who need to communicate via mobile apps or directly carry out production tasks.

Tutorial for viewing the rankings

1. First, verify the release date and version of the topic shown on the page for LiveBench.
2. View the model name, specific version, inference intensity, and context settings.
3. First, compare the categories corresponding to the target business, rather than focusing only on the overall average score.
4. Expand the task-level results to see if high scores are concentrated in a few types of questions.
5. Check for timeouts, empty responses, cost limits, or tool differences.
6. Place the candidate models in one’s own data and workflow for a second evaluation.

Run evaluation tutorials locally

1. Obtain the official repository and create a separate Python virtual environment.
2. Install the basic dependencies and task dependencies as specified in the project instructions.
3. Select a fixed LiveBench release option and record the version being submitted.
4. Configure the model name, API address, key, and generation parameters.
5. First, run a small number of questions to verify the output format and costs.
6. Perform reasoning and scoring, and save the original responses along with the execution logs.
7. Compare different models using the same version and parameters.
8. After rechecking the abnormal samples, the categories and total scores are compiled.

Tutorial on Agent Programming Evaluation

1. Prepare isolated hosts that will support Docker and set resource limits.
2. Install the additional dependencies and images required for agent programming tasks.
3. Verify that the model supports the required tool calls or interaction methods.
4. Fix the agent framework, step limit, timeout, and environment version.
5. Run a small-scale sample to check warehouse mounting and test commands.
6. Execute tasks in batches and save trajectories, patches, tests, and costs.
7. Failed tasks are categorized as model errors, environment errors, and timeouts.
8. Destroy untrusted containers and clean up temporary credentials.

How are scores calculated?

Task scoring is carried out using deterministic rules, standard answers, test cases, or specialized processing procedures, and the results are then aggregated by task and category. The overall score is usually derived from the average value of each category, rather than by simply adding up all the scores together.

Version updates may add new categories, replace tasks, or modify the scoring logic; therefore, it is necessary to record the date of release, code commits, model parameters, and the operating environment. Referring to only a score without specifying the version undermines reproducibility.

Version and result comparability

Pollution control methods

• New questions based on recent materials are published regularly.
• Build tasks from new papers, news, competitions, and datasets.
• Some verifiable questions are generated using a programmed approach.
• Update old tasks that are already saturated or suspected to be contaminated.
• Link the rankings and data to the specific publication dates.
• Using objective answers helps reduce biases in human and model judging.

These measures can only reduce the likelihood of pollution; they cannot prove that no models have ever come across such materials. In particular, open-source code, news articles, and research papers may soon be incorporated into training or retrieval systems.

Advantages and limitations of objective scoring

Price and operating costs

The LiveBench website, papers, and open-source repositories are accessible for free; the project does not offer any membership plans for users. The main costs associated with conducting one’s own evaluations stem from the use of model APIs, GPUs, agent sandboxes, storage resources, as well as the time required for development work.

How should model selection be carried out?

LiveBench is useful for reducing the number of candidate models from dozens to just a few, but it should not be used as the sole criterion for making a purchase decision. The business side still needs to evaluate factors such as price, latency, throughput, data policies, stability, tool integration, and performance in local languages.

• First, select the category and task that are most relevant to the business.
• Compare the results of the same version with the same reasoning budget.
• Retain a set of internal real samples for the final testing.
• It also records quality, tokens, latency, and failure rate.
• Security and fact-checking evaluations are added for high-risk scenarios.
• Concurrent, throttling, and stability tests are carried out based on the actual traffic volume.

Safety considerations for evaluation

Programming and agent tasks execute code or commands generated by models, and these must be run within isolated containers. Production keys, host directories, and internal network credentials should not be exposed to untrusted code.

• Use disposable containers, minimum permissions, and network isolation.
• Limit CPU, memory, disk, number of processes, and runtime.
• Do not mount Docker control interfaces and sensitive host directories on the container.
• Use short-term API keys and set budget and rate limits.
• Record model versions, parameters, images, and code commit versions.
• Destroy the container and revoke the temporary credentials once the task is completed.

Product advantages

• The question bank is continuously updated to reduce the risk of long-term contamination caused by a static set of questions.
• Use verifiable standard answers to reduce biases in LLM evaluations.
• It covers skills in mathematics, programming, reasoning, language, and data analysis.
• Include evaluation of agent programming in a real warehouse environment.
• Provides versioned rankings, papers, data, and evaluation code.
• Task-level results facilitate the analysis of a model’s specific strengths and weaknesses.
• It is capable of evaluating commercial APIs as well as self-developed compatible interface models.
• Papers and open-source communities provide a foundation for method validation.

Usage restrictions and precautions

• Pollution can only be restricted; it is impossible to prove that it does not exist at all.
• Scores from different release dates cannot be directly considered on the same scale.
• The overall ranking masks differences at the category and task levels.
• Public rankings may not cover all new models and inference settings.
• API timeouts, as well as price and rate limits, can affect the model’s score.
• Evaluation primarily relies on objective tasks, making it difficult to take into account aesthetic considerations and open-ended writing.
• Chinese language, industry knowledge, and security and privacy capabilities are not among its main focuses.
• The results of agent programming are influenced by the framework, the number of steps, and the sandbox environment.
• The direct inference paths for local models are currently limited; the official recommendation is to use compatible services.
• High scores cannot replace actual business verification and risk assessment.

GitHub and the open-source status

The official LiveBench repository is maintained by the LiveBench organization; it has around 1,300 stars at the time of verification. The repository includes testing scripts, task processors, tools for displaying results, and a record of changes, and it accepts issues as well as code contributions.

Its LICENSE file combines license texts from sources such as FastChat and LiveCodeBench, including the Apache-2.0 and MIT terms. When using, modifying, or redistributing it, it is necessary to comply with each of the requirements specified by the respective source files and third-party statements; the entire repository cannot be reduced to a single license.

Basic information

Recommendation score

4.7 / 5. LiveBench addresses the issues of contamination and bias in traditional LLM ranking systems by using dynamic questions and objective scoring methods; it is suitable for research and technology selection. However, aspects such as version comparability, task coverage, and operational costs still require careful consideration.

Frequently Asked Questions

Is LiveBench free?

Rankings, papers, code, and public data are available for free access. Conducting your own evaluations incurs costs related to model APIs, GPUs, computing resources, and maintenance efforts.

Is LiveBench really free from pollution?

It cannot be guaranteed that there will be absolutely no contamination. It reduces the likelihood of contamination through new questions, recent materials, and continuous updates; to be more precise, contamination is kept to a minimum.

Why not use large models as referees?

The project gives priority to standard answers and definite scoring methods, in order to avoid biases related to the judging model, sequence effects, and personal preferences; as a result, less emphasis is placed on open-ended assessments of quality.

Can scores from different versions be compared?

Direct comparison is not recommended. The questions, tasks, and difficulty levels may vary; it is necessary to indicate the date of publication, the model parameters, and the code version at the same time.

Can I test my own model?

Yes, that’s possible. The most reliable approach is to deploy the model as an interface that is compatible with certain standards, and then use the official scripts to generate responses and scores.

Why isn’t a certain latest model on the ranking list?

Official evaluations require API availability, stability, as well as budget and time; models that are too expensive or experience frequent timeouts may not allow for fair testing to be carried out.

Can LiveBench directly determine which model to purchase?

No. It is suitable for preliminary screening only; ultimately, real business data must be used to assess quality, latency, costs, stability, and compliance.

Is LiveBench open source?

Yes, but the repository contains code from multiple sources as well as text containing composite licenses. When redistributing it, it is necessary to check the Apache-2.0, MIT, and other relevant licenses.