Codeball

What is Codeball?

Codeball is an automated GitHub Pull Request review tool developed by the Sturdy team; it was created in 2020 and selected for Y Combinator’s winter class of 2021. It uses machine learning models to assign a score ranging from 0 to 1 to code contributions, and based on certain thresholds it approves, marks, or rejects those PRs.

Its focus is not on explaining each line of code one by one like a chatbot, but rather on determining whether a particular change is secure enough to be approved quickly through manual review. The team can also mark changes with moderate to low confidence as requiring review or careful examination.

Current operating status

As of August 21, 2026, Codeball is no longer able to provide its regular AI code review services. The Y Combinator website lists it as Inactive, and the original product’s website has been transformed into a site named AI BallCode, which features articles related to gambling.

The domain name of the remote API to which the open-source Action connects by default cannot be resolved at the moment; as a result, new workflows are unable to create proper Codeball review tasks. GitHub Marketplace still displays the v2.6.0 version and the “Verified Publisher” status, but this only indicates that the Action was once released by an official partner organization, and it does not prove that the model service is still operational.

A one-sentence summary

Codeball used GitHub Actions to analyze Pull Requests and automatically approve security changes with a high degree of confidence; however, its company, its official website, and the backend for its remote models are no longer in a functional state, and they can only be used for historical research and cleanup purposes.

Core historical functions

Pull Request risk scoring

Codeball assigns a score of 0 to 1 to each PR; the higher the score, the more likely the model is to consider that contribution acceptable. The default approval threshold is 0.935, and scores below 0.300 indicate that careful review is required.

The threshold can be modified within the workflow, but lowering it increases the risk of misclassification. Scores merely reflect the statistical judgments of the old model; they cannot replace testing, security scans, business validation, or the approval of the code owner.

Automatically approve secure PRs

When the model’s score reaches the approval threshold, Action can submit it for an approving review. This functionality requires that the repository or organization allow GitHub Actions to create and approve Pull Requests.

• Automatically approve contributions deemed safe by the model.
• Mark PRs with high confidence as codeball:approved.
• Remove the old risk labels that conflict with the new assessment.
• Support is available for using dedicated robot accounts to fulfill the CODEOWNERS process.
• Automatic approval can be turned off, leaving only the tag observation results.

Risk label

Action can assign three types of tags to PRs based on their scores, allowing maintainers to address high-risk changes first. The tagging feature is more suitable for trial runs than automatic approval, as it does not directly change the review status required for a merge.

• codeball:approved indicates that the model has reached a high confidence threshold.
• codeball:needs-review indicates that a regular manual inspection is required.
• codeball:needs-careful-review indicates that a thorough review is required.
• Each time it is run, old tags that do not match the current score are removed.
• The team can enable or disable different tags separately.

CI blockade

The failJobsWhenReviewNeeded option allows the workflow to be set to a failed state when the model’s contributions have not been approved. The default value is false, which means that Codeball does not prevent the CI process merely because the model receives a low score.

Setting the model results to require mandatory merging checks will amplify the impact of service failures. Since the remote API is no longer usable, maintaining such mandatory checks could prevent all new PRs from being approved.

Path filtering and draft skipping

GitHub Actions’ built-in path conditions allow Codeball to analyze only a specific service or directory, and workflow conditions can be used to skip drafts of PRs. This helps to prevent unnecessary changes from triggering reviews.

Suggestions for converting comments into code

codeSuggestionsFromComments was once used as a testing feature to convert PR comments into code suggestions. This option is disabled by default, and the existing documentation does not indicate that this testing capability has reached a stable state.

Codeball AI Writer app

Codeball AI Writer is a GitHub App that enables PRs originating from Fork to also receive approval and tagging permissions. It can provide write access to both public and private repositories, but it relies on the Codeball remote service as well.

Installing the GitHub App grants permissions to the repository; one should not continue to trust it just because the Marketplace page is still available. Existing users should check whether the app is still installed and revoke any unnecessary access rights after discontinuing use of the service.

The historical working principle of Codeball

1. GitHub triggers workflows when a new PR is created or an existing PR is updated.
2. Action reads the Pull Request address and GitHub token.
3. The client sends the PR address, access token, and threshold to the Codeball remote API.
4. The server uses a token to retrieve the relevant contribution information and create asynchronous review tasks.
5. The Action task polls for its status to obtain the confidence level, approval result, and suggested status.
6. Add or remove tags based on the configuration, and decide whether to submit it for approval review.
7. If forced failure is enabled, an unapproved result will cause the workflow to return a non-zero status.

Old documents state that the model takes into account hundreds of factors such as the perceptual hash of code differences, the author’s recent experience with relevant files, the frequency of file changes, as well as historical rollbacks or fixes. The backend of this model is not open-source in the Action repository, and it is therefore not possible to reproduce the original scoring at present.

Historical installation tutorial

1. Create a GitHub Actions workflow directory in the warehouse.
2. New workflow configuration files for Codeball have been added.
3. Set the trigger event to Pull Request, and add comment events as needed.
4. In the assignment, use codeball-action v2 released by sturdy-dev.
5. First, turn off automatic approval and CI failures, and enable tags only for monitoring purposes.
6. Set permissions for reading content, creating Issues, and creating Pull Requests for the workflow.
7. Submit to the testing branch to view the tag results for PRs with different risks.
8. After verifying the accuracy rate and the cost of misclassification, decide whether to expand automation.

The above describes the traditional method of configuration; it is not recommended to introduce this workflow at present. The remote API cannot parse such data, and the original website has also changed. Installing a new version will not only prevent the completion of the review process but will also introduce outdated dependencies as well as an unnecessary design for token transmission.

Tutorials on safe deactivation and migration

1. Search for references to sturdy-dev/codeball-action in the repository workflow files.
2. Check whether the branch protection rules set Codeball as a required status check.
3. First, remove the necessary checks to prevent merging from being blocked after the workflow is deleted.
4. Delete or disable the Codeball workflow, and submit a separate cleanup PR.
5. Uninstall the Codeball AI Writer application in the warehouse and organization settings.
6. Delete the dedicated CODEBALL_BOT_TOKEN and other related Secrets.
7. Revoke the warehouse permissions and Personal Access Tokens associated with the dedicated robot account that are no longer needed.
8. Check the recent workflow logs to see if any abnormal responses or sensitive information were printed.
9. Select the alternative review process that is still under maintenance, and first test it in comment-only mode.

Historical applicable users

Codeball is not suitable for new users at the moment. The following types of users are only appropriate if they want to understand its former product positioning, or if they need to migrate existing repositories and ensure their security.

• Open-source project maintainer: Once hoped to make rapid progress through low-risk community contributions.
• High-frequency PR team: Uses tags to focus manual effort on suspicious changes.
• Platform engineering team: Add model evaluation to the GitHub Actions pipeline.
• Multi-service warehouse: Reviews only the changes in the specified directories based on the path.
• Security and DevOps personnel: Investigate old workflows, application permissions, and tokens.
• AI code review researchers: Analyze early PR scoring methods based on statistical features.

Historical use cases

• Automatically approve documents, dependency updates, or other low-risk PRs.
• Add a label indicating that complex changes require careful review.
• Converting a model’s low score to CI failed, preventing unreviewed merges.
• Skip the draft PR and run it once the author is ready to finish it.
• The model is triggered only when the specified service directory changes.
• Use a robot account to provide approval authority for contributions to Fork.
• Model scores and manual review results are collected to assess the value of automation.

Historical advantages

• It is possible to integrate with existing PR workflows through a single GitHub Actions configuration.
• The 0 to 1 score system and the three-level labels facilitate a gradual implementation.
• The three strategies – automatic approval, tag-only, and CI failure – can be combined independently.
• It supports path filtering, skipping drafts, and customizing robot identities.
• The source code of the Action client is available publicly, allowing for the review of key inputs and permissions.
• The default threshold is set to favor high precision, with the goal of reducing incorrect automatic approvals.

These advantages depend on the proper functioning of remote scoring services. The open-source nature of the client side cannot compensate for the unavailability of models, training data, the backend infrastructure, or operational systems; therefore, past design advantages should not be considered as an equivalent to current purchasing capabilities.

Usage restrictions and risks

• The company has been marked as Inactive, as there is no evidence of ongoing maintenance or customer support.
• The original website has been turned into a site with other types of content, and it now features topics related to gambling.
• By default, the remote API domain name cannot be resolved, so the Action is unable to create tasks properly.
• The latest release for the Action repository dates back to April 2023, and its dependencies as well as runtime environment are outdated.
• The presence of a GitHub Marketplace entry does not mean that the service is online.
• Action will send the PR address and GitHub access token to the remote API.
• The original privacy policy and terms of service cannot be viewed on the product’s website at the moment.
• The models and the backend for training are not open source, which makes it impossible to rebuild the scoring service locally.
• Automatically approving faulty PRs can introduce defects, security issues, or malicious changes into the main branch.
• The forced failure configuration converts failures in external services into combined blockages.
• Dedicated robots and GitHub Apps expand writing permissions, which need to be revoked promptly.
• The accuracy and training scale in old promotions cannot be considered as guarantees of current performance.
• The open-source warehouse code uses the older Node Action runtime, which lacks long-term compatibility.
• Do not log in to or install any files via the original website that has been repurposed.

Price and availability

Historical release information for Codeball once offered a free trial, and some third-party directories have recorded information regarding payment details; however, there is currently no official page that shows the pricing, billing information, or subscription options. Any old prices should not be used as references for the current packages.

The fact that open-source actions can be downloaded for free does not mean that the Codeball service is free, nor does it imply that one can run a complete product on their own. In the absence of backend models and API implementations, the client side can only serve as an example of historical code and workflows.

Platform support and integration

Permissions, Privacy, and Security

The permissions recommended for GitHub Actions in historical contexts include the ability to read the contents of a repository, create Issues, and create Pull Requests. If automatic approval is required, it is also necessary to allow GitHub Actions to create and approve PRs; when using dedicated bots, a Personal Access Token with write permissions must be saved.

The open-source client shows that when a task is created, the PR or comment address, the GitHub access token, and the approval threshold are sent to the remote API. Due to the malfunctioning of this API and the unavailability of the current privacy policy, it can no longer be assumed that the previous commitments regarding data access, retention, and deletion still hold valid.

• Adjust the necessary checks for branch protection before removing the Codeball workflow.
• Remove CODEBALL_BOT_TOKEN from the repository and organization.
• Revoke the old Personal Access Token for the dedicated robot account.
• Uninstall Codeball AI Writer and check the scope of its repository permissions.
• Review the workflow history logs to ensure that no tokens or sensitive responses have been leaked.
• Do not rely on remote AI approval as the sole criterion for code quality.
• The alternative tool should first operate in a mode that allows only commenting or only tagging.
• Private repositories are required to have verifiable policies for data processing, deletion, and model training.

API, GitHub, and open-source status

Codeball does indeed have an official GitHub repository named sturdy-dev/codeball-action, which contains the GitHub Action client, various sub-actions, as well as tools for testing and configuration. The license applied to this repository is Apache 2.0; as of the time of verification, it had around 324 stars and 35 forks.

However, the repository only contains the workflow client and the logic for interacting with GitHub; the scoring model, the training pipeline, and the backend cloud API are not made available to the public. The product cannot be classified as fully open source – it can only be described as having an open-source action client, while the models and service backend are not open source and are no longer available.

Basic information

Recommendation score

The recommendation score is 1.3 out of 5. Codeball’s scores, tags, automatic approval mechanisms, and CI blocking features remain useful for understanding the early stages of AI code review processes; the Action source code also helps in gaining insight into GitHub permissions and remote calls.

However, the company is no longer active; the original website has been repurposed, the default API domain name cannot be resolved, and the repository has not been maintained in any significant way for many years. It is not suitable for new deployments, nor should access to its private code or write permissions for that repository be retained.

Frequently Asked Questions

Can Codeball still be used?

It cannot be considered a usable service. The Action entry is still available on GitHub Marketplace, but the remote API domain required to create tasks cannot be resolved at the moment.

Why isn’t the Codeball official website a code review tool?

The original domain name now displays content from AI BallCode, along with articles related to gambling, which indicates that the content and control status of the domain have changed. Do not use this website for logging in, downloading anything, or granting access to any repositories.

Is Codeball free?

Currently, there are no verifiable official prices or purchase options available. The source code for Action can be obtained freely under the Apache 2.0 license, but the remote model backend is not available; therefore, it cannot be considered that any free cloud services are available.

Will it automatically approve the PR?

Historically, it was possible to submit requests for approval review when the value exceeded the default threshold of 0.935. Currently, the remote scoring service is not available, so there should be no further reliance on this automatic approval process.

What permissions are required for Action?

Historical configurations require reading the contents of the repository and writing them to Issues and Pull Requests. Automatic approval also demands that the repository allow GitHub Actions to create and approve PRs.

Why delete old tokens?

The client sends the GitHub access token to the remote Codeball API; the dedicated robot solution may also store long-term tokens. Once the service is discontinued, there is no longer any need to retain these credentials.

Is Codeball open source?

Only the GitHub Action client is open source, licensed under the Apache 2.0 license. The scoring models, training pipelines, and backend services are not open source; therefore, the complete product is not an open-source solution.

Can Codeball be deployed on my own?

It is not possible to deploy a complete service solely based on public warehouses; these warehouses do not include the original scoring models or the API backend, and the client still needs a remote task interface.

How to remove old workflows?

First, remove the mandatory checks associated with Codeball from the branch protection settings; then delete the workflows, applications, and Secrets. Finally, revoke the robot tokens and verify that the new PR no longer waits for the old status.

What does the Verified status on GitHub mean?

It indicates that GitHub manually verified that the creator of the Action is an official partner organization; this does not mean that GitHub endorses the AI-generated results, nor does it guarantee that the corresponding remote service is still operational.

Can the Codeball source code be modified to create a new tool?

It is possible to study or modify the client code under the Apache 2.0 license, but it is necessary to develop new scoring backends, implement permission systems, and establish data policies on one’s own. It is no longer possible to assume that the original API protocols or models will remain usable.

Summary

Codeball once provided a clear approach for AI-driven code review: it calculated a confidence level for each pull request, and then used labels, approvals, or CI status to integrate the model’s judgments into the team’s workflow. The public Action version still retains these examples of client design and configuration.

What is truly important at the moment is to ensure safe handling after discontinuation of use, rather than continuing with installations. The team should remove any necessary checks, workflows, GitHub Apps, and old tokens, and switch to a review process that is still under maintenance, has clear data policies, and can be tested in comment-only mode first.