Higress

A one-sentence summary

Higress is an open-source, AI-native API gateway developed on top of Istio and Envoy; it enables unified access to large-model APIs, AI Agents, MCP Servers, and Kubernetes inference services, and it handles routing, protocol conversion, quota management, security measures, caching, and monitoring functions at the entry layer.

Tool Introduction

Higress was originally developed within Alibaba to address issues such as the impact of gateway overload on long-term connections, as well as insufficient load balancing in gRPC and Dubbo. It was later placed in the CNCF Sandbox for open-source governance. It can function as an AI gateway, as well as a Kubernetes Ingress gateway, a microservices gateway, and a security protection gateway.

As of August 23, 2026, the latest stable version of the main warehouse is v2.2.4, and its source code is licensed under the Apache-2.0 license. This product is intended for use by platform engineering teams, those involved in cloud-native operations, AI infrastructure specialists, and development teams; it does not provide chat or content generation interfaces for ordinary users.

Core positioning

Main functions

Unified proxy for over 100 models

Higress enables the consolidation of authentication methods, interface structures, and model names from different model providers into a single entry point; the product page indicates support for over 100 mainstream models. This approach helps reduce the need for vendor-specific code, but it is still necessary to address the differences among these models in terms of tool invocation, multi-modal functionality, context handling, and error formatting.

Multi-model routing and fault degradation

A team can configure multiple models or authentication tokens on the same route, and improve availability through polling, load balancing, health checks, and fallback mechanisms. Downgrading can only reduce the risk of single-point failures; it does not guarantee that the output quality, context length, and compliance policies of different models will be identical.

Adaptive model load balancing

Version 2.2.4 introduces the AdaptiveScore mode, which is based on real-time load; it allows Redis to detect the level of concurrency at the cluster level while still enabling local fallback strategies. This feature is suitable for clusters that handle high levels of concurrent processing, but the availability of Redis, metric latency, and policy parameters also influence the scheduling results.

API Key pool and token downgrade

The same model provider can configure multiple API Keys, and set exception thresholds and health recovery checks for each token. Key pools help with capacity management and failover, but they cannot be used to bypass the account restrictions imposed by the model provider or to violate the service terms.

Token throttling and quotas

Higress enables throttling not only based on QPS, but also using input and output tokens, the number of requests, concurrency levels, and consumer settings. Version 2.2.4 supports the simultaneous application of multiple matching rules; when upgrading, it is necessary to examine the logic used in the older version to avoid sudden changes in quotas.

Precise caching and semantic caching

Precise caching reuses the exact same results for requests, while semantic caching returns existing responses based on similarity, which helps to reduce the number of repeated model calls, as well as associated Token costs and latency. Caches can store sensitive information and answers; they may also return incorrect responses to questions that appear similar but are actually different. Therefore, it is necessary to implement tenant isolation, set expiration times, and define exclusion rules.

AI observability

Monitoring allows for the tracking of various metrics such as the number of tokens input and output per second, the usage of resources by suppliers and models, latency, as well as the number of successful and failed calls. Version 2.2.4 adds a count of failed LLM calls, filling in the gap in monitoring related to error responses in stream-based and non-stream-based scenarios when there is no token usage.

Prompt and request response processing

Plugins can modify prompts, apply templates, save historical conversations, identify intents, enforce JSON output, and transform requests or responses. Changes made at the entry layer affect all downstream applications; in a production environment, it is necessary to test template versions, field compatibility, and rollback procedures.

RAG and search enhancement

Higress can connect to knowledge bases, vector retrieval, and search services through plugins, and incorporate the results of those searches into model requests. The gateway is responsible for coordinating these processes, but it does not guarantee the quality of the documents, the accuracy of the retrieved results, the correctness of the references, or compliance with intellectual property rules.

AI data masking and content security

The data masking plugin can process sensitive fields before requests reach external models, while the content security plugin is used to check inputs and outputs. Some plugins rely on external security services or custom-built models; version 2.2.4 also includes integration with Qwen3Guard. The actual level of coverage depends on the rules, models, and deployment in use.

Stream-based request and response processing

The data plane of Envoy and Wasm extensions enable full streaming processing of request and response bodies, as well as support for protocols such as SSE. Streaming connections impose higher requirements in terms of timeout handling, connection cleanup, token tracking, and plugin memory management; therefore, they cannot be tested using ordinary short-lived connections.

Unified governance of MCP

Native MCP proxy

Higress can proxy the disparate MCP Servers to a unified entry point, enabling centralized handling of authentication, fine-grained rate limiting, tool discovery, call auditing, and monitoring. Clients only need to connect to the gateway; however, incorrect configuration of the gateway’s permissions can expose additional tools to users who should not have access to them.

Convert existing APIs to MCP

Through the MCP Bridge and OpenAPI conversion tools, existing REST or web services can be exposed as remote MCP tools without having to rewrite the business logic. After automatic conversion, manual review is still required for the tool names, parameter formats, authentication mechanisms, error handling, and any potentially risky operations.

Protocol version compatibility

Version 2.2.4 adds support for the MCP 2026-07-28 standard, including stateless request boundaries, deterministic tool discovery, typed parameter validation, Origin protection, and isolation between old and new protocols. Before upgrading, it is necessary to conduct end-to-end compatibility testing of the protocol versions on both the client and server sides.

MCP operation and maintenance interface

The Higress Ops MCP Server can be used to configure and manage gateways; in the new version, Basic authentication is required for sensitive debugging tasks as well as for managing Envoy. The operation-and-maintenance type MCP possesses high levels of permission, and it should be separated from regular business tools, using separate credentials, network boundaries, and audit mechanisms.

Kubernetes inference and Gateway API

Higress supports both the Ingress API and the Gateway API; it was upgraded to Gateway API v1.6.0 in version 2.2.4, and at the same time received the Inference Extension v1.4 runtime capabilities. It can direct Gateway routing resources to inference pools, and make selections and aggregations based on the status of the inference endpoints.

Microservices and traditional API governance

• Service discovery: It can connect to registration centers such as Nacos, ZooKeeper, Consul, and Eureka.
• Protocols: It supports plain-text as well as encrypted transmission over web pages, and gRPC; it can also be used in conjunction with MCP Bridge to handle services such as Dubbo.
• Routing: Offers features such as rewriting, gradual deployment, traffic splitting, load balancing, and fault handling.
• Authentication: Offers various strategies such as Key Auth, JWT, Basic, HMAC, OIDC, OAuth2, and external authentication.
• Protection: CC protection at the WAF, IP, or Cookie level, along with local or cluster-based rate limiting, can be utilized.
• Migration: It is compatible with a large number of Nginx Ingress annotations, making it suitable for phased migration rather than a one-time replacement.
• Console: Provides a graphical interface for management, while still allowing access to Kubernetes resources and automated configuration.

Wasm plugin ecosystem

Higress relies on WebAssembly as its primary mechanism for extension; plugins can be developed using languages such as Go, Rust, and JavaScript, and sandbox isolation is used to reduce memory security risks. Plugin versions can be upgraded independently and updated in real time, without the need to restart the entire gateway.

Deployment method

Docker port

Getting started with Docker quickly

1. Prepare a test host capable of supporting containers as well as a separate working directory, and ensure that ports 8001, 8080, and 8443 are not in use.
2. Choose an Higress mirror repository that is located near the deployment area; in a production environment, use a specific version tag instead of sticking to the \"latest\" version.
3. Mount the persistent directory and start the All-in-One container; the console ports are accessible only from trusted networks.
4. As soon as you access the console for the first time, configure administrator authentication, access boundaries, and backup policies.
5. Add a test model provider and a API key with minimal permissions; do not import all production keys at once.
6. Create AI routes, and set up model mapping, consumer authentication, token throttling, and fault degradation.
7. Use small-scale, non-sensitive requests to test normal and streaming responses, errors, timeouts, and Fallback.
8. Check the Token, latency, error, and consumer metrics, then gradually add caching, security, and MCP plugins.
9. Only after verifying configuration persistence and restoration should one consider migrating to Helm or a managed production version.

Kubernetes deployment process

1. Check the version compatibility matrix for Kubernetes, Helm, Gateway API, and Higress, and back up the existing Ingress resources.
2. Create a separate namespace, service account, network policies, and a controlled image repository for Higress.
3. Install or upgrade the required CRDs, and check for warnings regarding version mismatches when the controller starts up.
4. Use Helm to install a fixed version, set up image repositories by region, and restrict the exposure of the control plane and console.
5. First, migrate a domain with low risk, and check the compatibility of the Nginx Ingress annotations one by one.
6. Configure Gateway routing, backend services, certificates, and authentication plugins, and conduct rollback drills.
7. Token quotas, timeouts, retries, circuit breaking, fallback options, and auditing are added for large model and MCP traffic.
8. Integrate with Prometheus, logging, and alerting to monitor configuration propagation, data plane resources, and long connections.
9. After completing capacity, failure, security, and upgrade tests, scale it up to production traffic.

AI gateway configuration process

1. List the model providers, self-built inference services, Agents, and MCP Servers that need to be integrated, and indicate the level of data sensitivity for each.
2. Create separate keys, consumer identities, and routing for each environment to prevent the use of the same resources for testing and production.
3. Standardize the external invocation protocol, while retaining compatibility checks and fallback mechanisms for model-specific parameters.
4. Set the QPS, concurrency, token limit, daily budget, and allowed models for each consumer.
5. Configure the preferred model, backup model, health thresholds, timeout, and maximum number of retries.
6. Caching is enabled only for requests that can be safely reused, with users, tenants, and permissions included in the cache keys.
7. Enable minimization, data masking, and prompt protection at the input side, and conduct checks for content and data leaks at the output side.
8. Logs the model, token, latency, failures, fallbacks, cache hits, and MCP tool calls.
9. Use fault injection to verify the behavior of the model in cases of unavailability, Redis interruptions, plugin errors, and MCP service timeouts.

MCP online check

1. Only the approved OpenAPI operations are imported; unnecessary interfaces for deletion, transfer, execution, and administration are removed.
2. Define a clear name, description, typed parameters, required fields, and error structure for each tool.
3. Separate read-only tools from write tools, and require manual confirmation for high-risk operations.
4. Leverage consumer identities, minimum permissions, and tenant isolation to prevent models from sharing credentials with high privileges.
5. Verify Origin, authentication, rate limiting, parameter injection, timeouts, retries, and idempotency.
6. The logging tool captures the discovery details, parameters, user who initiated the operation, results, errors, and the audit ID for downstream processes.
7. Test both the old and new MCP clients simultaneously to avoid implicit incompatibilities resulting from protocol upgrades.
8. Place the Higress Ops management tool on a separate internal network and enable strong authentication.

Community version and Alibaba Cloud version

The community version of Higress is available freely and open-source; users are responsible for the costs related to servers, networking, storage, monitoring, upgrades, and maintenance. Alibaba Cloud AI Gateway builds on Higress to offer managed Serverless solutions as well as dedicated instance-based products, along with a cloud control panel, scalability, service level guarantees, and commercial support.

Free beta version limits

In the new version of Serverless available for beta testing, the free tier covers only the cost of the gateway itself; data transfer over the public network is still charged according to standard cloud data transmission rates. Model services, logging, monitoring, and other cloud services are billed separately. After the beta period ends, commercial pricing may come into effect, and the actual prices and any discounts will need to be confirmed again in the control panel.

Billing structure for the exclusive version

The dedicated instance version consists of a gateway instance, the amount of data processed for requests and responses, and public network traffic; the specific cost varies depending on specifications, location, network type, and purchase duration. There is no fixed price that applies to all deployment scenarios.

Open source, GitHub, and licenses

The main Higress repository is currently maintained by the higress-group organization; the previous Alibaba-related entry points now point to this new organization. The licensing for the core code is Apache-2.0. The main repository is under active maintenance, and version 2.2.4 was released on August 13, 2026.

Open source does not mean zero cost.

Apache-2.0 permits the use, modification, and distribution of the core code, but companies still need to comply with the license terms and the NOTICE requirements; they are also responsible for covering the costs related to servers, model invocation, external plugins, monitoring, security, and personnel maintenance. Alibaba Cloud’s commercial version cannot be considered free or allow for the replication of its services just because an open-source kernel is used.

Safety and privacy

By using a self-built Higress, prompts, responses, keys, and audit data can remain within the network and storage under the company’s control; however, requests may still be forwarded to external models, search engines, vector databases, or security services. The actual boundaries of the data are determined by routing mechanisms, plugins, and upstream suppliers.

Product advantages

• A single entry point handles LLM APIs, Agents, MCP, and Kubernetes inference workloads.
• Based on Istio and Envoy, it is suitable for long-lived connections, SSE, and high-bandwidth AI traffic.
• It supports over 100 mainstream models and offers a unified protocol, load balancing, and fallback options.
• Token throttling, quotas, caching, and observability are all designed with AI costs and stability in mind.
• The MCP proxy, the conversion of existing APIs to MCP format, and auditing functions help reduce the complexity associated with integrating tools from multiple teams.
• It is compatible with both Ingress API and Gateway API, allowing it to integrate with existing cloud-native entry systems.
• Wasm plugins can be developed in various languages, upgraded independently, and updated dynamically.
• It supports independent Docker deployment, Helm-based cloud-native deployment, and Alibaba Cloud-hosted versions.
• The main project is licensed under Apache-2.0 and is governed openly by the CNCF Sandbox community.

Usage restrictions and precautions

• Higress is an infrastructure; it is not a ready-to-use terminal chat or knowledge base product.
• Production deployment requires expertise in gateways, Kubernetes, networking, security, and observability.
• The community version does not come with any commercial SLAs; fault response, upgrades, and capacity management are handled by the team in charge of maintaining it.
• A unified interface cannot eliminate differences in model functionality, quality, context, and error format.
• Falling back to different models can change the responses, compliance outcomes, and costs.
• The semantic cache may return old answers that are not suitable for the current user or context.
• Token statistics depend on the supplier’s response and the way text is segmented, so they cannot be considered equivalent to invoices.
• The MCP tool connects the model to real-world business processes, and warnings about potential injection issues can turn into actual operational risks.
• Some plugins rely on Redis, vector databases, search functions, or external security services, which increases costs and the potential for failures.
• Custom Wasm plugins may still pose issues related to business logic, the supply chain, and resource consumption.
• When upgrading to v2.2.4, pay attention to changes in behaviors such as multi-rule throttling semantics.
• CNCF Sandbox indicates that it is in the early stage of project governance, and this does not equate to a mature Graduated level.
• The pricing and capabilities of the new version of Serverless in the beta phase may change once it becomes commercially available.
• The free scope of the cloud version does not include public network traffic, models, logs, monitoring, and related cloud services.

Basic information

Frequently Asked Questions

Is Higress free?

The community moderator code is licensed under Apache-2.0, allowing free implementation, but the servers, network infrastructure, models, storage, and maintenance services are not free. During the beta phase of Alibaba Cloud’s Serverless version, the gateway itself costs nothing, but other cloud services and data transfer volumes may still incur fees.

Is Higress a large model?

No. It is located between applications and models, Agents, MCP tools, and inference services; its role is to handle connectivity and management tasks, and it does not replace the upstream models to generate content on its own.

Which models are supported?

The current product page states that it supports over 100 popular models, and can act as a proxy for OpenAI’s compatible interfaces as well as those of various domestic and international suppliers. As for multi-modal capabilities, tool calls, and parameter compatibility, these need to be tested using the suppliers’ plugins.

Is it possible to do without Kubernetes?

Yes. Docker All-in-One, Docker Compose, and Standalone can be used to operate independently of Kubernetes; however, for handling large-scale operations, ensuring persistence, and achieving high availability, additional design efforts are required.

Is it possible to turn a regular API into an MCP tool?

Yes, it is possible to generate remote MCP configurations using the MCP Bridge and OpenAPI conversion tools. After automatic generation, manual review is required for dangerous operations, parameters, authentication details, and tool instructions.

Is Higress the same as Alibaba Cloud AI Gateway?

They are not exactly the same. Higress is a project from the open-source community, while Alibaba Cloud AI Gateway builds upon its capabilities to offer managed Serverless services, dedicated instances, cloud integration, SLAs, and commercial support.

Can the Standard version of Serverless be used in production?

It is more suitable for low-traffic scenarios, testing, and development; it has a default rate of 50 QPS and lacks a separate entry point or custom domain name. For actual public services, it is advisable to consider the enterprise version or the dedicated instance version.

Will Higress reduce the cost of the model?

Caching, quotas, routing, and fallback mechanisms can help reduce waste or enable the selection of more appropriate models, but they do not guarantee a reduction in overall costs. It is also necessary to take into account the costs associated with gateways, caching, monitoring, and maintenance.

Is Higress open source?

Yes, the main repository uses Apache-2.0. The licensing terms and conditions for the relevant consoles, Standalone versions, plugins, and commercial cloud services need to be checked separately.

Summary

Higress is suitable for development and platform teams that need to manage multiple models, agents, MCP tools, and cloud-native APIs through a unified entry point. It handles token management, fallback mechanisms, caching, content security, stream-based monitoring, and MCP auditing at the gateway layer, while preserving the performance of Envoy, the standards of Gateway API, and the capabilities for Wasm extensions.

When making a selection, it is necessary to distinguish between the open-source version from the Apache community and the version hosted by Alibaba Cloud; moreover, the costs associated with model invocation, traffic, logging, Redis, plugins, and maintenance should all be taken into account in the total cost. Before going live, it is important to verify aspects such as cache isolation, key protection, minimum permissions via MCP, compatibility during version upgrades, and failover mechanisms, rather than focusing only on setting up a model proxy.