DeepInfra

What is DeepInfra?

DeepInfra is an AI inference cloud operated by the American company Deep Infra Inc., designed for developers and enterprises that need to integrate language, image, audio, video, and traditional machine learning models into their products.

The platform offers shared Serverless inference capabilities, batch processing, deployment of private models, GPU instances, code sandboxes, and managed agents. It addresses issues such as deploying models, enabling elastic scaling, ensuring interface compatibility, and controlling inference costs, rather than simply providing a chat interface.

Key capabilities

• Shared model inference: Select from the model catalog languages, embedding, rearrangement, image, video, audio, or classification models; billing is based on the actual number of tokens or executions, with no need to manage GPUs manually.
• OpenAI-compatible interfaces: For chatting, text completion, embedding, and image generation, the OpenAI SDK can be reused – all that is needed is to change the base address, model name, and key.
• Native multimodal interface: It covers tasks such as image generation, speech recognition, object detection, image classification, zero-shot classification, entity recognition, and mask filling – tasks that are not addressed by the existing compatible interfaces.
• Batch processing: A large number of non-real-time requests are submitted asynchronously in the form of files and batches; this approach is suitable for offline evaluation, content processing, and data annotation. The cost of annotating public documents is 20% lower compared to real-time inference.
• Private models: Deploy Hugging Face models, custom weights, language LoRA, or image LoRA on dedicated GPUs, offering isolated endpoints, automatic scaling, and more predictable latency.
• GPU instances: Rent B200 containers to run training, fine-tuning, inference, or custom workloads via SSH; billing is based on the duration of usage.
• Sandboxes: Isolated Linux microVMs can be started as needed, allowing agents to execute code, read and write files in the workspace, and access the public internet; no computing fees are charged once they are stopped.

Supported model tasks

Differences between the two sets of APIs

All interfaces use Bearer Tokens for authentication. The parameters of different models, the length of the context, stream-based output, tool calls, structured output, and multi-modal support vary from one model to another; it is not possible to assume that the behavior will remain identical just by replacing the model name.

Fast access process

1. Register an account, create API Tokens in the console, and use different keys for development, testing, and production environments.
2. Select a model from the model directory, and check the license, context, input type, output format, unit price, rate limits, and data processing exceptions.
3. The compatible interface allows continued use of the OpenAI SDK; for special tasks, the native interface is used, with requests being constructed in accordance with the model’s schema.
4. First, test for timeout, streaming response, tool calls, error retry, and output quality using a low data volume; thereafter, set limits on concurrency, budget, and automatic top-up.
5. After going live, monitoring parameters such as usage volume and request costs, as well as token usage, cache hit rates, latency, and failure rates, helps to prevent unlimited retries.

Billing model for shared reasoning

Language models are typically charged based on every 1 million input and output tokens; some of these models offer a lower rate per unit of cached input. Models for images, audio, video, and other types of content may be billed according to the length of the image or audio, the number of seconds it takes to generate the content, or the time required to process it.

The above is a representative example of the real-time price page; it does not represent fixed packages or price guarantees. The model in use, its version, caching rules, and unit prices can all change. For an accurate estimate, it is necessary to consult the details of the selected model page as well as the account settlement information.

Free credit, top-ups, and refunds

• At present, there is no confirmed fixed free API package that remains valid for all accounts on a long-term basis, nor any monthly free quota; registration bonuses and promotional credits cannot be considered permanent benefits.
• Shared reasoning operates on a pay-as-you-go basis; users can add funds to their account and set up automatic top-ups. Insufficient balance may result in the suspension of requests, sandbox operations, or instance activities.
• For business services, the payment terms, support options, and other conditions can be specified through service orders; there is no unified pricing listed on the public page.
• The current terms do not provide a general refund guarantee applicable to all prepaid balances and amounts consumed; in the event of a service failure that makes it impossible to fulfill the obligations, the terms consider such refunds as a specific form of relief.

Batch processing and cost optimization

• Use the Batch API for tasks that do not require an immediate response; a 20% discount is available by following the instructions in the documentation, and this approach also reduces the load on real-time processing.
• When reusing long prompt prefixes, a prompt cache can be utilized to reduce pre-filling latency and the cost of cache reads; the available retention options include short-term and one-hour durations.
• Ensure that the response is generated only to the desired length, and set limits on the maximum output tokens, concurrent operations, and the number of retries.
• First, use small models for classification, routing, and extraction, and assign only the more complex requests to large models; this approach is usually more manageable than using high-cost models for all requests.

Private models and dedicated deployment

In a private deployment, charging continues even without any requests as long as the GPUs are running. The examples in the documentation show that failing to turn off a dual-GPU setup for 64 hours can result in costs of around 256 dollars; therefore, it is necessary to set budget limits and enable automatic scaling down.

Sandboxes Code Sandboxes

• Each Sandbox is an independent Linux microvirtual machine that can execute commands and Python scripts, transfer files, and access the public internet; however, it does not accept incoming connections, nor can it access other sandboxes or the internal network of DeepInfra.
• Charging is based on seconds with no minimum duration; fees are applied for the processes of creation, startup, operation, and shutdown; no charge is incurred when the system is in shutdown mode.
• Only the workspace directory is retained between normal shutdowns and restarts, with a maximum retention period of 7 days after shutdown; directories in a failed state are deleted within a few minutes, and there is no guarantee that the workspace can be restored.
• The default idle timeout is 1 hour; the system will stop automatically after 24 hours of continuous operation. A single command can run for up to 30 minutes, and each account can have a maximum of 5 sandboxes that are not in a stopped state.
• The maximum size for a single read or write operation on a file is 100 MiB; important results should be copied to external storage promptly.

Data privacy and the zero-retention principle

• The inputs for ordinary reasoning exist only in memory during processing and are not saved to disk; once the outputs are returned, they are deleted from memory, and the platform states that these data are not used to train models.
• The image outputs are stored temporarily for easy access; batch processing may encrypt the data and save it on disk, deleting it shortly after the results are returned.
• When invoking Google or Anthropic models, the data is sent to the corresponding model endpoint, and the storage of outputs, as well as the rules for logging and training, are governed by the policies of the respective providers.
• Platforms usually do not record the content of requests, but they do keep metadata such as request IDs, costs, and sampling parameters, and retain the ability to log a small number of requests when needed for debugging or security purposes.
• Account, communication, device, usage, network address, and billing information are handled in accordance with the privacy policy; once an account is deleted, personal information is usually removed after 30 days, with exceptions possible for legal, tax, or security records.

Safety and compliance

DeepInfra states that its security measures comply with SOC 2 and ISO 27001, and it offers technical and organizational solutions in line with GDPR and HIPAA requirements. Certification and controls do not automatically ensure that a customer’s applications are compliant; before handling health, financial, or other sensitive data, it is still necessary to verify the contract terms, DPA agreements, the selected models, as well as the regional requirements.

• API Tokens should be stored only on the server side or in a key management system, organized by environment, and rotated regularly.
• When granting restricted access to end users, JWTs with defined scopes and expiration times can be used, thereby avoiding the issuance of main account tokens.
• Before using third-party models, check where the data will be sent; do not apply the rule of no data retention at all to the explicitly listed exceptions.
• Set up access controls and deletion procedures for logs, callbacks, and batch files, in order to prevent sensitive user data from being stored in one’s own observability system.

SDK, integration, and open-source status

• The official TypeScript client, deepinfra-node, is licensed under the MIT license and provides native interfaces for text, embedding, images, speech, and classification.
• The Python deepinfra package covers inference and Sandboxes; the README of the current repository describes it as the official SDK, which is licensed under the MIT license.
• The document also provides information on how to integrate LangChain, LlamaIndex, Vercel AI SDK, AutoGen, Anthropic SDK, and various agent frameworks.
• What is open source are the client and some of the integration code; this does not mean that the DeepInfra cloud platform, the scheduling system, or the hosted models are also open source. Each third-party model remains subject to its own license and usage policies.

Suitability for users and restrictions

• Application developers: They can quickly integrate multiple models while maintaining compatible interfaces, but they need to conduct tests and implement fallback solutions due to the differences between these models.
• RAG and the search team: combine embedding, rearrangement, and generation models, and optimize costs through batch processing and caching.
• Teams with dedicated weights: They can use private GPUs to achieve isolation and stable latency, but they also have to bear the costs associated with ongoing operation.
• Agent developers: Use Sandbox to execute untrusted code, but they must take into account the short lifespan of such code and the risk of data loss when designing their solutions.
• Projects that require absolutely fixed model behavior, permanently free quotas, or complete absence of third-party processing should not rely solely on the default settings of shared directories.

Frequently Asked Questions

Does DeepInfra offer a permanently free API?

At present, there is no permanent free plan or fixed monthly quota available for all accounts. The promotional balance can vary depending on promotions and accounts, and production costs should be estimated based on the current unit prices and actual invoices.

Can I use the OpenAI SDK directly?

Yes, the OpenAI SDK can be reused for functions such as chatting, text completion, embedding, and handling images. It is still necessary to change the base address, the DeepInfra Token, and the model name, in addition to checking parameter compatibility.

Does DeepInfra save the prompts and use them for training?

Ordinary inference inputs are not saved to disk, and the outputs are deleted after being returned; the platform states that they are not used for training. Exceptions apply to images, batch processing, as well as Google and Anthropic models, and a small number of requests may be logged in debugging or security scenarios.

How to choose between shared inference and private deployment?

In the event of traffic fluctuations or when the model is still under evaluation, inference should be shared first; private deployment should be chosen when self-weighting, isolation, stable latency, or dedicated capacity are required, with ongoing monitoring of GPU usage costs.

Is DeepInfra an open-source platform?

The cloud platform itself is not a fully open-source project that is publicly available. The official Python and TypeScript clients, as well as certain integration components, are governed by open-source licenses, while the hosted models follow their own respective licenses.

Why is batch processing cheaper?

Batch processing enables the platform to schedule non-urgent requests asynchronously; according to the official documentation, its performance is 20% lower than that of real-time inference. It is not suitable for providing interactive responses, and it may encrypt inputs and outputs on a short-term basis.

Summary

The advantage of DeepInfra is that it allows a single account to cover compatible interfaces, native multimodal models, batch processing, proprietary weights, GPUs, and code sandboxes; it also enables billing based on tokens, execution volume, or GPU hours, depending on the workload. Before going live, it is essential to verify the price of each model and its licensing terms, set budget limits and retry thresholds, and properly handle exceptions related to no data retention.