assemblyai

What is AssemblyAI?

AssemblyAI is a voice AI platform designed for developers and enterprises, whose core functionality is to convert pre-recorded or real-time audio into structured text. Development teams can integrate features such as transcription, speaker recognition, speech understanding, and voice bots into their own products using APIs and official SDKs.

The platform is not a standard web tool for converting audio to text; rather, it is a developer infrastructure service that charges based on usage. Its offerings include asynchronous pre-recording transcription, real-time streaming transcription, low-latency synchronization interfaces for short audio clips, end-to-end Voice Agent APIs, Speech Understanding capabilities, and an LLM Gateway.

Main products and features

• Pre-recorded audio transcription: Submit audio or video files, and a complete transcription will be generated asynchronously.
• Real-time streaming transcription: Audio streams are sent via WebSocket, and text events are received within a few hundred milliseconds.
• Synchronous short audio transcription: Quickly processes short segments such as voice commands and transcriptions using a single request.
• Voice Agent API: Combines speech recognition, language models, speech synthesis, and dialogue orchestration within a real-time connection.
• Speaker separation: Identifying different speakers and adding labels to words or turns of speech.
• Keywords and general guidelines: Use terms or natural language to improve the recognition in a specific context.
• Speech understanding: Performs tasks such as entity recognition, topic identification, sentiment analysis, translation, and other post-processing operations.
• Privacy safeguards: Provide processing of sensitive text or audio, content filtering, and profanity removal.
• LLM Gateway: Utilizes unified credentials to call multiple language models for processing summarization and speech-related tasks.

Speech-to-text model

How to choose between pre-recorded, real-time, and synchronous interfaces

Speech comprehension ability

• Speaker recognition: Identifying different speakers in meetings, interviews, or calls.
• Entity detection: Extracts persons, organizations, locations, and other structured entities.
• Emotion analysis: Determining the positive, negative, or neutral tone expressed in a text segment.
• Topic detection: Identifies the theme of audio discussions by classifying them based on their content.
• Keywords and highlights: Identify repetitive or representative phrases.
• Generate the target language text while preserving the original transcription structure.
• Custom format: Adjust numbers, abbreviations, and professional expressions in accordance with business rules.
• Abstract and Chapters: More flexible post-processing results can be generated through the current language model workflow.

What use cases are suitable?

• Meeting assistant: Records multi-person meetings, identifies speakers, and generates action items.
• Call center: Transcribes customer service calls to analyze topics, emotions, and quality issues.
• Media platforms: Generate subtitles and indexes for podcasts, videos, live streams, and courses.
• Voice agent: It recognizes what the user says and enables real-time conversations as well as the invocation of various tools.
• Sales analysis: Organize call details, objections, competitive information, and next steps for follow-up.
• Medical voice: Handles specialized terminology within the applicable languages and in compliance with relevant regulations.
• Content review: Processing sensitive information, inappropriate content, and personal data within audio files.
• Voice search: Converts large amounts of audio and video content into searchable text.

Which development teams are suitable?

• SaaS teams that need to integrate a speech recognition API into their products.
• Engineers who develop real-time subtitles, voice robots, or telephone automation systems.
• Media and data teams that need to process large volumes of historical audio and video files in bulk.
• An analysis platform that requires a speaker, timestamps, and structured results.
• Developers who wish to achieve rapid integration using Python, JavaScript, or other SDKs.
• Organizations with corporate requirements regarding data areas, retention times, permissions, and auditing.

Quick Start Tutorial

1. Register an account and create an API key dedicated to the testing environment.
2. Compare models in the console Playground using real but anonymized audio.
3. Choose from pre-recorded, real-time, synchronized short audio clips, or the Voice Agent interface.
4. Install the official SDK for the target programming language and specify a compatible version.
5. Specify the speech model explicitly; do not rely on default values that may change.
6. Configure timeout, retry, callback verification, and failure logging.
7. Calculate the word error rate and the error rate in key fields by manually checking the text.
8. Set reminders for balance, usage, and abnormal charges before entering the production environment.

Real-time transcription tutorial

1. Confirm the sampling rate, encoding, and channels of the microphone or phone audio.
2. Establish a WebSocket connection and send the correct session parameters.
3. Continuously transmit audio blocks, and handle turn-taking, timestamps, and error events.
4. Configure keywords or prompts based on business terminology.
5. Enable speaker separation, medical mode, or Voice Focus as needed.
6. Distinguish between temporary states and the immutable final text in the interface.
7. The user sends a session termination message after ending the call.
8. Confirm that the connection is truly closed, and monitor the actual duration of session billing.

Base price

AssemblyAI operates on a pay-as-you-go basis; there is no mandatory monthly subscription, no minimum spending requirement, and no long-term contracts. New accounts receive a $50 credit free of charge, and no credit card is needed. This credit can be used for text-to-speech, Voice Agent, speech recognition, and safety features, but it does not include LLM Gateway.

Price of value-added features

Billing rules and common misconceptions

• Pre-recorded transcriptions are charged based on the actual number of seconds of audio submitted; no fee is applied for failed transcriptions.
• Flow transcription is charged based on the total time from activation to deactivation, with silent and idle periods also being taken into account.
• Flow sessions that are not properly closed may be automatically terminated after 3 hours, resulting in full charges.
• Multiple concurrent streams each accumulate their own billing duration.
• Multi-channel transcription is charged separately for each channel; one hour of three-channel service counts as three hours of usage.
• Value-added features will be added to the price of the basic model.
• The mute and recovery windows for Voice Agent connections may also be subject to charging.
• If the Voice Agent invokes the LLM Gateway separately, the associated token costs may be charged additionally.

Cost control tutorial

1. Select the model that meets the minimum requirements in terms of accuracy, latency, and functional capabilities.
2. Specify the model explicitly in the request to avoid differences between the default values for free and paid accounts.
3. Pre-process the long silences and determine whether multi-channel separate transcription is truly necessary.
4. Only enable value-added features that generate real business value.
5. A termination event is sent immediately upon the end of the call, and the WebSocket is closed.
6. Split testing, production, and usage by different customers across projects and API keys.
7. Set limits for automatic top-up of the balance, as well as alerts for fees and monitoring of abnormal sessions.
8. Regularly compare the cost of manual proofreading, model accuracy, and the cost per successful task.

Voice Agent API

The Voice Agent API integrates real-time speech recognition, language models, speech synthesis, turn-taking determination, interruption handling, and tool invocation within a single WebSocket connection. Its public pricing is $4.50 per hour, making it suitable for teams that wish to reduce the effort required for managing multiple services.

Data Security and Governance

• Project isolation: Different projects have separate API keys and distinct ranges of historical data.
• Role permissions: The account supports three types of roles: Owner, Admin, and Reader.
• Multi-factor authentication: It can be enabled for members and enforced by the organization.
• Data control: Paid accounts allow users to set retention periods and determine when model training should be halted.
• Business Partner Agreement: Medical-related teams can evaluate and sign a BAA.
• Regional processing: You can choose endpoints in the United States or the European Union to meet the requirements regarding data residency.
• Activity log: Records account and management actions for audit purposes.
• PII processing: Sensitive information in text or audio can be concealed using value-added masking techniques.

Product advantages

• It covers pre-recorded content, real-time audio, short audio clips, as well as end-to-end voice agents.
• The pricing is made public on an hourly basis, and pre-recorded audio is charged in seconds.
• It provides information on the speaker, timestamps, cues, and enhanced voice understanding capabilities.
• Official SDKs such as Python and JavaScript reduce the effort required for integration.
• Playground allows for comparing real audio with different models before encoding.
• Projects, member roles, MFA, data retention, and auditing are suitable for team governance.
• The $50 free credit makes it easy to conduct evaluations of representative prototypes.

Usage restrictions

• The accuracy of recognition is affected by language, accent, noise, overlapping speech, and technical terms.
• Stream sessions are billed based on the duration of the connection, and errors in connection management can result in additional charges.
• When multiple audio channels and various additional features are added, the cost can be significantly higher than the basic price.
• Some features are available only for specific models, languages, or pre-recorded modes.
• Old model parameters and the deprecated summary feature may cause migration issues.
• Cloud APIs are designed to send audio to external services, and they are not suitable for unapproved sensitive data.
• LLM Gateway is not included in the free tier and is charged separately based on the tokens of each model.
• The production system still needs to handle retries, callback security, rate limiting, and manual verification.

GitHub and SDK

AssemblyAI has an official GitHub organization, and it maintains Python SDKs, JavaScript SDKs, examples, as well as starter projects for voice agents. The fact that the SDKs are open source does not mean that the cloud-based voice models, training data, and hosted APIs are also open source.

Developers should rely on the current official documentation and SDK versions, paying special attention to changes in old real-time endpoints, model aliases, and parameters related to singular and plural forms. In a production environment, it is necessary to lock down the versions of dependencies and run regression tests before making any upgrades.

Basic information

Recommendation score

The overall recommendation score is 4.6 out of 5 points. AssemblyAI offers relatively complete interfaces, voice capabilities, documentation, and price transparency, making it suitable for developing production-grade voice applications; however, monitoring is necessary regarding real-time connection billing and additional premium features.

Frequently Asked Questions

What does AssemblyAI do mainly?

It converts pre-recorded or real-time audio into text through APIs, and offers voice recognition as well as agent capabilities.

Is there a free quota?

New accounts receive a $50 credit free of charge, no credit card is required, but LLM Gateway is not included.

How much does pre-recorded transcription cost?

Universal-3.5 Pro costs $0.21 per hour, while Universal-2 costs $0.15 per hour.

How much does real-time transcription cost?

Universal-3.5 Pro Realtime costs $0.45 per hour, while Universal-Streaming costs $0.15 per hour.

How is billing handled for flow-based recognition?

Charging is based on the duration of the WebSocket connection, including periods of inactivity and silence.

Is speaker separation supported?

Yes, the additional costs for preview mode and real-time mode are different.

How much does the Voice Agent API cost?

The public price is $4.50 per hour, calculated based on the duration of the session connection.

Is AssemblyAI open source?

The official SDKs and examples are open source, but the hosted models, training data, and the platform itself are not.

Is it suitable for medical data?

Medical models, BAA agreements, and data control can be assessed, but the organization still needs to carry out its own compliance checks.