Calling-AI — Realtime AI Voice Calling Agent

Calling-AI is a realtime AI voice calling platform. It automates outbound conversations for cold outreach, product demonstrations, lead qualification, and service follow-ups.

Our team designed the complete voice pipeline: realtime audio transport, speech-to-speech LLM integration, SIP telephony, agent prompt management, call analytics, and post-call task extraction. The platform uses LiveKit, Google Gemini Live, and Zadarma SIP to connect conversational AI agents with real telephone numbers while maintaining a time to first response of approximately 600 ms.

The Background

Traditional outbound calling is expensive to scale. Every additional campaign requires trained agents, more working hours, quality monitoring, and repeated onboarding around scripts and brand communication rules.

Call-center automation can reduce this workload, but basic IVR systems and prerecorded messages cannot hold a flexible conversation. Chatbots solve a different problem: they wait for users to open a website or messenger and type a question. They do not call a prospect, respond to interruptions, clarify an answer, or adjust the conversation based on what the person says.

Calling-AI was created as a voice-first alternative. The agent places real outbound calls, follows a defined business objective, listens to the customer, and generates spoken responses during the conversation.

Each agent can have its own use case, language, opening line, and system prompt. One agent may handle cold outreach, while another demonstrates a product or follows up with an existing client. Businesses can test these agents in a browser before connecting them to live phone campaigns.

The platform also processes completed conversations. Instead of keeping only a recording and call duration, it extracts sentiment, conversation topics, quality scores, conversion outcomes, and requested follow-up actions.

The Goal

The main engineering task was to build a realtime conversational AI pipeline that could respond naturally enough for an actual telephone conversation while remaining configurable across multiple brands and business scenarios.

Realtime Voice Pipeline

LiveKit provides the realtime media layer between the user and the agent. Its WebRTC infrastructure handles continuous audio streams, room management, and low-latency communication. Gemini Live processes the incoming speech and generates native audio responses, while the application layer controls the agent’s prompt, conversation state, tools, and business rules. LiveKit is designed for realtime WebRTC media exchange, while Gemini Live supports continuous, bidirectional audio interactions.

The system streams audio rather than waiting for a complete recording. User speech is sent to the model as it arrives, and generated output is returned in chunks for immediate playback. Audio capture, model processing, and response delivery run as parallel stages rather than one long sequential request.

This architecture is central to low-latency voice AI. A text chatbot can take several seconds to respond without making the interaction unusable. In a voice call, the same pause feels unnatural and often causes the customer to repeat themselves or interrupt the agent.

Maintaining TTFT at Approximately 600 ms

The platform tracks TTFT – the time between the end of a user turn and the beginning of the agent’s response. For Calling-AI, the measured value is approximately 600 ms.

To keep this delay stable, the team reduced unnecessary transformations between the audio transport and the model, used streamed model responses, and avoided blocking analytics or database operations inside the active conversation loop.

Call logging, transcript processing, scoring, and topic extraction happen outside the latency-sensitive response path. This separation prevents post-call AI tasks from slowing down the live conversation.

Voice options are tested under the same conditions, with TTFT displayed alongside each voice profile. This makes latency a visible quality metric rather than an assumption hidden inside the infrastructure.

SIP Telephony Integration

Browser testing is useful during development, but the product also needed to call standard mobile and landline numbers.

The team integrated Zadarma through SIP, creating a bridge between LiveKit’s WebRTC-based agent environment and the public telephone network. SIP trunking connects internet-based voice software with PSTN numbers and supports real outbound call routing.

Managers select an agent, enter a destination number in E.164 format, choose a voice, and start an outbound call. The telephony layer handles call initiation and status events, while the realtime AI session manages the conversation once the call is answered.

The platform records whether a call was completed, failed, or unanswered, along with its duration, agent, brand, campaign, and telephone number.

Voice Selection and Evaluation

Different calls require different vocal characteristics. A warm, friendly voice may fit customer service, while B2B outreach benefits from a clearer and more direct delivery. Delicate conversations may require a slower and softer style.

Calling-AI includes a library of multilingual voices such as Aoede, Leda, Kore, Zephyr, Puck, and Charon. Each voice has a short description, recommended use case, measured response time, and internal quality rating.

The testing interface also includes a notes field. Team members can document pronunciation issues, pacing, emotional tone, or situations where a particular voice performs poorly. This creates a repeatable evaluation process instead of selecting a voice based on one short demo.

Brand Voice Injection

Every brand in the workspace has its own communication rules. For example, the RexSoft brand voice specifies a professional, friendly, and confident tone, Ukrainian language, and short, direct answers.

This configuration is stored separately from individual agent prompts. When an agent session starts, the platform injects the brand-level rules into the agent’s system prompt.

The approach allows one business to maintain consistent communication across several agents without copying the same instructions into every configuration. Updating the brand voice changes the shared layer while preserving each agent’s specific objective, opening phrase, knowledge, and conversation logic.

Two-Level Testing Workflow

Prompt iteration can become expensive when every change requires a paid telephone call. To control development costs, the team separated testing into two workflows.

The browser test runs through LiveKit and Gemini Live without connecting to the telephone network. It allows prompt engineers and managers to check the opening line, conversation flow, language, interruptions, and voice behavior quickly.

Once the agent performs correctly in the browser, it can be tested through a real outbound SIP call. This second stage verifies telephone audio quality, number formatting, call routing, connection stability, and behavior under normal mobile-network conditions.

The separation supports faster AI voice agent development because most prompt and dialogue issues can be corrected before incurring telephony costs.

Multi-Brand and Multi-Agent Architecture

The platform is organized around workspaces, brands, and agents.

A workspace can contain several brands. Each brand has its own description, website, industry, contacts, campaigns, knowledge sources, agents, and brand voice. Every agent belongs to a brand but has an independent use case, language, opening line, and system prompt.

This structure supports agencies, product groups, and companies operating several customer-facing brands from one account. Team access is managed through roles such as Manager, tenant administrator, and super administrator.

Call Analytics and AI Scoring

The conversational model handles the live dialogue, but a separate AI pipeline analyzes the conversation after the call.

Once a transcript is available, the platform uses an LLM to assess the call, estimate customer sentiment, identify the main topics, and determine whether the campaign objective was achieved. The result is stored alongside operational call data.

The analytics dashboard combines these AI outputs with standard metrics such as total calls, average duration, answer rate, failed calls, and conversion rate. Users can filter the journal by agent, campaign, status, rating, and availability of AI analysis, or search directly inside call transcripts.

Separating live conversation AI from post-call analysis also makes the system easier to maintain. The realtime agent remains focused on fast responses, while the analytics layer can use longer prompts and more detailed evaluation criteria without affecting call latency.

AI-Detected Follow-Ups

Customers frequently ask for information after a call: a presentation, pricing details, product documentation, or a message in a preferred messenger.

Calling-AI detects these requests in the transcript and creates a follow-up task containing the contact, requested material, communication channel, related call, campaign, and status.

The current workflow keeps execution under human control. The AI identifies and structures the request, while a manager sends the material through channels such as Telegram or Viber and marks the task as completed.

This reduces the chance that an important promise made during a call is lost inside a recording or transcript.

The Development Team

Voice AI and Realtime Engineer

The realtime engineer built the LiveKit and Gemini Live pipeline, managed streamed audio sessions, optimized the response path, and monitored latency across browser and telephone calls.

Back-End Developer

The back-end developer implemented SIP telephony integration for AI agents, call lifecycle handling, workspace and brand isolation, access roles, campaign data, quotas, logs, and post-call processing.

Prompt Engineer

The prompt engineer designed the agent instructions, opening-line logic, brand voice composition, use-case rules, and evaluation prompts used for sentiment, scoring, topic extraction, and follow-up detection.

Front-End Developer

The front-end developer created the workspace dashboard, agent settings, call testing tools, voice comparison interface, call journal, analytics views, and follow-up queue.

Project Manager

The project manager coordinated the realtime, telephony, AI, and interface workstreams. The role also covered test planning, agent-use-case definition, prioritization, and communication between technical and business stakeholders.

The Results

Calling-AI became a working internal and demonstration platform for launching AI agents on real outbound calls.

A team can configure a brand, define its communication style, create agents for different use cases, test them in the browser, compare multilingual voices, and move to a real telephone call from the same workspace.

After deployment, managers can monitor the number of agents, campaigns, and calls; review answered and failed calls; search transcripts; compare duration and AI scores; track conversion and answer rates; and see the topics and customer sentiment extracted from conversations.

The follow-up module converts verbal customer requests into structured tasks instead of leaving them inside call recordings. The multi-brand architecture allows the same platform to support separate communication styles, agent prompts, languages, and campaigns.

The result is an AI calling platform development case focused on the complete operational pipeline: LiveKit integration, Gemini Live conversations, SIP telephony, low-latency audio, prompt management, AI call analytics, and team workflows.

For businesses exploring voice AI for call centers, sales outreach, lead qualification, or service follow-ups, Calling-AI demonstrates how realtime voice models can be turned into a controlled and measurable calling system.