KDDI Cuts RAG Latency by 38% Using Automated Evaluation and Google ADK

KDDI tackles the performance challenge behind consumer RAG

Building a consumer-facing Retrieval-Augmented Generation (RAG) application involves more than connecting a large language model to a document collection. Applications must retrieve relevant information, generate reliable answers and return those answers quickly enough for users.

Japanese telecommunications company KDDI encountered this challenge while developing Buffmee, a consumer RAG application designed around the idea of AI that helps users grow their knowledge.

Buffmee grounds responses in more than 100 sources, including books, magazines and web media. The service allows users to search for information, summarise key points and explore learning and hobby interests, while source citations are used to support information reliability.

However, KDDI faced latency problems while working with a large and diverse collection of proprietary content. The team also needed a systematic approach to evaluate answer quality and reduce hallucinations.

According to Google Cloud, the resulting optimisation reduced total application response latency by 38%, while Time To First Token (TTFT) improved by nearly 18%.

Automated evaluation replaces large-scale manual testing

KDDI built an automated evaluation process using the Gemini Enterprise Agent Platform Evaluation Service.

The team used approaches including LLM-as-a-Judge and the Rule of Hundreds to create a structured testing process. Rather than depending entirely on manual testing, KDDI ingested its document corpus, generated hundreds of evaluation tests and created a benchmark dataset for measuring answer reliability across different use cases.

Google Cloud reports that this approach improved KDDI's groundedness scores by 25%.

The evaluation process also retained a human role. Automated systems generated questions and evaluated answers, while humans helped calibrate thresholds and investigate edge cases.

This combination is important because automated evaluation can provide scale, but product teams still need to determine what constitutes an acceptable answer for their specific application.

Four practices shaped the evaluation strategy

KDDI's approach centred on four technical practices.

1. Binary evaluation for critical metrics

The team selectively moved from ambiguous 1-to-5 ratings to a binary pass-or-fail model for critical metrics.

The goal was to reduce variation and noise in automated evaluation, making results easier to use within an automated workflow.

2. Representative content sampling

KDDI did not attempt to evaluate every document individually.

Instead, it classified its content using two dimensions: file format and media composition. File formats included web articles, EPUBs, PDFs and structured data, while media composition distinguished text-heavy, image-heavy and mixed content.

The team then selected representative samples from the resulting difficulty grid.

Google Cloud says this reduced the evaluation workload by 75% while maintaining test coverage.

3. Human-calibrated thresholds

KDDI's product owner reviewed randomly sampled answers alongside automated scores.

This allowed the team to calibrate evaluation thresholds around what was considered sufficiently good for the intended user experience rather than relying only on default parameters.

4. Breaking large prompts into ADK Skills

The team also addressed prompt complexity.

KDDI split large system prompts according to function using Agent Development Kit Skills. The approach allowed required logic to be loaded dynamically instead of placing all instructions into one large prompt.

Google Cloud says the team specifically examined prompts exceeding 800 lines as part of its optimisation work, connecting prompt size with attention drift and latency concerns.

Production logs reveal where latency comes from

Evaluation alone does not explain why an AI application is slow.

KDDI therefore analysed actual production behaviour using BigQuery Agent Analytics and an Agent Development Kit log analysis agent.

The team used production logs to examine how different parts of the agent architecture affected Time To First Token. This included investigating skill division, prompt size and sub-agent routing.

The analysis helped identify deeper bottlenecks in the agent stack. KDDI then optimised the system prompt, including the way skills were integrated, and reviewed sub-agent routing.

The objective was to improve response performance without sacrificing answer accuracy.

What KDDI's approach shows about production RAG

KDDI's experience highlights that RAG performance is not solely a retrieval problem.

The application has several interconnected layers: content ingestion, retrieval, prompts, agent skills, sub-agent routing, model generation and evaluation. A bottleneck in one layer can affect the overall user experience.

The approach also demonstrates the value of treating evaluation and optimisation as an ongoing engineering process rather than a final testing phase.

Automated evaluation provides repeatable measurements, while production telemetry can reveal performance problems that may not appear in controlled tests. Human review then provides a reference point for determining whether automated scores reflect the actual product experience.

For enterprises building RAG applications, this suggests a practical workflow:

1. Establish measurable quality criteria.

2. Build representative evaluation datasets.

3. Automate recurring tests.

4. Monitor production execution.

5. Identify latency bottlenecks from real workloads.

6. Optimise prompts, skills and routing.

7. Re-evaluate quality after performance changes.

Broader industry perspective

The announcement highlights a broader industry shift toward treating generative AI applications as production software systems that require continuous measurement and optimisation.

As RAG applications consume increasingly diverse enterprise content, teams may need to evaluate not only whether an answer is correct, but also how quickly the system produces it and how consistently it performs across different content types.

KDDI's Buffmee case provides one example of this approach. The company combined automated evaluation, representative content sampling, human calibration and production log analysis to improve both quality measurement and application performance.

Google Cloud recommends using its Gen AI evaluation service for systematic quality measurement, structuring agents with ADK and inspecting retrieval queries when applying similar techniques.