Evaluation snapshot

10

Languages evaluated

85

Model chains

‍

500

Audio samples

Low-Resource Language Evaluations

As an AI orchestration platform focused on global impact, Gooey.AI - with the support of the Gates Foundation - actively aggregates and evaluates the latest private and open source AI models to assess their understanding of digitally underrepresented languages.

Our goal is to highlight real-world use cases of common health, agriculture and education audio questions and determine how well and how quickly different AI model combinations respond to these questions (as determined by their semantic similarity to expert provided "golden" answers).

Every evaluation, prompt, model settings and AI workflows are public and immediately forkable for organizations to test AI any workflow with your own data and use case. Organizations are encouraged to create audio evaluations with their own data too.

Read the PaperTalk with Us

Evaluation snapshot

10

Languages evaluated

85

Model chains

‍

500

Audio samples

Pakistani and Nigerian Health

In April 2026, we evaluated the latest and best-performing private (Gemini 3.1, OpenAI 5.4, Intron) and open source/sovereign deployable (Omnilingual, Gemma 4 26B, Kimi2.5) models for their performance on health-related audio questions in the languages commonly found in Pakistan and Nigeria. In our test, Gemini 3.1 Pro as the LLM + Intron or Meta's Omnilingual as the speech recognition model (ASR) scored most accurate, making the combination appropriate for WhatsApp and other async messaging based deployments, though likely too slow for usable voice-only services.

Pakistani Languages

For Urdu, GPT-5.5 + Omni and Gemini 3.1 Flash-Lite + Omni both lead on accuracy at ~0.7, with Flash-Lite + Omni also the fastest among the top performers, and in general, at ~9s latency. Gemini 3.1 Pro Preview + Omni has fairly good accuracy but the highest latency (~32s), making it better suited to async use cases.

Urdu Evaluation

For Pashto, GPT-5.5 + Omni clearly leads on accuracy (~0.71) with moderate latency (~14s). Compared with the Urdu results, Pashto appears more accuracy-constrained overall: even the top model scores are lower than Urdu’s best performers, and the strongest real-time balance is likely Gemini 3.1 Flash-Lite + Omni, which keeps latency under ~8.5s while reaching mid-range accuracy.

Pashto Evaluation

For Sindhi, GPT-5.5 + Omni is the most accurate option (~0.73) at roughly 12s latency, with Gemini 3.1 Pro Preview and Gemini 3.1 Pro Preview + Omni close behind but slightly slower. Compared with Urdu, Sindhi’s top accuracy is lower, but it looks closer to Pashto in overall performance; Gemini 3.1 Flash-Lite + Omni again offers the practical speed–accuracy middle ground.

Sindhi Evaluation

Nigerian Languages

For Hausa, Gemini 3.1 Pro achieved the highest Semantic Similarity Score (Accuracy) while maintaining lower latency than most tested configurations (0.71; ~16.3s). Claude Sonnet 5 + Omni was fastest overall, but its accuracy was considerably lower (0.5; ~13.6s).

‍

Hausa Evaluation

For Nigerian Fulfulde, Gemini 3.7 Flash + Omni achieved the highest Semantic Similarity Score (Accuracy) while delivering nearly the lowest latency, making it the strongest overall configuration (0.59; ~15.5s). Claude Sonnet 5 + Omni was marginally faster, but its accuracy was substantially lower (0.23; ~15.3s).
‍

Nigerian Fulfulde Evaluation

For Igbo, Kimi K3 + Intron achieved the highest Semantic Similarity Score (Accuracy) (0.55; ~23s), but GPT 5.6 Sol + Omni followed closely while delivering the lowest latency (0.51; ~14.91s). This made GPT 5.6 Sol + Omni the strongest overall speed–accuracy trade-off.

Igbo Evaluation

For Yoruba, Gemini 3.1 Pro achieved the highest Semantic Similarity Score (Accuracy) by a clear margin while maintaining moderate latency (0.61; ~17.2s). Claude Sonnet 5 + Omni was fastest overall, but its accuracy was substantially lower (0.27; ~13.9s).

Yoruba Evaluation

For Nigerian Pidgin, Kimi K3 + Intron achieved the highest Semantic Similarity Score (Accuracy) (0.75; ~22.2s), while Kimi K3 + Omni followed closely with lower latency (0.71; ~18.8s). Claude Sonnet 5 + Omni was fastest overall, but its accuracy was considerably lower (0.62; ~16.2s).

Pidgin Evaluation

Kinyarwanda, Swahili and Kikuyu

In December, 2025 We tested new LLM+ASR AI workflows to understand real-world deployment Swahili, Kikuyu, and Kinyarwanda voice applications. Fine-tuned African ASR models + modern LLMs (GPT-5.1, Gemini 3) achieved 85-96% accuracy with 4-10 second response times—fast enough for phone-based services. In short, Voice AI for many African languages is now technically viable for WhatsApp and basic phone deployments at scale.

Read the Paper

For Kinyarwanda, DeepSeek V4 + Omni achieved the highest Semantic Similarity Score (Accuracy) while maintaining relatively low latency (0.92; ~7.1s). Kimi K3 + Omni was fastest overall, but its accuracy was noticeably lower (0.78; ~6.7s).

Kinyarwanda Evaluation

For Swahili, Gemini 3.1 Pro was the clear leader for Swahili, achieving both the highest semantic accuracy and lowest response latency (0.95; ~5.8s). It delivered the strongest speed–accuracy balance of every model configuration tested.

Swahili Evaluation

For Kikuyu, Gemini 3.1 Pro + Kikuyu Whisper v2 was the most accurate by a clear margin (0.86; ~9.5s). Claude Sonnet 5 + Kikuyu Whisper v2 was fastest overall (0.63; ~6.3s), while Gemini 3.7 Flash + Kikuyu Whisper v2 offered a strong balance of speed and accuracy (0.82; ~7.1s).

Kikuyu Evaluation

Theory of Change

AI systems don’t understand local-language speech

The context

Many of Africa’s most widely spoken languages—such as Swahili, Kikuyu and Kinyarwanda—have historically been “invisible” to AI due to limited multilingual datasets used to train LLMs. This results in unequal access to AI and underrepresentation of people in underserved areas.

The ground reality

Unfortunately, capturing this diversity of language is hard for tech companies attempting to make speech recognition and translation models. This poor performance then implies that incredible tools like GPT4 and Gemini - which are primarily trained on English - don’t work particularly well for speakers of low-resource languages.

The need for AI-driven support

AI interventions such as small shareholder agriculture advisory show significant promise in aiding livelihood opportunities in the Global South. Everyone regardless of their language or location should be able to access digital services through voice, not text.

Interventions and Process

The possibilities in 2025

Local NLP and AI teams such as Jacaranda Health and MBaza released better performing fine-tuned African speech recognition models. Additionally, Meta released Omnilingual ASR with support for 1600+ languages.

New LLM generations (GPT-5+, Gemini 3) dramatically improved multilingual text comprehension of local-language text.

Model chains and Architectures

We use various Models & Architectures to create the evaluation process. This is very easily configurable with the help of Gooey.AI's Orchestration tech that allows us to quickly chain ASR -> LLM -> TTS or more.

Our methodology

Golden QnA
Native speakers record audio questions in three African languages; human experts provide gold-standard English answers for consistent evaluation across systems.
‍
LLM-as-a-judge
GPT-5.1 scores system responses against expert answers on 0-1 scale, measuring semantic correctness since African language evaluation remains unreliable.

Latency measurements
Measures total time from audio question to audio answer, including ASR, translation, LLM inference, and TTS on A10 GPUs.

Key Findings

Breaking the audio barrier for African languages

Fine-tuned African ASR models—Jacaranda, MBaza, and Sunbird—trained on high-quality, diverse audio datasets, now reliably convert speech into local-language text with significantly improved accuracy. By addressing dialectal variation and acquiring local and region-specific finetuned training data, these models remove the biggest historical bottleneck for low-resource African languages, enabling scalable speech-to-text for research, civic tech, education, and voice interfaces across the continent and supporting inclusive digital access everywhere today.

Native text understanding without translation layers

New large language models such as GPT-5+ and Gemini 3 can natively understand Swahili, Kikuyu, and Kinyarwanda text, often without requiring machine translation. Even in Q3 2025 these models required Machine Translations to understand and translate information to the LLMs. This update in LLMs improves downstream tasks like information retrieval, summarization, and question answering. Native comprehension reduces latency, translation errors, and developer overhead, enabling more accurate, accessible applications for local services, education, and other frontline work.

End-to-end voice AI ready for real-world use

ASR→LLM→TTS pipelines now deliver acceptable latency and robustness for phone-based voice services on basic feature phones and low-bandwidth networks. SIP via Livekit and other telephony improvements for AI models reduce latency significantly. These end-to-end systems enable automated call handling, IVR, and conversational agents across several African languages, making scalable, affordable voice services for healthcare, banking, government, and small businesses with deployable latency and cost profiles today.

Relevant Links

Explore workflows

Sign Manifesto

10

85

500