Suyog K
Back to all writing

Teaching a model to speak Sanskrit the Nepali way

In Nepal, Sanskrit isn't a museum language. You hear it at weddings, at a bratabandha, at every morning puja, recited by pujaris in a style passed down by ear for generations. Ask a computer to say the same verse, though, and it stumbles. It stops halfway through a long line. It reads the numbers in Hindi or Marathi.

NepSanskrit is my attempt to fix that. It takes Nepali text, turns it into Sanskrit and reads the Sanskrit aloud. The aim is a voice that is clear, quick and light enough to run on an ordinary laptop, and one day a voice that carries the Nepali recitation tradition, something none of the existing systems were built for.

Here's where it stands today, in a real run on my laptop:

A Nepali sentence about three temples in a village becomes Sanskrit. The digit 3 is spoken as the Sanskrit word for three, and the recogniser scores the result at 6.5% letter error. Recorded on a MacBook with an M5 chip and 16 GB of memory.

Under the hood, a sentence travels through five steps, and doubtful translations take a detour to a human expert:

Pipeline: Nepali text, translation, Sanskrit cleanup, voice, clarity check. Doubtful translations are flagged and sent to a pujari for review. Translation, cleanup, the clarity check and flagging are built or fine-tuned by me; the voice is an open model used as is.
How a sentence travels through NepSanskrit. The boxes outlined in orange are the parts I built or fine-tuned myself.

Starting with almost nothing

The first surprise was how little there is to learn from. Apart from about 2,000 sentences in Meta's FLORES-200 benchmark, which exist in both Nepali and Sanskrit, I couldn't find a single dedicated Nepali to Sanskrit corpus or benchmark. So most of this project is about getting a lot out of very little.

Step one: translation

I started from IndicTrans2, AI4Bharat's translation model for Indian languages, and fine-tuned it on 897 sentence pairs from the FLORES dev set. I then tested it on the separate FLORES devtest set of 1,012 sentences, so nothing it was tested on was seen in training.

It reached a chrF++ score of 30.2, a standard measure of translation quality, which is clearly better than the open models I tried: NLLB, and IndicTrans2 before tuning. To be honest about where it stands, the output is understandable rather than fluent. You'd get the meaning, but a Sanskrit teacher would reach for a red pen. I also haven't compared it with Google Translate or large language models yet, and that's on my list.

Step two: teaching the voice some manners

For the voice I use Indic Parler-TTS. Out of the box it had two bad habits. On long sentences it would simply stop early, as if it had run out of breath. And numbers came out in Hindi or Marathi, which is jarring in the middle of a Sanskrit line. Three changes made a real difference:

  • speaking in short chunks instead of whole long sentences,
  • writing every number out as Sanskrit words before the voice sees it,
  • automatically redoing any recording that ends too early.
Letter error rate of Indic Parler-TTS dropped from 8.3 percent before the fixes to 5.0 percent after. Real human chanting scores 8.9 percent on the same recogniser.
Letter error rate on my own 76 sentence test set, before and after the three fixes.

How do you measure whether a voice speaks clearly? I let another machine listen. The Vakyansh Sanskrit speech recogniser transcribes each recording, and I count how many letters it gets wrong. As a sanity check, I ran the same recogniser on real human chanting released by the Vāgdhenu project, and it scored 8.9%.

So 5.0% doesn't mean the voice is better than a human. Chanting is melodic and stretched, which probably makes it harder for a machine to transcribe. The number comes from a single machine judge, and I haven't run a listening test with real people yet. What it does say is that the voice is now easy to understand. It measures clarity, not beauty, and beauty is the harder problem still ahead.

Speed is the other gap. Right now Indic Parler-TTS takes about two seconds to produce each second of speech on my MacBook. That's fine for a demo, but not yet the instant, laptop-friendly voice I'm aiming for.

Where voices slip

Sanskrit packs a lot of meaning into very small sounds. The visarga, the soft breath at the end of a word, is a good test. The voice in the app handles it well: Indic Parler-TTS keeps 99% of visargas.

The trouble showed up when I tried IndicF5, a voice that can clone a speaker from a short sample and my best route to a pujari's voice. Used straight out of the box, it keeps only 8% of visargas and 72% of aspirated sounds, because it learnt Devanagari from Hindi and Marathi, not Sanskrit. Drop the visarga from मूषकाः (mice) and the word is no longer quite right. Across the full 76 sentence test, IndicF5 scored 14.9% letter error, and it takes about four seconds per second of speech.

The text says मूषकाः with its visarga. Zero-shot IndicF5 said मूषका, with the visarga missing. Visargas kept: Indic Parler-TTS 99 percent, IndicF5 zero-shot 8 percent.
Out of the box, IndicF5 swallows the visarga. That's exactly what fine-tuning on Sanskrit should fix.

Spending expert time wisely

The judgement of a traditional Nepali pujari is rare and precious, so it shouldn't be spent reading translations the machines already agree on. Instead, a flagging step ranks every translation by how likely it is to be wrong, and only the most doubtful ones go for review.

Review funnel: every translation is ranked, the top 20 percent are flagged for the pujari, and these contain 39 percent of all the bad translations.
Reviewing a fifth of the translations catches about two fifths of the mistakes.

Reviewing just the top 20% catches about 39% of all the bad translations, roughly twice what picking at random would find (random: about 20%). It helps a lot, but it doesn't catch everything, and making it sharper is part of the work ahead.

What's next

The source of new Sanskrit speech is IndicVoices-R: 33 hours of Sanskrit recorded by 147 different speakers. From it, I've picked the cleanest 20 hours, from 131 speakers, and the first job is fine-tuning IndicF5 on them so it learns to keep every visarga and aspirate.

After that, I'll train a small, fast voice of my own: an MMS VITS model from Meta with 36 million parameters. Then comes the exciting part, teaching it to sound like a Nepali pujari, either by adapting the model to a pujari's own recordings or by cloning a voice with IndicF5. Recent research on linguistic kinship in speech synthesis suggests a voice can borrow from closely related languages, which is promising when Sanskrit recordings are this scarce.

Later on, I want to scale this up by inviting Sanskrit speakers and the wider community to contribute recordings, so the voice can keep getting better.

The question I can't answer yet

Is there really a distinct Nepali way of reciting Sanskrit, and can a model learn it? For now that is still a hypothesis. The answer hangs on small details, like how sounds such as ष, an initial य and ज्ञ are pronounced in the Nepali tradition. That's the part I'm most curious about, and I'll write about it here once I know more.

Built on open work

NepSanskrit stands on the shoulders of several research groups, and each deserves credit. My own contribution sits between their pieces: fine-tuning the translation for Nepali to Sanskrit, cleaning up the Sanskrit for speaking, checking every recording for clarity and sending only the doubtful translations for expert review. Every change is measured on a fixed test set before it goes into the app.

Sources feeding into NepSanskrit: AI4Bharat (IndicTrans2, IndicF5), Meta (FLORES, NLLB, MMS), Hugging Face (Indic Parler-TTS), Vakyansh (Sanskrit recogniser), Vāgdhenu (human chant audio) and Ambuda (verse and stotra texts).
The open projects NepSanskrit builds on.

Translation starts from IndicTrans2 by AI4Bharat, adapted to Nepali to Sanskrit with the FLORES-200 benchmark from Meta's No Language Left Behind project and compared against NLLB itself. The voice comes from Indic Parler-TTS, an AI4Bharat and Hugging Face model based on the Parler-TTS approach. The next voice experiments use IndicF5 from AI4Bharat, which builds on F5-TTS, and Meta's compact MMS voices, learning from the IndicVoices-R Sanskrit speech. Clarity is judged with the Vakyansh Sanskrit recogniser, checked against human chant recordings from the Vāgdhenu project, and Sanskrit verse and stotra texts come from the Ambuda library.

A note on licences: Indic Parler-TTS, the voice in the demo, is openly licensed, while the MMS voices are for non-commercial use only. Any recordings of a pujari will only ever be used with their consent.

References

  • Gala et al. (2023). IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages. arXiv:2305.16307
  • NLLB Team (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv:2207.04672
  • Lyth and King (2024). Natural language guidance of high-fidelity text-to-speech with synthetic annotations. arXiv:2402.01912
  • Chen et al. (2024). F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. arXiv:2410.06885
  • Pratap et al. (2023). Scaling Speech Technology to 1,000+ Languages. arXiv:2305.13516
  • Sankar et al. (2024). IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS. NeurIPS 2024. arXiv:2409.05356
  • Pathak et al. (2025). Kinship in Speech: Leveraging Linguistic Relatedness for Zero-Shot TTS in Indian Languages. Interspeech 2025. arXiv:2506.03884
  • Chadha et al. (2022). Vakyansh: ASR Toolkit for Low Resource Indic languages. arXiv:2203.16512
  • Vāgdhenu project, human Sanskrit chant recordings.
  • Ambuda, a free and open library of Sanskrit heritage. ambuda.org