Skip to main content

whisper_normalizer

A Python package for text normalization/standardisation for ASR evaluation and TTS. It implements the Whisper normalizer and provides language-aware normalizers for 101 languages, including dedicated Indic and international support. Indic normalization is derived from indic-nlp-library.

PyPI Downloads Github license Github Stars PyPI version Humans behind

Based on the text normalization approach from OpenAI Whisper; see Appendix C of Robust Speech Recognition via Large-Scale Weak Supervision. Full documentation: kurianbenoy.github.io/whisper_normalizer.

Installation

pip install whisper_normalizer

Or install from GitHub:

pip install git+https://github.com/kurianbenoy/whisper_normalizer.git

Quick start

Use Normalizer to select a language-aware normalizer:

from whisper_normalizer import Normalizer

normalizer = Normalizer("en")
normalizer("I'm a little teapot, short and stout. Tip me over and pour me out!")
# 'i am a little teapot short and stout tip me over and pour me out'

EnglishTextNormalizer and BasicTextNormalizer are also available as direct classes:

from whisper_normalizer.basic import BasicTextNormalizer

normalizer = BasicTextNormalizer()
normalizer("I'm a little teapot, short and stout. Tip me over and pour me out!")
'i m a little teapot short and stout tip me over and pour me out '

Supported languages

The package covers 101 languages in three tiers. Dedicated and script-tier normalizers:

Language Code Normalizer Tier Digits → words Marks tts_mode
Arabic ar ArabicTextNormalizer dedicated tts_mode stripped yes
Bengali bn BengaliNormalizer dedicated always kept yes
Chinese zh ChineseTextNormalizer dedicated never kept no
English en EnglishTextNormalizer dedicated tts_mode stripped yes
French fr FrenchTextNormalizer dedicated tts_mode kept yes
Gujarati gu GujaratiNormalizer dedicated always kept yes
Hindi hi HindiNormalizer dedicated always kept yes
Kannada kn KannadaNormalizer dedicated always kept yes
Malayalam ml MalayalamNormalizer dedicated always kept yes
Odia or OdiaNormalizer dedicated always kept yes
Punjabi pa PunjabiNormalizer dedicated always kept yes
Russian ru RussianTextNormalizer dedicated tts_mode kept yes
Spanish es SpanishTextNormalizer dedicated tts_mode kept yes
Tamil ta TamilNormalizer dedicated always kept yes
Telugu te TeluguNormalizer dedicated always kept yes
Assamese as BengaliNormalizer script always kept yes
Marathi mr DevanagariNormalizer script always kept yes
Nepali ne DevanagariNormalizer script always (Hindi number words) kept no
Sanskrit sa DevanagariNormalizer script always (Hindi number words) kept no

The other 82 languages use the generic tier: Unicode NFC, brackets and punctuation removed, case folded, combining marks kept. Digits become words only with tts_mode=True, which 33 of them support.

Normalizer(code) exposes .tier and .capabilities, while supported_languages() returns the full list.

Indic languages and combining marks

Whisper’s BasicTextNormalizer can strip combining marks and is not recommended for Indic languages; see Dr Kavya Manohar’s post. Use a dedicated Indic normalizer or BasicTextNormalizer(preserve_marks=True), which keeps vowel signs, viramas and other combining marks.

For example:

from whisper_normalizer.indic import MalayalamNormalizer

normalizer = MalayalamNormalizer()
normalizer("എന്റെ കമ്പ്യൂട്ടറിനു് എന്റെ ഭാഷ.")
'എന്റെ കമ്പ്യൂട്ടറിന് എന്റെ ഭാഷ.'

TTS mode

tts_mode=True produces speakable text by expanding numbers, currency, decimals, URLs, emails, symbols and acronyms. It is available for many languages, including English, French, Spanish, Arabic and Russian, and needs the optional dependency:

pip install "whisper_normalizer[tts]"
from whisper_normalizer import Normalizer

Normalizer("hi", tts_mode=True)("नमस्ते ₹100")
# 'नमस्ते रुपये एक सौ'

Unsupported options raise ValueError.

Citation

If you use whisper_normalizer in your work and would like to cite it, please thank Kurian Benoy and link to this GitHub repository: github.com/kurianbenoy/whisper_normalizer.