Indic Languages Normalizer

This code is from: https://github.com/anoopkunchukuttan/indic_nlp_library

Also use Indic Numtowords: https://github.com/raj-sutariya/indic-num2words, https://github.com/AI4Bharat/indic-numtowords

This code has been modified by Kurian to suit to Whisper-normalizer style of coding and the logic for Malayalam normalization is expanded beyond the Indic NLP library by Dr Kavya.

---

source

NormalizerI

def NormalizerI(
    *args, **kwargs
):

The normalizer classes do the following: * Some characters have multiple Unicode codepoints. The normalizer chooses a single standard representation * Some control characters are deleted * While typing using the Latin keyboard, certain typical mistakes occur which are corrected by the module Base class for normalizer. Performs some common normalization, which includes: * Byte order mark, word joiner, etc. removal * ZERO_WIDTH_NON_JOINER and ZERO_WIDTH_JOINER removal * ZERO_WIDTH_SPACE and NO_BREAK_SPACE replaced by spaces Script specific normalizers should derive from this class and override the normalize() method. They can call the super class ’normalize() method to avail of the common normalization

---

source

TtsLexicon

def TtsLexicon(
    rupee:str, dollar:str, won:str, point:str, http:str, https:str, dot:str, slash:str, at:str,
    spell_fraction_digits:bool=False
)->None:

The spoken words a language contributes to tts_mode; the rewrite rules themselves are shared.

---

source

BaseNormalizer

def BaseNormalizer(
    lang, remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, tts_mode:bool=False
):

Common class used in most of indic languages inherit from this code.

---

source

DevanagariNormalizer

def DevanagariNormalizer(
    lang:str='hi', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, tts_mode:bool=False
):

Normalizer for the Devanagari script. In addition to basic normalization by the super class, * Replaces the composite characters containing nuktas by their decomposed form * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script

norm = DevanagariNormalizer()
TEST_RESULT = "चीर धाराएं समुद्र तट पर लगकर लौटने वाली लहरों का प्रवाह होता है अक्सर एक चट्टान या इसी तरह के पदार्थों पर"
hi_text = "चीर धाराएं समुद्र तट पर लगकर लौटने वाली लहरों का प्रवाह होता है अक्सर एक चट्टान या इसी तरह के पदार्थों पर"
norm(hi_text)

'चीर धाराएं समुद्र तट पर लगकर लौटने वाली लहरों का प्रवाह होता है अक्सर एक चट्टान या इसी तरह के पदार्थों पर'

assert norm(hi_text) == TEST_RESULT

sample_text = "भारत का सकल घरेलू उत्पाद 1.5 ट्रिलियन अमेरिकी डॉलर है।"
norm(sample_text)

'भारत का सकल घरेलू उत्पाद एक.पाँच ट्रिलियन अमेरिकी डॉलर है।'

---

source

HindiNormalizer

def HindiNormalizer(
    lang:str='hi', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, tts_mode:bool=False
):

Fork of Devanagiri normalizer. With additional changes for Hindi and tts_mode.

norm = HindiNormalizer(tts_mode=True)
TEST_RESULT = "चीर धाराएं समुद्र तट पर लगकर लौटने वाली लहरों का प्रवाह होता है अक्सर एक चट्टान या इसी तरह के पदार्थों पर"
hi_text = "चीर धाराएं समुद्र तट पर लगकर लौटने वाली लहरों का प्रवाह होता है अक्सर एक चट्टान या इसी तरह के पदार्थों पर $134.5"
norm(hi_text)

'चीर धाराएं समुद्र तट पर लगकर लौटने वाली लहरों का प्रवाह होता है अक्सर एक चट्टान या इसी तरह के पदार्थों पर डॉलर एक सौ चौंतीस पॉइंट पाँच'

normalizer = HindiNormalizer(tts_mode=True)
output = normalizer("He spent Rs. 500 and $20 on groceries.")
print(output)

he spent रुपये पाँच सौ and डॉलर बीस on groceries.

normalizer = HindiNormalizer(tts_mode=True)
output = normalizer(
    "Visit https://example.com or mail us at help@support.in & get 20% off!"
)
print(output)

visit एच टी टी पी एस कोलन स्लैश स्लैश e x a m p l e डॉट c o m or mail us at help at s u p p o r t डॉट i n  or  get बीस percent  off!

text = "आज की तारीख में भारत में ₹100 अमेरिका में $1.1855 के बराबर है alexerws@gmail.com"
normalizer(text)

'आज की तारीख में भारत में रुपये एक सौ अमेरिका में डॉलर एक पॉइंट एक आठ पाँच पाँच के बराबर है alexerws at g m a i l डॉट c o m'

text = "प्रश्न 8 चे उत्तर 17.825 आहे."
normalizer(text)

'प्रश्न आठ चे उत्तर सत्रह पॉइंट आठ दो पाँच आहे.'

sample_text = "भारत का सकल घरेलू उत्पाद 1.5 ट्रिलियन अमेरिकी डॉलर है।"
normalizer(sample_text)

'भारत का सकल घरेलू उत्पाद एक पॉइंट पाँच ट्रिलियन अमेरिकी डॉलर है।'

---

source

PunjabiNormalizer

def PunjabiNormalizer(
    lang:str='pa', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, do_canonicalize_addak:bool=False, do_canonicalize_tippi:bool=False,
    do_replace_vowel_bases:bool=False, tts_mode:bool=False
):

Normalizer for the Gurmukhi script. In addition to basic normalization by the super class, * Replaces the composite characters containing nuktas by their decomposed form * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script

normalizer = PunjabiNormalizer(tts_mode=True)
punjabi_text = "ਪੰਜੀਵੀ ਜੀਡੀਪੀ 1.5 ਟ੍ਰੀਲੀਅਨ ਅਮਰੀਕਾ ਡੋਲਰ ਛੇ।"
normalizer(punjabi_text)

'ਪੰਜੀਵੀ ਜੀਡੀਪੀ ਇੱਕ ਪੌਇੰਟ ਪੰਜ ਟ੍ਰੀਲੀਅਨ ਅਮਰੀਕਾ ਡੋਲਰ ਛੇ।'

punjabi_text = (
    "ਸਤਿ ਸ੍ਰੀ ਅਕਾਲ ਦੁਨੀਆਂ। 4 ਮਈ, 2025 ਨੂੰ ਰਿਲੀਜ਼ ਹੋਏ ਪੰਜਾਬੀ ਨੋਰਮਲਾਈਜ਼ਰ ਵਿੱਚ ਤੁਹਾਡਾ ਸਵਾਗਤ ਹੈ।"
)
normalizer(punjabi_text)

'ਸਤਿ ਸ੍ਰੀ ਅਕਾਲ ਦੁਨੀਆਂ। ਚਾਰ ਮਈ, ਦੋ ਹਜ਼ਾਰ ਪੱਚੀ ਨੂੰ ਰਿਲੀਜ਼ ਹੋਏ ਪੰਜਾਬੀ ਨੋਰਮਲਾਈਜ਼ਰ ਵਿੱਚ ਤੁਹਾਡਾ ਸਵਾਗਤ ਹੈ।'

---

source

TeluguNormalizer

def TeluguNormalizer(
    lang:str='te', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, tts_mode:bool=False
):

Normalizer for the Teluguscript. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script

norm = TeluguNormalizer()
te_text = "భారతదేశ జీడీపీ 1.5 ట్రిలియన్ అమెరికా డాలర్లు."
norm(te_text)

'భారతదేశ జీడీపీ ఒకటి.ఐదు ట్రిలియన్ అమెరికా డాలర్లు.'

norm = TeluguNormalizer(tts_mode=True)
norm(te_text)

'భారతదేశ జీడీపీ ఒకటి పాయింట్ ఐదు ట్రిలియన్ అమెరికా డాలర్లు.'

---

source

GujaratiNormalizer

def GujaratiNormalizer(
    lang:str='gu', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, tts_mode:bool=False
):

Normalizer for the Gujarati script. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * replace colon ‘:’ by visarga if the colon follows a charcter in this script

norm = GujaratiNormalizer(tts_mode=True)
gujarati_text = "ભારતનો જીડીપી 1.5 ટ્રિલિયન અમેરિકન ડોલર છે."
norm(gujarati_text)

'ભારતનો જીડીપી એક પોઈન્ટ પાંચ ટ્રિલિયન અમેરિકન ડોલર છે.'

---

source

OdiaNormalizer

def OdiaNormalizer(
    lang:str='or', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, do_remap_wa:bool=False, tts_mode:bool=False
):

Normalizer for the Oriya script. In addition to basic normalization by the super class, * Replaces the composite characters containing nuktas by their decomposed form * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * Canonicalize two part dependent vowels * Replace ‘va’ with ‘ba’ * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script

norm = OdiaNormalizer(tts_mode=True)
odia_text = "ଭାରତର ଜିଡିପି 1.5 ଟ୍ରିଲିଅନ୍ ଆମେରିକୀୟ ଡଲାର ଅଟେ।"
norm(odia_text)

'ଭାରତର ଜିଡିପି ଏକ ପଏଣ୍ଟ ପାଞ୍ଚ ଟ୍ରିଲିଅନ୍ ଆମେରିକୀୟ ଡଲାର ଅଟେ।'

---

source

BengaliNormalizer

def BengaliNormalizer(
    lang:str='bn', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, do_remap_assamese_chars:bool=False, tts_mode:bool=False
):

Normalizer for the Bengali script. In addition to basic normalization by the super class, * Replaces the composite characters containing nuktas by their decomposed form * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * Canonicalize two part dependent vowels * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script

norm = BengaliNormalizer(tts_mode=True)

TEST_RESULT = "ভারতের জিডিপি ১.৫ ট্রিলিয়ন মার্কিন ডলার।"  # Claude generated output
bn_text = "ভারতের জিডিপি 1.5 ট্রিলিয়ন মার্কিন ডলার।"
norm(bn_text)

'ভারতের জিডিপি এক পয়েন্ট পাঁচ ট্রিলিয়ন মার্কিন ডলার।'

# Regression test: tts_mode=False must not apply TTS-only substitutions.
# BengaliNormalizer used to ignore the tts_mode constructor argument and always
# behaved as though tts_mode=True; this pins the fix.
default_norm = BengaliNormalizer()
assert default_norm.tts_mode is False
default_result = default_norm("Price is $100")
assert "$" in default_result and "ডলার" not in default_result

tts_result = BengaliNormalizer(tts_mode=True)("Price is $100")
assert "ডলার" in tts_result and "$" not in tts_result

---

source

TamilNormalizer

def TamilNormalizer(
    lang:str='ta', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, tts_mode:bool=False
):

Normalizer for the Tamil script. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script

norm = TamilNormalizer(tts_mode=True)
ta_text = "இந்தியாவின் மொத்த உள்நாட்டு உற்பத்தி 1.5 டிரில்லியன் அமெரிக்க டாலர்."
norm(ta_text)

'இந்தியாவின் மொத்த உள்நாட்டு உற்பத்தி ஒன்று பாயிண்ட் ஐந்து டிரில்லியன் அமெரிக்க டாலர்.'

norm = TamilNormalizer(tts_mode=True)
ta_text = "அவர் Rs. 500 மற்றும் $20 க்கு உணவுப்பொருட்கள் வாங்கினார். இணையதளம்: www.amazon.in."
norm(ta_text)

'அவர் ரூபாய் ஐநூறு மற்றும் டாலர் இருபது க்கு உணவுப்பொருட்கள் வாங்கினார். இணையதளம்ஃ w w w டாட் a m a z o n டாட் i n.'

---

source

KannadaNormalizer

def KannadaNormalizer(
    lang:str='kn', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, tts_mode:bool=False
):

Normalizer for the Kannada script. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script

norm = KannadaNormalizer(tts_mode=True)
kannada_text = "ಭಾರತದ ಜಿಡಿಪಿ 1.5 ಟ್ರಿಲಿಯನ್ ಅಮೇರಿಕನ್ ಡಾಲರ್ ಆಗಿದೆ."
norm(kannada_text)

'ಭಾರತದ ಜಿಡಿಪಿ ಒಂದು ಪಾಯಿಂಟ್ ಐದು ಟ್ರಿಲಿಯನ್ ಅಮೇರಿಕನ್ ಡಾಲರ್ ಆಗಿದೆ.'

---

source

MalayalamNormalizer

def MalayalamNormalizer(
    lang:str='ml', remove_nuktas:bool=False, nasals_mode:str='do_nothing', do_normalize_chandras:bool=False,
    do_normalize_vowel_ending:bool=False, do_canonicalize_chillus:bool=False, do_correct_geminated_T:bool=False,
    tts_mode:bool=False
):

Normalizer for the Malayalam script. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * Change from old encoding of chillus (till Unicode 5.0) to new encoding * replace colon ‘:’ by visarga if the colon follows a charcter in this script

Tests

normalizer = MalayalamNormalizer()

TEST_RESULT = "എന്റെ കമ്പ്യൂട്ടറിന് എന്റെ ഭാഷ."
text_result = normalizer("എന്റെ കമ്പ്യൂട്ടറിനു് എന്റെ ഭാഷ.")

assert text_result == TEST_RESULT

TESTCASE_RESULT = "യുപിഎ ഭരണകാലത്തെ സാമ്പത്തിക വീഴ്ച; ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ.\n\nയുപിഎ സർക്കാരിന്റെ കാലത്തെ ധനവിനിയോഗത്തിലെ വീഴ്ചകൾ വ്യക്തമാക്കുന്ന ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ തീരുമാനം. ബജറ്റ് സമ്മേളനം ഇതിനായി ഒരു ദിവസം കൂടി നീട്ടും. വിഹിതങ്ങൾ എപ്രകാരം തെറ്റായി വിനിയോഗിക്കപ്പെട്ടു എന്നതുൾപ്പെടെയുള്ള കാര്യങ്ങൾ വിശദീകരിക്കാനാണ് കേന്ദ്രസർക്കാർ നീക്കം.\n\n"
text_result = normalizer(
    """യുപിഎ ഭരണകാലത്തെ സാമ്പത്തിക വീഴ്ച; ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ.

യുപിഎ സർക്കാരിന്റെ കാലത്തെ ധനവിനിയോഗത്തിലെ വീഴ്ചകൾ വ്യക്തമാക്കുന്ന ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ തീരുമാനം. ബജറ്റ് സമ്മേളനം ഇതിനായി ഒരു ദിവസം കൂടി നീട്ടും. വിഹിതങ്ങൾ എപ്രകാരം തെറ്റായി വിനിയോഗിക്കപ്പെട്ടു എന്നതുൾപ്പെടെയുള്ള കാര്യങ്ങൾ വിശദീകരിക്കാനാണ് കേന്ദ്രസർക്കാർ നീക്കം.

"""
)
text_result

'യുപിഎ ഭരണകാലത്തെ സാമ്പത്തിക വീഴ്ച; ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ.\n\nയുപിഎ സർക്കാരിന്റെ കാലത്തെ ധനവിനിയോഗത്തിലെ വീഴ്ചകൾ വ്യക്തമാക്കുന്ന ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ തീരുമാനം. ബജറ്റ് സമ്മേളനം ഇതിനായി ഒരു ദിവസം കൂടി നീട്ടും. വിഹിതങ്ങൾ എപ്രകാരം തെറ്റായി വിനിയോഗിക്കപ്പെട്ടു എന്നതുൾപ്പെടെയുള്ള കാര്യങ്ങൾ വിശദീകരിക്കാനാണ് കേന്ദ്രസർക്കാർ നീക്കം.\n\n'

assert text_result == TESTCASE_RESULT

normalizer = MalayalamNormalizer(tts_mode=True)
malaylam_text = "ഇന്ത്യയുടെ ജിഡിപി 1.5 ട്രില്യൺ യുഎസ് ഡോളറാണ്."
TESTCASE_RESULT = "ഇന്ത്യയുടെ ജിഡിപി ഒന്ന് പോയിന്റ് അഞ്ച് ട്രില്യൺ യുഎസ് ഡോളറാണ്."
text_result = normalizer(malaylam_text)
text_result

'ഇന്ത്യയുടെ ജിഡിപി ഒന്ന് പോയിന്റ് അഞ്ച് ട്രില്യൺ യുഎസ് ഡോളറാണ്.'

assert text_result == TESTCASE_RESULT

TESTCASE_RESULT = "ദുഃഖം"
text_result = normalizer("ദു:ഖം")
text_result

'ദുഃഖം'

assert text_result == TESTCASE_RESULT

TESTCASE_RESULT = "എന്റെ"
text_result = normalizer("എൻറെ")
text_result

# Still fails

'എൻറെ'

normalizer(
    "1000 രൂപ കൊടുത്തു. അയാൾ 500 ഡോളറിനും 20 ഡോളറിനും പലചരക്ക് സാധനങ്ങൾ വാങ്ങി. വെബ്സൈറ്റ്: www.amazon.in."
)

'ആയിരം രൂപ കൊടുത്തു. അയാൾ അഞ്ഞൂറ് ഡോളറിനും ഇരുപത് ഡോളറിനും പലചരക്ക് സാധനങ്ങൾ വാങ്ങി. വെബ്സൈറ്റ്ഃ w w w ഡോട്ട് a m a z o n ഡോട്ട് i n.'

# TESTCASE_RESULT  = "കാണ്മാനില്ല"
# text_result = normalizer("കാണ്മാനില്ല")
# text_result

# Still fails

Behaviour tests: shared TTS pipeline across Indic normalizers

These tests drive the deepening of the Indic normalizers (one shared TTS/digit pipeline, per-language data). They only use the public interface: XNormalizer(tts_mode=...)(text) -> str. Expected values are literals or come from indic_numtowords (an independent oracle for numerals), never from recomputing the normalizer’s own logic.

# Per-language spoken words (literals) used as the expected values in the tests below.
INDIC_TTS = {
    "hi": dict(cls=HindiNormalizer, rupee="रुपये", dollar="डॉलर", won="कोरियाई वॉन", dot="डॉट", at="एट"),
    "pa": dict(cls=PunjabiNormalizer, rupee="ਰੁਪਏ", dollar="ਡਾਲਰ", won="ਕੋਰੀਆਈ ਵੌਨ", dot="ਡਾਟ", at="ਐਟ"),
    "te": dict(cls=TeluguNormalizer, rupee="రూపాయలు", dollar="డాలర్లు", won="కొరియన్ వాన్", dot="డాట్", at="అట్"),
    "gu": dict(cls=GujaratiNormalizer, rupee="રૂપિયા", dollar="ડોલર", won="કોરિયન વોન", dot="ડોટ", at="એટ"),
    "or": dict(cls=OdiaNormalizer, rupee="ଟଙ୍କା", dollar="ଡଲାର", won="କୋରିଆନ୍ ୱନ୍", dot="ଡଟ୍", at="ଏଟ୍"),
    "bn": dict(cls=BengaliNormalizer, rupee="রুপি", dollar="ডলার", won="কোরিয়ান ওন", dot="ডট", at="অ্যাট"),
    "ta": dict(cls=TamilNormalizer, rupee="ரூபாய்", dollar="டாலர்", won="கொரிய வான்", dot="டாட்", at="அட்"),
    "kn": dict(cls=KannadaNormalizer, rupee="ರೂಪಾಯಿ", dollar="ಡಾಲರ್", won="ಕೊರಿಯನ್ ವಾನ್", dot="ಡಾಟ್", at="ಅಟ್"),
    "ml": dict(cls=MalayalamNormalizer, rupee="രൂപ", dollar="ഡോളർ", won="കൊറിയൻ വോൺ", dot="ഡോട്ട്", at="അറ്റ്"),
}


def spoken(code, number):
    return num2words(str(number), lang=code)

# tts_mode: a dollar sign before a number is spoken as the language's word for dollar.
bad = {}
for code, spec in INDIC_TTS.items():
    out = spec["cls"](tts_mode=True)("$20")
    expected = f"{spec['dollar']} {spoken(code, 20)}"
    if out != expected:
        bad[code] = (out, expected)
assert not bad, bad

# A colon after a character of the script becomes that script's visarga; the character itself is kept.
# Unicode block bases are fixed by the standard: consonant KA = base+0x15, vowel sign I = base+0x3F, visarga = base+0x03.
BLOCK_BASE = {"hi": 0x0900, "bn": 0x0980, "pa": 0x0A00, "gu": 0x0A80, "or": 0x0B00,
              "ta": 0x0B80, "te": 0x0C00, "kn": 0x0C80, "ml": 0x0D00}
bad = {}
for code, spec in INDIC_TTS.items():
    base = BLOCK_BASE[code]
    word = chr(base + 0x15) + chr(base + 0x3F)
    out = spec["cls"]()(word + ":")
    expected = word + chr(base + 0x03)
    if out != expected:
        bad[code] = (out, expected)
assert not bad, bad

# A colon that does not follow a character of the script is left alone.
bad = {code: out for code, spec in INDIC_TTS.items() if (out := spec["cls"]()("ok:")) != "ok:"}
assert not bad, bad

# Punjabi: addak + consonant is canonicalized to consonant + halant + consonant (addak is U+0A71).
out = PunjabiNormalizer(do_canonicalize_addak=True)("\u0a05\u0a71\u0a17")
assert out == "\u0a05\u0a17\u0a4d\u0a17", out

# The rule is opt-in.
assert PunjabiNormalizer()("\u0a05\u0a71\u0a17") == "\u0a05\u0a71\u0a17"

# Punjabi: tippi (U+0A70) is canonicalized to bindi (U+0A02) when requested.
assert PunjabiNormalizer(do_canonicalize_tippi=True)("\u0a05\u0a70\u0a17") == "\u0a05\u0a02\u0a17"

POINT = {"hi": "पॉइंट", "pa": "ਪੌਇੰਟ", "te": "పాయింట్", "gu": "પોઈન્ટ", "or": "ପଏଣ୍ଟ",
         "bn": "পয়েন্ট", "ta": "பாயிண்ட்", "kn": "ಪಾಯಿಂಟ್", "ml": "പോയിന്റ്"}


def mismatches(make_case):
    """Run make_case(code, spec) -> (got_thunk, expected) for every language; return {code: got-or-exception} for failures."""
    bad = {}
    for code, spec in INDIC_TTS.items():
        got, expected = make_case(code, spec)
        try:
            value = got()
        except Exception as e:  # a crash is a failure to report, not to hide
            bad[code] = f"raised {type(e).__name__}: {e}"
            continue
        if value != expected:
            bad[code] = (value, expected)
    return bad

# Digits are spelled as words, and each number is replaced on its own: a "1" must not rewrite the "1" inside "10".
bad = mismatches(lambda code, spec: (lambda: spec["cls"]()("1 10"), f"{spoken(code, 1)} {spoken(code, 10)}"))
assert not bad, bad

# Default mode: a decimal number must not crash; each side of the dot is spelled and the dot is kept.
bad = mismatches(lambda code, spec: (lambda: spec["cls"]()("1.5"), f"{spoken(code, 1)}.{spoken(code, 5)}"))
assert not bad, bad

# tts_mode: a decimal is spoken as <whole> <point-word> <fraction>.
bad = mismatches(lambda code, spec: (lambda: spec["cls"](tts_mode=True)("1.5"), f"{spoken(code, 1)} {POINT[code]} {spoken(code, 5)}"))
assert not bad, bad


# Multi-digit fraction (existing behaviour, pinned): Hindi reads the fraction digit by digit; the other languages read it as one number.
def fraction_expected(code):
    frac = f"{spoken(code, 2)} {spoken(code, 5)}" if code == "hi" else spoken(code, 25)
    return f"{spoken(code, 1)} {POINT[code]} {frac}"


bad = mismatches(lambda code, spec: (lambda: spec["cls"](tts_mode=True)("1.25"), fraction_expected(code)))
assert not bad, bad

Pinning tests for the shared TTS pipeline (currency, URL, e-mail, symbols, acronyms, lowercase policy)

# tts_mode: every currency marker is spoken as the language's currency word followed by the amount in words.
CURRENCY_CASES = [("Rs. 500", "rupee", 500), ("INR 7", "rupee", 7), ("₹100", "rupee", 100),
                  ("USD 3", "dollar", 3), ("$20", "dollar", 20), ("KRW 9", "won", 9), ("₩9", "won", 9)]
bad = {}
for text, word, amount in CURRENCY_CASES:
    for code, spec in INDIC_TTS.items():
        out = spec["cls"](tts_mode=True)(text)
        expected = f"{spec[word]} {spoken(code, amount)}"
        if out != expected:
            bad[(text, code)] = (out, expected)
assert not bad, bad

# tts_mode: in an e-mail address the local part is kept, "@" is spoken in English, and the domain is spelled out with the language's "dot" word (Punjabi keeps Latin case).
bad = {}
for code, spec in INDIC_TTS.items():
    out = spec["cls"](tts_mode=True)("mail help@support.in")
    expected = ("mail help at s u p p o r t " + spec["dot"] + " i n") if code != "pa" else ("mail help at S U P P O R T " + spec["dot"] + " I N")
    if out != expected:
        bad[code] = (out, expected)
assert not bad, bad

# Golden outputs captured from the pre-refactor normalizers (after the bug fixes above, reviewed by hand).
# They pin URL / e-mail / symbol / acronym handling and each language's lowercase policy:
# Punjabi keeps Latin case, Hindi lowercases in tts_mode only, the rest always lowercase.
GOLDEN_TTS = {'Visit https://example.com/a/b now': {'hi': 'visit एच टी टी पी एस कोलन स्लैश स्लैश e x a m p l e '
                                             'डॉट c o m स्लैश a स्लैश b now',
                                       'pa': 'Visit ਐਚ ਟੀ ਟੀ ਪੀ ਐੱਸ ਕੋਲਨ ਸਲੈਸ਼ ਸਲੈਸ਼ E X A M P L E '
                                             'ਡਾਟ C O M ਸਲੈਸ਼ A ਸਲੈਸ਼ B now',
                                       'te': 'visit హెచ్ టి టి పి ఎస్ కొలన్ స్లాష్ స్లాష్ e x a m '
                                             'p l e డాట్ c o m స్లాష్ a స్లాష్ b now',
                                       'gu': 'visit એચ ટી ટી પીએસ કોલન સ્લેશ સ્લેશ e x a m p l e '
                                             'ડોટ c o m સ્લેશ a સ્લેશ b now',
                                       'or': 'visit ଏଚ୍ ଟି ଟି ପି ଏସ୍ କୋଲନ୍ ସ୍ଲାଶ୍ ସ୍ଲାଶ୍ e x a m p '
                                             'l e ଡଟ୍ c o m ସ୍ଲାଶ୍ a ସ୍ଲାଶ୍ b now',
                                       'bn': 'visit এইচ টি টি পি এস কোলন স্ল্যাশ স্ল্যাশ e x a m p '
                                             'l e ডট c o m স্ল্যাশ a স্ল্যাশ b now',
                                       'ta': 'visit எச் டி டி பி எஸ் கோலன் ஸ்லாஷ் ஸ்லாஷ் e x a m p '
                                             'l e டாட் c o m ஸ்லாஷ் a ஸ்லாஷ் b now',
                                       'kn': 'visit ಎಚ್ ಟಿ ಟಿ ಪಿ ಎಸ್ ಕೋಲನ್ ಸ್ಲ್ಯಾಶ್ ಸ್ಲ್ಯಾಶ್ e x a '
                                             'm p l e ಡಾಟ್ c o m ಸ್ಲ್ಯಾಶ್ a ಸ್ಲ್ಯಾಶ್ b now',
                                       'ml': 'visit എച്ച് ടി ടി പി എസ് കോളൺ സ്ലാഷ് സ്ലാഷ് e x a m '
                                             'p l e ഡോട്ട് c o m സ്ലാഷ് a സ്ലാഷ് b now'},
 'http://x.org': {'hi': 'एच टी टी पी कोलन स्लैश स्लैश x डॉट o r g',
                  'pa': 'ਐਚ ਟੀ ਟੀ ਪੀ ਕੋਲਨ ਸਲੈਸ਼ ਸਲੈਸ਼ X ਡਾਟ O R G',
                  'te': 'హెచ్ టి టి పి కొలన్ స్లాష్ స్లాష్ x డాట్ o r g',
                  'gu': 'એચ ટી ટી પી કોલન સ્લેશ સ્લેશ x ડોટ o r g',
                  'or': 'ଏଚ୍ ଟି ଟି ପି କୋଲନ୍ ସ୍ଲାଶ୍ ସ୍ଲାଶ୍ x ଡଟ୍ o r g',
                  'bn': 'এইচ টি টি পি কোলন স্ল্যাশ স্ল্যাশ x ডট o r g',
                  'ta': 'எச் டி டி பி கோலன் ஸ்லாஷ் ஸ்லாஷ் x டாட் o r g',
                  'kn': 'ಎಚ್ ಟಿ ಟಿ ಪಿ ಕೋಲನ್ ಸ್ಲ್ಯಾಶ್ ಸ್ಲ್ಯಾಶ್ x ಡಾಟ್ o r g',
                  'ml': 'എച്ച് ടി ടി പി കോളൺ സ്ലാഷ് സ്ലാഷ് x ഡോട്ട് o r g'},
 'see www.amazon.in today': {'hi': 'see w w w डॉट a m a z o n डॉट i n today',
                             'pa': 'see W W W ਡਾਟ A M A Z O N ਡਾਟ I N today',
                             'te': 'see w w w డాట్ a m a z o n డాట్ i n today',
                             'gu': 'see w w w ડોટ a m a z o n ડોટ i n today',
                             'or': 'see w w w ଡଟ୍ a m a z o n ଡଟ୍ i n today',
                             'bn': 'see w w w ডট a m a z o n ডট i n today',
                             'ta': 'see w w w டாட் a m a z o n டாட் i n today',
                             'kn': 'see w w w ಡಾಟ್ a m a z o n ಡಾಟ್ i n today',
                             'ml': 'see w w w ഡോട്ട് a m a z o n ഡോട്ട് i n today'},
 'mail help@support.in': {'hi': 'mail help at s u p p o r t डॉट i n',
                          'pa': 'mail help at S U P P O R T ਡਾਟ I N',
                          'te': 'mail help at s u p p o r t డాట్ i n',
                          'gu': 'mail help at s u p p o r t ડોટ i n',
                          'or': 'mail help at s u p p o r t ଡଟ୍ i n',
                          'bn': 'mail help at s u p p o r t ডট i n',
                          'ta': 'mail help at s u p p o r t டாட் i n',
                          'kn': 'mail help at s u p p o r t ಡಾಟ್ i n',
                          'ml': 'mail help at s u p p o r t ഡോട്ട് i n'},
 'a & b 50% off + 2 = 4 * 3 # 1 - 2': {'hi': 'a and b पचास percent off plus दो equals चार star तीन '
                                             'hash एक minus दो',
                                       'pa': 'a and b ਪੰਜਾਹ percent off plus ਦੋ equals ਚਾਰ star '
                                             'ਤਿੰਨ hash ਇੱਕ minus ਦੋ',
                                       'te': 'a and b యాభై percent off plus రెండు equals నాలుగు '
                                             'star మూడు hash ఒకటి minus రెండు',
                                       'gu': 'a and b પચાસ percent off plus બે equals ચાર star '
                                             'ત્રણ hash એક minus બે',
                                       'or': 'a and b ପଚାଶ percent off plus ଦୁଇ equals ଚାରି star '
                                             'ତିନି hash ଏକ minus ଦୁଇ',
                                       'bn': 'a and b পঞ্চাশ percent off plus দুই equals চার star '
                                             'তিন hash এক minus দুই',
                                       'ta': 'a and b ஐம்பது percent off plus இரண்டு equals நான்கு '
                                             'star மூன்று hash ஒன்று minus இரண்டு',
                                       'kn': 'a and b ಐವತ್ತು percent off plus ಎರಡು equals ನಾಲ್ಕು '
                                             'star ಮೂರು hash ಒಂದು minus ಎರಡು',
                                       'ml': 'a and b അമ്പത് percent off plus രണ്ട് equals നാല് '
                                             'star മൂന്ന് hash ഒന്ന് minus രണ്ട്'},
 'USA and IBM': {'hi': 'u s a and i b m',
                 'pa': 'U S A and I B M',
                 'te': 'u s a and i b m',
                 'gu': 'u s a and i b m',
                 'or': 'u s a and i b m',
                 'bn': 'u s a and i b m',
                 'ta': 'u s a and i b m',
                 'kn': 'u s a and i b m',
                 'ml': 'u s a and i b m'},
 'Rs. 500 INR 7 ₹100 USD 3 $20 KRW 9 ₩9': {'hi': 'रुपये पाँच सौ रुपये सात रुपये एक सौ डॉलर तीन '
                                                 'डॉलर बीस कोरियाई वॉन नौ कोरियाई वॉन नौ',
                                           'pa': 'ਰੁਪਏ ਪੰਜ ਸੌ ਰੁਪਏ ਸੱਤ ਰੁਪਏ ਸੌ ਡਾਲਰ ਤਿੰਨ ਡਾਲਰ ਵੀਹ '
                                                 'ਕੋਰੀਆਈ ਵੌਨ ਨੌਂ ਕੋਰੀਆਈ ਵੌਨ ਨੌਂ',
                                           'te': 'రూపాయలు ఐదు వందలు రూపాయలు ఏడు రూపాయలు వంద '
                                                 'డాలర్లు మూడు డాలర్లు ఇరవై కొరియన్ వాన్ తొమ్మిది '
                                                 'కొరియన్ వాన్ తొమ్మిది',
                                           'gu': 'રૂપિયા પાંચ સો રૂપિયા સાત રૂપિયા એક સો ડોલર ત્રણ '
                                                 'ડોલર વીસ કોરિયન વોન નવ કોરિયન વોન નવ',
                                           'or': 'ଟଙ୍କା ପାଞ୍ଚ ଶହ ଟଙ୍କା ସାତ ଟଙ୍କା ଶହେ ଡଲାର ତିନି '
                                                 'ଡଲାର କୋଡିଏ କୋରିଆନ୍ ୱନ୍ ନଅ କୋରିଆନ୍ ୱନ୍ ନଅ',
                                           'bn': 'রুপি পাঁচশো রুপি সাত রুপি একশো ডলার তিন ডলার বিশ '
                                                 'কোরিয়ান ওন নয় কোরিয়ান ওন নয়',
                                           'ta': 'ரூபாய் ஐநூறு ரூபாய் ஏழு ரூபாய் நூறு டாலர் மூன்று '
                                                 'டாலர் இருபது கொரிய வான் ஒன்பது கொரிய வான் ஒன்பது',
                                           'kn': 'ರೂಪಾಯಿ ಐನೂರು ರೂಪಾಯಿ ಏಳು ರೂಪಾಯಿ ನೂರು ಡಾಲರ್ ಮೂರು '
                                                 'ಡಾಲರ್ ಇಪ್ಪತ್ತು ಕೊರಿಯನ್ ವಾನ್ ಒಂಬತ್ತು ಕೊರಿಯನ್ ವಾನ್ '
                                                 'ಒಂಬತ್ತು',
                                           'ml': 'രൂപ അഞ്ഞൂറ് രൂപ ഏഴ് രൂപ നൂറ് ഡോളർ മൂന്ന് ഡോളർ '
                                                 'ഇരുപത് കൊറിയൻ വോൺ ഒമ്പത് കൊറിയൻ വോൺ ഒമ്പത്'}}

bad = {}
for text, expected in GOLDEN_TTS.items():
    for code, spec in INDIC_TTS.items():
        out = spec["cls"](tts_mode=True)(text)
        if out != expected[code]:
            bad[(text, code)] = (out, expected[code])
assert not bad, bad

GOLDEN_PLAIN = {'Hello World 5 USA': {'hi': 'Hello World पाँच USA',
                       'pa': 'Hello World ਪੰਜ USA',
                       'te': 'hello world ఐదు usa',
                       'gu': 'hello world પાંચ usa',
                       'or': 'hello world ପାଞ୍ଚ usa',
                       'bn': 'hello world পাঁচ usa',
                       'ta': 'hello world ஐந்து usa',
                       'kn': 'hello world ಐದು usa',
                       'ml': 'hello world അഞ്ച് usa'},
 'Rs. 500 INR 7 ₹100 USD 3 $20 KRW 9 ₩9': {'hi': 'Rs. पाँच सौ INR सात ₹एक सौ USD तीन $बीस KRW नौ '
                                                 '₩नौ',
                                           'pa': 'Rs. ਪੰਜ ਸੌ INR ਸੱਤ ₹ਸੌ USD ਤਿੰਨ $ਵੀਹ KRW ਨੌਂ '
                                                 '₩ਨੌਂ',
                                           'te': 'rs. ఐదు వందలు inr ఏడు ₹వంద usd మూడు $ఇరవై krw '
                                                 'తొమ్మిది ₩తొమ్మిది',
                                           'gu': 'rs. પાંચ સો inr સાત ₹એક સો usd ત્રણ $વીસ krw નવ '
                                                 '₩નવ',
                                           'or': 'rs. ପାଞ୍ଚ ଶହ inr ସାତ ₹ଶହେ usd ତିନି $କୋଡିଏ krw ନଅ '
                                                 '₩ନଅ',
                                           'bn': 'rs. পাঁচশো inr সাত ₹একশো usd তিন $বিশ krw নয় '
                                                 '₩নয়',
                                           'ta': 'rs. ஐநூறு inr ஏழு ₹நூறு usd மூன்று $இருபது krw '
                                                 'ஒன்பது ₩ஒன்பது',
                                           'kn': 'rs. ಐನೂರು inr ಏಳು ₹ನೂರು usd ಮೂರು $ಇಪ್ಪತ್ತು krw '
                                                 'ಒಂಬತ್ತು ₩ಒಂಬತ್ತು',
                                           'ml': 'rs. അഞ്ഞൂറ് inr ഏഴ് ₹നൂറ് usd മൂന്ന് $ഇരുപത് krw '
                                                 'ഒമ്പത് ₩ഒമ്പത്'},
 'see www.amazon.in a & b': {'hi': 'see www.amazon.in a & b',
                             'pa': 'see www.amazon.in a & b',
                             'te': 'see www.amazon.in a & b',
                             'gu': 'see www.amazon.in a & b',
                             'or': 'see www.amazon.in a & b',
                             'bn': 'see www.amazon.in a & b',
                             'ta': 'see www.amazon.in a & b',
                             'kn': 'see www.amazon.in a & b',
                             'ml': 'see www.amazon.in a & b'}}

# Default mode (tts_mode=False): no spoken-form rewrites, but digits are still spelled and the lowercase policy applies.
bad = {}
for text, expected in GOLDEN_PLAIN.items():
    for code, spec in INDIC_TTS.items():
        out = spec["cls"]()(text)
        if out != expected[code]:
            bad[(text, code)] = (out, expected[code])
assert not bad, bad

# Script-sharing languages spell digits in their own language when indic_numtowords knows it, else in the script's main language.
assert DevanagariNormalizer(lang="mr")("5 10") == f"{spoken('mr', 5)} {spoken('mr', 10)}"
assert DevanagariNormalizer(lang="ne")("5 10") == f"{spoken('hi', 5)} {spoken('hi', 10)}"  # no Nepali number words: Hindi is used
assert BengaliNormalizer(lang="as")("5 10") == f"{spoken('as', 5)} {spoken('as', 10)}"

# tts_mode on an unsupported Devanagari language is rejected up front.
try:
    DevanagariNormalizer(lang="ne", tts_mode=True)
    raise AssertionError("expected ValueError")
except ValueError:
    pass

# The visarga rule also holds for the languages that reuse a script's normalizer (Marathi, Nepali, Assamese).
for make, base in [(lambda: DevanagariNormalizer(lang="mr"), 0x0900), (lambda: DevanagariNormalizer(lang="ne"), 0x0900),
                   (lambda: BengaliNormalizer(lang="as"), 0x0980)]:
    word = chr(base + 0x15) + chr(base + 0x3F)
    out = make()(word + ":")
    assert out == word + chr(base + 0x03), (hex(base), out)

# tts_mode: "&" is spoken as "and", and symbol words are separated by single spaces.
bad = mismatches(lambda code, spec: (lambda: spec["cls"](tts_mode=True)("20% & 5"), f"{spoken(code, 20)} percent and {spoken(code, 5)}"))
assert not bad, bad

bad = mismatches(lambda code, spec: (lambda: spec["cls"](tts_mode=True)("a & b + 2 = 4"), f"a and b plus {spoken(code, 2)} equals {spoken(code, 4)}"))
assert not bad, bad

# Devanagari (Marathi) already spoke "&" as "and"; it stays that way.
assert DevanagariNormalizer(lang="mr", tts_mode=True)("a & b") == "a and b"

# Numbers written in a script's own digits are spelled like the same ASCII number (the native digit zero is base + 0x66).
NATIVE_DIGIT_CASES = [
    ("hi", HindiNormalizer(), 0x0900), ("mr", DevanagariNormalizer(lang="mr"), 0x0900),
    ("ne", DevanagariNormalizer(lang="ne"), 0x0900), ("bn", BengaliNormalizer(), 0x0980),
    ("as", BengaliNormalizer(lang="as"), 0x0980), ("pa", PunjabiNormalizer(), 0x0A00),
    ("gu", GujaratiNormalizer(), 0x0A80), ("or", OdiaNormalizer(), 0x0B00), ("ta", TamilNormalizer(), 0x0B80),
    ("te", TeluguNormalizer(), 0x0C00), ("kn", KannadaNormalizer(), 0x0C80), ("ml", MalayalamNormalizer(), 0x0D00),
]
bad = {}
for code, normalizer, base in NATIVE_DIGIT_CASES:
    native_50 = chr(base + 0x66 + 5) + chr(base + 0x66)
    try:
        out = normalizer(native_50)
    except Exception as e:  # a crash is the failure being reported
        bad[code] = f"raised {type(e).__name__}: {e}"
        continue
    expected = normalizer("50")
    if out != expected or any(ch.isdigit() for ch in out):
        bad[code] = (out, expected)
assert not bad, bad