Indic Languages Normalizer

This code is from: https://github.com/anoopkunchukuttan/indic_nlp_library

This code has been modified by Kurian to suit to Whisper-normalizer style of coding and the logic for Malayalam normalization is expanded beyond the Indic NLP library by Dr Kavya.

source

NormalizerI

 NormalizerI ()

The normalizer classes do the following: Some characters have multiple Unicode codepoints. The normalizer chooses a single standard representation * Some control characters are deleted * While typing using the Latin keyboard, certain typical mistakes occur which are corrected by the module Base class for normalizer. Performs some common normalization, which includes: * Byte order mark, word joiner, etc. removal * ZERO_WIDTH_NON_JOINER and ZERO_WIDTH_JOINER removal * ZERO_WIDTH_SPACE and NO_BREAK_SPACE replaced by spaces Script specific normalizers should derive from this class and override the normalize() method. They can call the super class ’normalize() method to avail of the common normalization*

source

BaseNormalizer

 BaseNormalizer (lang, remove_nuktas=False, nasals_mode='do_nothing',
                 do_normalize_chandras=False,
                 do_normalize_vowel_ending=False)

Common class used in most of indic languages inherit from this code.

source

DevanagariNormalizer

 DevanagariNormalizer (lang='hi', remove_nuktas=False,
                       nasals_mode='do_nothing',
                       do_normalize_chandras=False,
                       do_normalize_vowel_ending=False)

Normalizer for the Devanagari script. In addition to basic normalization by the super class, Replaces the composite characters containing nuktas by their decomposed form * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

norm = DevanagariNormalizer()
TEST_RESULT = 'चीर धाराएं समुद्र तट पर लगकर लौटने वाली लहरों का प्रवाह होता है अक्सर एक चट्टान या इसी तरह के पदार्थों पर'
hi_text = "चीर धाराएं समुद्र तट पर लगकर लौटने वाली लहरों का प्रवाह होता है अक्सर एक चट्टान या इसी तरह के पदार्थों पर"
norm(hi_text)
'चीर धाराएं समुद्र तट पर लगकर लौटने वाली लहरों का प्रवाह होता है अक्सर एक चट्टान या इसी तरह के पदार्थों पर'
assert norm(hi_text)==TEST_RESULT

source

HindiNormalizer

 HindiNormalizer (lang='hi', remove_nuktas=False,
                  nasals_mode='do_nothing', do_normalize_chandras=False,
                  do_normalize_vowel_ending=False)

Duplicate of Devanagiri normalizer.

source

GurmukhiNormalizer

 GurmukhiNormalizer (lang='pa', remove_nuktas=False,
                     nasals_mode='do_nothing',
                     do_normalize_chandras=False,
                     do_normalize_vowel_ending=False,
                     do_canonicalize_addak=False,
                     do_canonicalize_tippi=False,
                     do_replace_vowel_bases=False)

Normalizer for the Gurmukhi script. In addition to basic normalization by the super class, Replaces the composite characters containing nuktas by their decomposed form * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

source

TeluguNormalizer

 TeluguNormalizer (lang='te', remove_nuktas=False,
                   nasals_mode='do_nothing', do_normalize_chandras=False,
                   do_normalize_vowel_ending=False)

Normalizer for the Teluguscript. In addition to basic normalization by the super class, Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

source

GujaratiNormalizer

 GujaratiNormalizer (lang='gu', remove_nuktas=False,
                     nasals_mode='do_nothing',
                     do_normalize_chandras=False,
                     do_normalize_vowel_ending=False)

Normalizer for the Gujarati script. In addition to basic normalization by the super class, Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

source

OriyaNormalizer

 OriyaNormalizer (lang='or', remove_nuktas=False,
                  nasals_mode='do_nothing', do_normalize_chandras=False,
                  do_normalize_vowel_ending=False, do_remap_wa=False)

Normalizer for the Oriya script. In addition to basic normalization by the super class, Replaces the composite characters containing nuktas by their decomposed form * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * Canonicalize two part dependent vowels * Replace ‘va’ with ‘ba’ * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

source

BengaliNormalizer

 BengaliNormalizer (lang='bn', remove_nuktas=False,
                    nasals_mode='do_nothing', do_normalize_chandras=False,
                    do_normalize_vowel_ending=False,
                    do_remap_assamese_chars=False)

Normalizer for the Bengali script. In addition to basic normalization by the super class, Replaces the composite characters containing nuktas by their decomposed form * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * Canonicalize two part dependent vowels * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

source

TamilNormalizer

 TamilNormalizer (lang='ta', remove_nuktas=False,
                  nasals_mode='do_nothing', do_normalize_chandras=False,
                  do_normalize_vowel_ending=False)

Normalizer for the Tamil script. In addition to basic normalization by the super class, Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

source

TeluguNormalizer

 TeluguNormalizer (lang='te', remove_nuktas=False,
                   nasals_mode='do_nothing', do_normalize_chandras=False,
                   do_normalize_vowel_ending=False)

Normalizer for the Teluguscript. In addition to basic normalization by the super class, Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

source

KannadaNormalizer

 KannadaNormalizer (lang='kn', remove_nuktas=False,
                    nasals_mode='do_nothing', do_normalize_chandras=False,
                    do_normalize_vowel_ending=False)

Normalizer for the Kannada script. In addition to basic normalization by the super class, Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

source

UrduNormalizer

 UrduNormalizer (lang, remove_nuktas=True)

*Uses UrduHack library. https://docs.urduhack.com/en/stable/_modules/urduhack/normalization/character.html#normalize*

source

MalayalamNormalizer

 MalayalamNormalizer (lang='ml', remove_nuktas=False,
                      nasals_mode='do_nothing',
                      do_normalize_chandras=False,
                      do_normalize_vowel_ending=False,
                      do_canonicalize_chillus=False,
                      do_correct_geminated_T=False)

Normalizer for the Malayalam script. In addition to basic normalization by the super class, Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * Change from old encoding of chillus (till Unicode 5.0) to new encoding * replace colon ‘:’ by visarga if the colon follows a charcter in this script*

Tests

normalizer = MalayalamNormalizer()
TEST_RESULT = "എന്റെ കമ്പ്യൂട്ടറിന് എന്റെ ഭാഷ"
text_result = normalizer("എന്റെ കമ്പ്യൂട്ടറിനു് എന്റെ ഭാഷ")

assert text_result == TEST_RESULT
TESTCASE_RESULT='യുപിഎ ഭരണകാലത്തെ സാമ്പത്തിക വീഴ്ച; ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ.\n\nയുപിഎ സർക്കാരിന്റെ കാലത്തെ ധനവിനിയോഗത്തിലെ വീഴ്ചകൾ വ്യക്തമാക്കുന്ന ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ തീരുമാനം. ബജറ്റ് സമ്മേളനം ഇതിനായി ഒരു ദിവസം കൂടി നീട്ടും. വിഹിതങ്ങൾ എപ്രകാരം തെറ്റായി വിനിയോഗിക്കപ്പെട്ടു എന്നതുൾപ്പെടെയുള്ള കാര്യങ്ങൾ വിശദീകരിക്കാനാണ് കേന്ദ്രസർക്കാർ നീക്കം.\n\n'
text_result = normalizer("""യുപിഎ ഭരണകാലത്തെ സാമ്പത്തിക വീഴ്ച; ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ.

യുപിഎ സർക്കാരിന്റെ കാലത്തെ ധനവിനിയോഗത്തിലെ വീഴ്ചകൾ വ്യക്തമാക്കുന്ന ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ തീരുമാനം. ബജറ്റ് സമ്മേളനം ഇതിനായി ഒരു ദിവസം കൂടി നീട്ടും. വിഹിതങ്ങൾ എപ്രകാരം തെറ്റായി വിനിയോഗിക്കപ്പെട്ടു എന്നതുൾപ്പെടെയുള്ള കാര്യങ്ങൾ വിശദീകരിക്കാനാണ് കേന്ദ്രസർക്കാർ നീക്കം.

""")
text_result
'യുപിഎ ഭരണകാലത്തെ സാമ്പത്തിക വീഴ്ച; ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ.\n\nയുപിഎ സർക്കാരിന്റെ കാലത്തെ ധനവിനിയോഗത്തിലെ വീഴ്ചകൾ വ്യക്തമാക്കുന്ന ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ തീരുമാനം. ബജറ്റ് സമ്മേളനം ഇതിനായി ഒരു ദിവസം കൂടി നീട്ടും. വിഹിതങ്ങൾ എപ്രകാരം തെറ്റായി വിനിയോഗിക്കപ്പെട്ടു എന്നതുൾപ്പെടെയുള്ള കാര്യങ്ങൾ വിശദീകരിക്കാനാണ് കേന്ദ്രസർക്കാർ നീക്കം.\n\n'
assert text_result == TESTCASE_RESULT