Indic Languages Normalizer

This code is from: https://github.com/anoopkunchukuttan/indic_nlp_library

source

NormalizerI

 NormalizerI ()

The normalizer classes do the following: * Some characters have multiple Unicode codepoints. The normalizer chooses a single standard representation * Some control characters are deleted * While typing using the Latin keyboard, certain typical mistakes occur which are corrected by the module Base class for normalizer. Performs some common normalization, which includes: * Byte order mark, word joiner, etc. removal * ZERO_WIDTH_NON_JOINER and ZERO_WIDTH_JOINER removal * ZERO_WIDTH_SPACE and NO_BREAK_SPACE replaced by spaces Script specific normalizers should derive from this class and override the normalize() method. They can call the super class ’normalize() method to avail of the common normalization

source

BaseNormalizer

 BaseNormalizer (lang, remove_nuktas=False, nasals_mode='do_nothing',
                 do_normalize_chandras=False,
                 do_normalize_vowel_ending=False)

Common class used in most of indic languages inherit from this code.

source

DevanagariNormalizer

 DevanagariNormalizer (lang='hi', remove_nuktas=False,
                       nasals_mode='do_nothing',
                       do_normalize_chandras=False,
                       do_normalize_vowel_ending=False)

Normalizer for the Devanagari script. In addition to basic normalization by the super class, * Replaces the composite characters containing nuktas by their decomposed form * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script

source

GurmukhiNormalizer

 GurmukhiNormalizer (lang='pa', remove_nuktas=False,
                     nasals_mode='do_nothing',
                     do_normalize_chandras=False,
                     do_normalize_vowel_ending=False,
                     do_canonicalize_addak=False,
                     do_canonicalize_tippi=False,
                     do_replace_vowel_bases=False)

Normalizer for the Gurmukhi script. In addition to basic normalization by the super class, * Replaces the composite characters containing nuktas by their decomposed form * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script

source

TeluguNormalizer

 TeluguNormalizer (lang='te', remove_nuktas=False,
                   nasals_mode='do_nothing', do_normalize_chandras=False,
                   do_normalize_vowel_ending=False)

Normalizer for the Teluguscript. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script

source

GujaratiNormalizer

 GujaratiNormalizer (lang='gu', remove_nuktas=False,
                     nasals_mode='do_nothing',
                     do_normalize_chandras=False,
                     do_normalize_vowel_ending=False)

Normalizer for the Gujarati script. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * replace colon ‘:’ by visarga if the colon follows a charcter in this script

source

OriyaNormalizer

 OriyaNormalizer (lang='or', remove_nuktas=False,
                  nasals_mode='do_nothing', do_normalize_chandras=False,
                  do_normalize_vowel_ending=False, do_remap_wa=False)

Normalizer for the Oriya script. In addition to basic normalization by the super class, * Replaces the composite characters containing nuktas by their decomposed form * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * Canonicalize two part dependent vowels * Replace ‘va’ with ‘ba’ * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script

source

BengaliNormalizer

 BengaliNormalizer (lang='bn', remove_nuktas=False,
                    nasals_mode='do_nothing', do_normalize_chandras=False,
                    do_normalize_vowel_ending=False,
                    do_remap_assamese_chars=False)

Normalizer for the Bengali script. In addition to basic normalization by the super class, * Replaces the composite characters containing nuktas by their decomposed form * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * Canonicalize two part dependent vowels * replace pipe character ‘|’ by poorna virama character * replace colon ‘:’ by visarga if the colon follows a charcter in this script

source

TamilNormalizer

 TamilNormalizer (lang='ta', remove_nuktas=False,
                  nasals_mode='do_nothing', do_normalize_chandras=False,
                  do_normalize_vowel_ending=False)

Normalizer for the Tamil script. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script

source

TeluguNormalizer

 TeluguNormalizer (lang='te', remove_nuktas=False,
                   nasals_mode='do_nothing', do_normalize_chandras=False,
                   do_normalize_vowel_ending=False)

Normalizer for the Teluguscript. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script

source

KannadaNormalizer

 KannadaNormalizer (lang='kn', remove_nuktas=False,
                    nasals_mode='do_nothing', do_normalize_chandras=False,
                    do_normalize_vowel_ending=False)

Normalizer for the Kannada script. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * replace colon ‘:’ by visarga if the colon follows a charcter in this script

source

UrduNormalizer

 UrduNormalizer (lang, remove_nuktas=True)

Uses UrduHack library. https://docs.urduhack.com/en/stable/_modules/urduhack/normalization/character.html#normalize

source

IndicNormalizerFactory

 IndicNormalizerFactory ()

Factory class to create language specific normalizers.

source

MalayalamNormalizer

 MalayalamNormalizer (lang='ml', remove_nuktas=False,
                      nasals_mode='do_nothing',
                      do_normalize_chandras=False,
                      do_normalize_vowel_ending=False,
                      do_canonicalize_chillus=False,
                      do_correct_geminated_T=False)

Normalizer for the Malayalam script. In addition to basic normalization by the super class, * Replace the reserved character for poorna virama (if used) with the recommended generic Indic scripts poorna virama * canonicalize two-part dependent vowel signs * Change from old encoding of chillus (till Unicode 5.0) to new encoding * replace colon ‘:’ by visarga if the colon follows a charcter in this script

Tests

normalizer = MalayalamNormalizer()
TEST_RESULT = "എന്റെ കമ്പ്യൂട്ടറിന് എന്റെ ഭാഷ"
text_result = normalizer("എന്റെ കമ്പ്യൂട്ടറിനു് എന്റെ ഭാഷ")

assert text_result == TEST_RESULT
TESTCASE_RESULT='യുപിഎ ഭരണകാലത്തെ സാമ്പത്തിക വീഴ്ച; ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ.\n\nയുപിഎ സർക്കാരിന്റെ കാലത്തെ ധനവിനിയോഗത്തിലെ വീഴ്ചകൾ വ്യക്തമാക്കുന്ന ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ തീരുമാനം. ബജറ്റ് സമ്മേളനം ഇതിനായി ഒരു ദിവസം കൂടി നീട്ടും. വിഹിതങ്ങൾ എപ്രകാരം തെറ്റായി വിനിയോഗിക്കപ്പെട്ടു എന്നതുൾപ്പെടെയുള്ള കാര്യങ്ങൾ വിശദീകരിക്കാനാണ് കേന്ദ്രസർക്കാർ നീക്കം.\n\n'
text_result = normalizer("""യുപിഎ ഭരണകാലത്തെ സാമ്പത്തിക വീഴ്ച; ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ.

യുപിഎ സർക്കാരിന്റെ കാലത്തെ ധനവിനിയോഗത്തിലെ വീഴ്ചകൾ വ്യക്തമാക്കുന്ന ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ തീരുമാനം. ബജറ്റ് സമ്മേളനം ഇതിനായി ഒരു ദിവസം കൂടി നീട്ടും. വിഹിതങ്ങൾ എപ്രകാരം തെറ്റായി വിനിയോഗിക്കപ്പെട്ടു എന്നതുൾപ്പെടെയുള്ള കാര്യങ്ങൾ വിശദീകരിക്കാനാണ് കേന്ദ്രസർക്കാർ നീക്കം.

""")
text_result
'യുപിഎ ഭരണകാലത്തെ സാമ്പത്തിക വീഴ്ച; ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ.\n\nയുപിഎ സർക്കാരിന്റെ കാലത്തെ ധനവിനിയോഗത്തിലെ വീഴ്ചകൾ വ്യക്തമാക്കുന്ന ധവളപത്രം ഇറക്കാൻ കേന്ദ്രസർക്കാർ തീരുമാനം. ബജറ്റ് സമ്മേളനം ഇതിനായി ഒരു ദിവസം കൂടി നീട്ടും. വിഹിതങ്ങൾ എപ്രകാരം തെറ്റായി വിനിയോഗിക്കപ്പെട്ടു എന്നതുൾപ്പെടെയുള്ള കാര്യങ്ങൾ വിശദീകരിക്കാനാണ് കേന്ദ്രസർക്കാർ നീക്കം.\n\n'
assert text_result == TESTCASE_RESULT