from whisper_normalizer.basic import BasicTextNormalizer
= BasicTextNormalizer()
normalizer "I'm a little teapot, short and stout. Tip me over and pour me out!") normalizer(
'i m a little teapot short and stout tip me over and pour me out '
A python package for text standardisation/normalization. It uses normalization algorithm mentioned in OpenAI whisper paper. Using Whisper normalization can cause issues in Indic languages and other low resource languages when using
BasicTextNormalizer
. So normalization in Indic languages is also implemented in this package which was derived from indic-nlp-library.
This package is a python implementation of the text standardisation/normalization approach which is being used in OpenAI whisper. The code was originally being released as open-source in Whisper source code. More details about the text normalization approach used by whisper can be found on Appendix Section C pp.21 the paper Robust Speech Recognition via Large-Scale Weak Supervision by OpenAI team.
or from github repository
This package is a python implementation of the text standardisation/normalization approach which is being used in OpenAI whisper text normalizer. If you want to use just text normalization alone, it’s better to use this instead reimplementing the same thing. OpenAI approach of text normalization is very helpful and is being used as normalization step when evaluating competitive models like AssemblyAI Conformer-1 model.
OpenAI open source approach of text normalization/standardization is mentioned in detail Appendix Section C pp.21 the paper Robust Speech Recognition via Large-Scale Weak Supervision.
Whisper Normalizer by default comes with two classes BasicTextNormalizer
and EnglishTextNormalizer
You can use the same thing in this package as follows:
from whisper_normalizer.basic import BasicTextNormalizer
normalizer = BasicTextNormalizer()
normalizer("I'm a little teapot, short and stout. Tip me over and pour me out!")
'i m a little teapot short and stout tip me over and pour me out '
from whisper_normalizer.english import EnglishTextNormalizer
english_normalizer = EnglishTextNormalizer()
english_normalizer("I'm a little teapot, short and stout. Tip me over and pour me out!")
'i am a little teapot short and stout tip me over and pour me out'
The logic for normalization in Indic languages is derived from indic-nlp-library. The logic for Malayalam normalization is expanded beyond the Indic NLP library by MalayalamNormalizer
.