WhisperNormalizer English Module

OpenAI’s English text standardisation module

What does this module do?

As per the text normalization/standardization approach Appendix Section C pp.21 the paper Robust Speech Recognition via Large-Scale Weak Supervision. The EnglishTextNormalizer does the following functionality:

Remove any phrases between matching brackets ([, ]).
Remove any phrases between matching parentheses ((, )).
Remove any of the following words: hmm, mm, mhm, mmm, uh, um
Remove whitespace characters that comes before an apostrophe ’
Convert standard or informal contracted forms of English into the original form.
Remove commas (,) between digits
Remove periods (.) not followed by numbers
Remove symbols as well as diacritics from the text, where symbols are the characters with the Unicode category starting with M, S, or P, except period, percent, and currency symbols that may be detected in the next step.
Detect any numeric expressions of numbers and currencies and replace with a form using Arabic numbers, e.g. “Ten thousand dollars” → “$10000”.
Convert British spellings into American spellings.
Remove remaining symbols that are not part of any numeric expressions.
Replace any successive whitespace characters with a space.

source

EnglishNumberNormalizer

def EnglishNumberNormalizer():

Convert any spelled-out numbers into arabic numbers, while handling:

remove any commas
keep the suffixes such as: 1960s, 274th, 32nd, etc.
spell out currency symbols after the number. e.g. $20 million -> 20000000 dollars
spell out one and ones
interpret successive single-digit numbers as nominal: one oh one -> 101

source

EnglishSpellingNormalizer

def EnglishSpellingNormalizer():

Applies British-American spelling mappings as listed in [1].

[1] https://web.archive.org/web/20230326222449/https://www.tysto.com/uk-us-spelling-list.html

n = EnglishSpellingNormalizer()
n("accessorise")

'accessorize'

source

EnglishTextNormalizer

def EnglishTextNormalizer(
    tts_mode:bool=False
):

Applies all the rules for normalizing English text as mentioned in OpenAI whisper paper. As per the text normalization/standardization approach Appendix Section C pp.21 the paper Robust Speech Recognition via Large-Scale Weak Supervision. The EnglishTextNormalizer does the following functionality:

Remove any phrases between matching brackets ([, ]).
Remove any phrases between matching parentheses ((, )).
Remove any of the following words: hmm, mm, mhm, mmm, uh, um
Remove whitespace characters that comes before an apostrophe ’
Convert standard or informal contracted forms of English into the original form.
Remove commas (,) between digits
Remove periods (.) not followed by numbers
Remove symbols as well as diacritics from the text, where symbols are the characters with the Unicode category starting with M, S, or P, except period, percent, and currency symbols that may be detected in the next step.
Detect any numeric expressions of numbers and currencies and replace with a form using Arabic numbers, e.g. “Ten thousand dollars” → “$10000”.
Convert British spellings into American spellings.
Remove remaining symbols that are not part of any numeric expressions.
Replace any successive whitespace characters with a space.

With tts_mode=True (requires pip install whisper_normalizer[tts]) the text is prepared for speech instead: step 9 is skipped so written numbers stay words, digits, currencies, percentages and decimals are spelled out, and &, +, = and @ are spoken as words.

Testing `EnglishTextNormalizer`

normalizer = EnglishTextNormalizer()
assert normalizer("I paid $20 million for 1,234 colour widgets.") == "i paid $20000000 for 1234 color widgets"
normalizer("I'm a little teapot, short and stout. Tip me over and pour me out!")

'i am a little teapot short and stout tip me over and pour me out'

article_text = """Language is like a map that we use to navigate the world, but it’s also like a prison that keeps us from seeing what’s beyond the walls.

But what if there was a way to break out of this prison, to expand our map, to explore new worlds with new words? This is the possibility and the challenge offered by instruction tuned language models like GPT 4, a cutting-edge technology that uses artificial neural networks to generate natural language texts based on user inputs.

GPT 4 can write anything from essays to novels to poems to tweets to code to recipes to jokes to lyrics to whatever you want. It can even write things that don’t exist yet, things that no human has ever thought of or said before.

As Wittgenstein’s quote suggests, language is a source of limitation and liberation. GPT 4 pushes this idea to the extreme by giving us access to unlimited language.

This could be the most significant new technology in modern history because it has the potential to change many domains and industries. From education to entertainment, from journalism to justice, from science to art, these models could enable new forms of learning, storytelling, reporting, reasoning, discovery, and creation.

They could also create new ethical, social, and cultural challenges that require careful reflection and regulation. How we use this technology will depend on how we recognize its implications for ourselves and others.

This technology is a form of “Artificial Intelligence”. The word “intelligence” derives from inter- (“between”) and legere (“to choose, pick out, read”). To be intelligent, then, is to be able to choose between things, to pick out what matters, to read what is written. Intelligence is not just a quantity or a quality; it is an activity, a process, a practice. It is something that we do with our minds and our words.

But when we let GPT 4 do this for us, are we not abdicating our intelligence? Are we not letting go of our ability to choose, to pick out, to read? Are we not becoming passive consumers of language instead of active producers?
"""
normalizer(article_text)

'language is like a map that we use to navigate the world but it s also like a prison that keeps us from seeing what s beyond the walls but what if there was a way to break out of this prison to expand our map to explore new worlds with new words this is the possibility and the challenge offered by instruction tuned language models like gpt 4 a cutting edge technology that uses artificial neural networks to generate natural language texts based on user inputs gpt 4 can write anything from essays to novels to poems to tweets to code to recipes to jokes to lyrics to whatever you want it can even write things that don t exist yet things that no human has ever thought of or said before as wittgenstein s quote suggests language is a source of limitation and liberation gpt 4 pushes this idea to the extreme by giving us access to unlimited language this could be the most significant new technology in modern history because it has the potential to change many domains and industries from education to entertainment from journalism to justice from science to art these models could enable new forms of learning storytelling reporting reasoning discovery and creation they could also create new ethical social and cultural challenges that require careful reflection and regulation how we use this technology will depend on how we recognize its implications for ourselves and others this technology is a form of artificial intelligence the word intelligence derives from inter and legere to be intelligent then is to be able to choose between things to pick out what matters to read what is written intelligence is not just a quantity or a quality it is an activity a process a practice it is something that we do with our minds and our words but when we let gpt 4 do this for us are we not abdicating our intelligence are we not letting go of our ability to choose to pick out to read are we not becoming passive consumers of language instead of active producers'

`tts_mode` for English

With tts_mode=True the text is prepared for speech: written numbers stay words (the usual words-to-digits step is skipped), digits, currencies and percentages are spelled out, and a few symbols are spoken. Default behaviour is unchanged.

# tts_mode spells digits as words.
tts = EnglishTextNormalizer(tts_mode=True)
assert tts("I have 3 apples and 42 pears") == "i have three apples and forty two pears"

# Written numbers stay words in tts_mode: the usual words-to-digits step is skipped.
assert tts("I was born in nineteen eighty four") == "i was born in nineteen eighty four"
assert tts("a hundred apples") == "a hundred apples"

# Default behaviour is unchanged: tts_mode is off, and words still become digits.
default = EnglishTextNormalizer()
assert default.tts_mode is False
assert default("I was born in nineteen eighty four") == "i was born in 1984"
assert default("a hundred apples") == "a 100 apples"
assert default("I have three apples and 42 pears") == "i have 3 apples and 42 pears"

# tts_mode: a currency symbol is spoken as the amount followed by the unit (singular for one).
for text, expected in [
    ("It costs $20", "it costs twenty dollars"),
    ("just $1", "just one dollar"),
    ("$1,200 total", "one thousand two hundred dollars total"),
    ("€5.50 each", "five euros fifty cents each"),
    ("£2.01", "two pounds one penny"),
    ("£3", "three pounds"),
    ("only 5¢", "only five cents"),
]:
    assert tts(text) == expected, (text, tts(text), expected)

# tts_mode: percentages and decimals are spoken (decimals digit by digit), and a few symbols become words.
for text, expected in [
    ("50% off", "fifty percent off"),
    ("3.5% growth", "three point five percent growth"),
    ("pi is 3.14", "pi is three point one four"),
    ("exactly 3.50", "exactly three point five zero"),
    ("salt & pepper", "salt and pepper"),
    ("2 + 2 = 4", "two plus two equals four"),
    ("meet me @ noon", "meet me at noon"),
]:
    assert tts(text) == expected, (text, tts(text), expected)

# Without the optional num2words package, tts_mode fails loudly with install instructions; default mode is unaffected.
import sys

_saved = sys.modules.get("num2words")
sys.modules["num2words"] = None  # makes `import num2words` raise ImportError
try:
    try:
        EnglishTextNormalizer(tts_mode=True)
        raise AssertionError("expected ImportError")
    except ImportError as e:
        assert "whisper_normalizer[tts]" in str(e)
    assert EnglishTextNormalizer()("I have three apples") == "i have 3 apples"
finally:
    if _saved is None:
        sys.modules.pop("num2words", None)
    else:
        sys.modules["num2words"] = _saved

# tts_mode: a price under one unit is just its minor unit; ordinals are spoken as ordinals.
for text, expected in [
    ("It costs $0.05", "it costs five cents"),
    ("£0.01", "one penny"),
    ("$0", "zero dollars"),
    ("the 1st and 2nd place", "the first and second place"),
    ("on the 21st, 3rd and 11th", "on the twenty first third and eleventh"),
    ("the 100th day", "the one hundredth day"),
]:
    assert tts(text) == expected, (text, tts(text), expected)

What does this module do?

EnglishNumberNormalizer

EnglishSpellingNormalizer

EnglishTextNormalizer

Testing EnglishTextNormalizer

tts_mode for English

Testing `EnglishTextNormalizer`

`tts_mode` for English