Words, tokens, n-grams and Language Models

This lecture covers the concepts of words, tokens, n-grams, and language models. It starts by discussing the ambiguity of defining words and tokens, then delves into n-gram models and their applications in language identification and spelling error correction. The lecture emphasizes the importance of understanding the probabilistic approach, including additive smoothing and Dirichlet priors. Key points include the challenges of out-of-vocabulary forms and the probabilistic approach to spelling error correction.