FazBrowse Wikipedia Viewer | Main Page |
Search Wikipedia:
|
Viewing: https://en.wikipedia.org/wiki/Statistical_Language_Model

Language model

A language model is a computational model that predicts sequences in natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, information retrieval and disaster response.

Large language models (LLMs), currently their most advanced form as of 2026, are predominantly based on transformers trained on larger datasets (frequently using texts scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as the word n-gram language model.

⟨s⟩ and Z(w1,…,wm−1) is the partition function, a or some form of regularization.

The log-bilinear model is another example of an exponential language model.

Skip-gram model

edit
1-skip-2-grams for the text "the rain in Spain falls mainly on the plain"

Skip-gram language model is an attempt at overcoming the data sparsity problem that the preceding model (i.e. word n-gram language model) faced. Words represented in an embedding vector were not necessarily consecutive anymore, but could leave gaps that are skipped over (thus the name "skip-gram").

Formally, a k-skip-k from each other.

For example, in the input text:

the rain in Spain falls mainly on the plain

the set of 1-skip-2-grams includes all the bigrams (2-grams), and in addition the subsequences

the in, rain Spain, in falls, Spain mainly, falls on, mainly the, and on plain.

In skip-gram model, semantic relations between words are represented by linear combinations, capturing a form of compositionality. For example, in some such models, if n-d vector representation, then

  1. ↑ Blank, Idan A. (November 2023). "What are large language models supposed to model?". Trends in Cognitive Sciences. 27 (11): 987–989. doi:10.1016/j.tics.2023.08.006. PMID 37659920."LLMs are supposed to model how utterances behave."
  2. ↑ Liu, Yang; Wu, Fanyou; Liu, Zhiyuan; Wang, Kai; Wang, Feiyue; Qu, Xiaobo (2023). "Can language models be used for real-world urban-delivery route optimization?". The Innovation. 4 (6) 100520. Bibcode:2023Innov...400520L. doi:10.1016/j.xinn.2023.100520. PMC 10587631. PMID 37869471.
  3. ↑ Ponte, Jay M.; Croft, W. Bruce (1998). A language modeling approach to information retrieval. Proceedings of the 21st ACM SIGIR Conference. Melbourne, Australia: ACM. pp. 275–281. doi:10.1145/290941.291008.
  4. ↑ Chomsky, Noam (September 1956). "Three models for the description of language". IRE Transactions on Information Theory. 2 (3): 113–124. doi:10.1109/TIT.1956.1056813.
  5. ↑ Bengio, Yoshua; Ducharme, Réjean; Vincent, Pascal; Jauvin, Christian (2003). "A Neural Probabilistic Language Model". Journal of Machine Learning Research. 3: 1137–1155.
  6. ↑ Jurafsky, Dan; Martin, James H. (7 January 2023). "N-gram Language Models". Speech and Language Processing (PDF) (3rd edition draft ed.). Retrieved 24 May 2022.
  7. ↑ Mikolov, Tomas; Chen, Kai; Corrado, Greg; Dean, Jeffrey (2013). "Efficient estimation of word representations in vector space". arXiv:1301.3781 [cs.CL].
  8. ↑ Bengio, Yoshua (2008). "Neural net language models". Scholarpedia. Vol. 3. p. 3881. Bibcode:2008SchpJ...3.3881B. doi:10.4249/scholarpedia.3881. Archived from the original on 26 October 2020. Retrieved 28 August 2015.
  9. ↑ Wolfram, Stephen (2023). What is ChatGPT doing ... and why does it work?. Champaign, Illinois: Wolfram Media, Inc. ISBN 978-1-57955-081-3.
  10. ↑ Karlgren, Jussi; Schutze, Hinrich (2015), "Evaluating Learning Language Representations", International Conference of the Cross-Language Evaluation Forum, Lecture Notes in Computer Science, Springer International Publishing, pp. 254–260, doi:10.1007/978-3-319-64206-2_8, ISBN 978-3-319-64205-5
  11. ↑ "The Corpus of Linguistic Acceptability (CoLA)". nyu-mll.github.io. Archived from the original on 7 December 2020. Retrieved 25 February 2019.
  12. ↑ "MultiNLI". cims.nyu.edu. Retrieved 9 May 2026.
  13. ↑ "The Stanford Question Answering Dataset". rajpurkar.github.io. Archived from the original on 30 October 2020. Retrieved 25 February 2019.
  14. ↑
View original on Wikipedia | Back | FazBrowse Home