Beyond the Buzz: Uncovering the Hidden Impact of LLMs on Our Future
Historical progress in NLP evolved from structural to symbolic, to statistical, to (neural network) pre-trained language models (PLMs) and lastly to LLMs – lately we have seen techniques for the distillation of LLMs and the generation of small language models but I don’t want to digress.
Language modeling before the era of deep learning focused on training task-specific models through supervision whereas PLMs are trained through self-supervision with the aim of learning representations that are common across different NLP tasks. As the size of PLMs increased, so did their performance on tasks. That led to LLMs that significantly increased the number of their model parameters and the size of their training dataset.
