Before the advent of neural NLP, the state of the art in language modelling was defined by n-gram models. For example, a 2-gram model approximates the probability of the next word $w_{t+1}$ by breaking it down into a product of atomic probabilities of the form $P(w_i|w_{i-1})$:

$P(w_{t+1}|w_1, \dots, w_t) \sim \prod_{i=1}^t P(w_{i+1}|w_i)$

Each atomic probability quantifies the likelihood of the next word $w_{i+1}$ in the sequence, given only the immediately preceding word $w_i$.

One problem with $n$-gram models was to get useful estimates of their atomic probabilities. Assuming a vocabulary of 100,000 words, how many atomic probabilities have to be estimated in a 3-gram model, which are conditioned on the two immediately preceding words?

GPT-3 maps texts to vectors of length 12288. Fine-tuning a pre-trained GPT-3 model on a classification task requires us to add a final linear layer that transforms these representations to $k$-dimensional vectors, where $k$ is the number of classes in the task at hand.

Consider a sentiment classification task with three classes: positive, negative and neutral. How many parameters do we need to train when fine-tuning a pre-trained GPT-3 model to this task?

This article explains why OpenAI chose to not immediately release their language model GPT-2. One of the reasons was …