Understanding GPT-2: Language Models are Unsupervised Multitask Learners

This is an article where I want to understand GPT-2 [1].

In a nutshell, GPT-2 is a massive version of GPT. As the paper title suggests, it is a paper that demonstrates that multi-tasking is possible without supervision.

Key Highlights of the GPT-2 Model

• They created their own dataset called WebText.
• The model is enormous (by the way, GPT-3 is even larger by an order of magnitude).
• They showed that it can perform various tasks in a zero-shot setting.

Background

GPT proved that various tasks could be executed without changing the model architecture.
However, GPT was not entirely unsupervised learning in the sense that it required fine-tuning.

GPT-2 is a model that aimed to achieve two things:

1. Being able to execute various tasks without changing the model structure.
2. Being able to execute various tasks in a completely unsupervised manner.

What it does is almost the same as GPT, but the dataset used for training and the architecture are slightly different.

Let's look at them in order.

WebText Dataset

The dataset used for training is called WebText, which is a dataset uniquely created by OpenAI.

It consists of over 8 million documents created from 45 million links, with a capacity of about 40GB.

While the WebText dataset is not available for us general users, there are several attempts to reproduce the WebText dataset, so I'll post the links here.

https://github.com/jcpeterson/openwebtext

https://skylion007.github.io/OpenWebTextCorpus/

I haven't looked into the details, so please be careful about the license when using them.

Architecture

Position of Layer Normalization

Comparing GPT-2 with GPT, the position of layer normalization is different (figure below).

Additionally, layer normalization has been added just before the final Linear & Softmax. While it is not explicitly stated in the paper why this change was made, it likely yielded better performance.

Left: GPT model (Excerpted from [2] Figure 1), Right: GPT-2 model. Note that the position of Layer normalization has changed.

Also, [1] evaluates four models: 12x, 24x, 36x, and 48x, where 12x corresponds to the GPT model from previous research (though the structure has changed slightly).

48x is what we typically refer to as GPT-2.

Weight Initialization of Residual Layers

To account for the accumulation of residual paths due to model depth, the weights of the residual layers are scaled by $1/ \sqrt N$, where $N$ is the number of residual layers.

Beyond this, nothing else is written in the paper and the details remain unclear, but this is usually not a concern when using GPT-2.

This is because pre-training is already complete, so weight initialization doesn't matter when using it for zero-shot tasks or even when fine-tuning.

There was a question regarding this weight initialization in a Hugging Face issue.

https://github.com/huggingface/transformers/issues/2158#issuecomment-565639081

Hi, this is for the initialization. We don't have any scripts that show how to pretrain GPT-2 (therefore no need for initialization), only scripts to fine-tune it from a checkpoint.

Based on this response, there doesn't seem to be information on how the initialization is actually performed. If anyone knows, I would appreciate it if you could let me know.

Tokenization

Byte Pair Encoding (BPE)[3]

In language models, preprocessing such as tokenization is performed to obtain input representations for the model.

GPT-2 uses a preprocessing method called Byte Pair Encoding (BPE). BPE has long been known as a data compression technique, and it is a simple method that repeatedly replaces the most frequent byte pairs in a data sequence with an unused byte until no more pairs remain.

In language models at that time, it seems that although they were called "Byte Pairs," they often actually used pairs of characters or character strings [3].

Actually, it was also used in GPT, but I omitted it because there was no particular explanation, just:

We used a bytepair encoding (BPE) vocabulary with 40,000 merges

In GPT-2, BPE is performed in its true sense.

Overview of BPE Execution Steps

1. Extract all characters in the corpus to build a vocabulary.
2. Count the frequency of occurrence of character pairs in the corpus for each word.
3. Merge the most frequent character pair to tokenize it and add it to the vocabulary.
4. Return to step 2 and repeat until the desired vocabulary size is reached.

Note that when actually encoding, it is common to encounter characters that do not exist in the vocabulary (such as emojis). This is handled by replacing characters not in the vocabulary with "[UNK]" (short for Unknown). However, in actual training, it is necessary to cover all Unicode characters, and it's easy to imagine that the vocabulary would become enormous.

Therefore, GPT-2 performs byte-level BPE (BPE in its true sense). This reduces the base vocabulary before tokenization to 256. However, if byte-level BPE is used as is, common frequent words like "dog" would have many variations (dog!, dog?, etc.), making the vocabulary redundant. To prevent this, GPT-2 treats whitespace characters specially and prevents merges across character categories.

There was no explanation of what the character categories are, but judging from the examples, I think it means considering characters appearing in words separately from characters like "!" or "?".

Hugging Face provides a very thorough explanation, so please refer here for details.

https://huggingface.co/course/chapter6/5?fw=pt

Results of Zero-shot Predictions

The figure below shows the results for natural language processing tasks in a zero-shot setting.

From [1] Figure 1

Although it does not reach SOTA, it shows decent performance except for Question Answering. And since the curve has not yet saturated, further performance improvement is expected by increasing the parameters (a precursor to GPT-3).

Experiments were also conducted on various other datasets. The results are shown in the figure below.

From [1] Table 3

As can be seen from the table, GPT-2 achieved SOTA on all datasets except for 1BW. The authors speculate that the poor performance on 1BW was likely due to it being a massive dataset and the destructive preprocessing (sentence-level shuffling).

Details for each task are described at the end of this article.

Summary

As stated at the beginning, it was shown that GPT-2 can handle various tasks in a zero-shot setting. The result was overwhelming, achieving SOTA in 7 out of 8 datasets. Not only that, it is shown that it is still in an underfitted state relative to the WebText dataset used for training.

From [1] Figure 4

In other words, it is suggested that performance can be improved by making the model even more massive, which leads to the development of the successor model, GPT-3.

Appendix

Children's Book Test (CBT) [4]

The Children's Book Test is a dataset created based on children's books to measure how well a language model can utilize a broad linguistic context through the following test:

1. Extract 20 consecutive sentences $S$ from a $context$.
2. Remove any one word $a$ from the 21st sentence following $S$, and let this sentence be the question $q$.
3. Provide the model with 10 word candidates $C$ (including $a$) contained within the context and have it predict the removed word.

[4] explains this with the example in the figure below.

From [4] Figure 1

The left side is the context, and the blue part is $S$. The right side corresponds to $S, q, C, a$ as explained above.

LAMBADA [5]

LAMBADA consists of passages composed of a $context$ (average of 4.6 sentences) and a $target\;sentence$.

The size of the context is the minimum size that includes at least 50 tokens from the complete sentences preceding the $target\;sentence$.

Given the $context$ and $target\;sentence$, the LAMBADA task is to predict the last word ($target\;word$).

The following is an excerpted example:

Excerpted from [5] Figure 1. When the target word appears in the context, that word is highlighted with an underline.

Winograd Schema Challenge [6]

The Winograd Schema (WS) is an evaluation dataset for the task of selecting one correct answer from two choices given for a question. [6] provides the following examples:

The trophy doesn’t fit in the brown suitcase because it’stoo big.
What is too big?
Answer 0: the trophy
Answer 1: the suitcase

Joan made sure to thank Susan for all the help she hadgiven.
Who had given the help?
Answer 0: Joan
Answer 1: Susan

You can obtain the actual evaluation sets from the following page. It's interesting to see that there are evaluation sets for Japanese and Chinese as well.

https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html

Reading Comprehension [7]

This task involves answering the next question in a conversation when given a passage and the conversation itself.

Each turn of the conversation includes a question and its answer. Here is an example:

From [7] Figure 2

Q is the question, A is the answer, and R is the rationale. Rationales are provided as an option to help answer the question, but they are not provided during testing.

Since GPT-2 was evaluated in a zero-shot setting, it is safe to assume that R was not provided.

Summarization

Document summarization is performed using the CNN/Daily Mail Corpus introduced in [8].

100 tokens are generated using a method called Top-k random sampling, and the first three generated sentences are used as the summary. Top-k random sampling is a method where, at each point in sentence generation, the $k$ most probable next words are selected, and then a random sample is taken from them [9].

Here is an example:

From [8] Table 2

Translation

English-to-French translation is performed using the WMT-14 English-French test set, and French-to-English translation is performed using the WMT-14 French-English test set.

https://www.statmt.org/wmt14/index.html

To perform translation tasks with GPT-2, the model is conditioned on example pairs in the format of "English sentence = French sentence," and at prediction time, a prompt like "English sentence =" is provided. Then, greedy sampling is performed from the model, and the first sentence generated is used as the translation.

Question Answering

Evaluation is performed on the task of answering questions using a dataset called Natural Questions [10].

Natural Questions consists of $(question, wikipedia\;page, long\;answer, short\;answer)$.

$question$ is literally a question seeking factual information, $wikipedia\;page$ is information that can be used to answer the question (though whether it's useful is another matter), $long\;answer$ is an excerpt from the Wikipedia page (e.g., a paragraph) that answers the question or indicates that there is no answer on that page, and $short\;answer$ is a short answer to the question, or either yes or no.

The model is given the $question$ and the $wikipedia\;page$ and is asked to predict the $long\;answer$ and $short\;answer$.

References

1. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners.
2. Radford, A., Narasimhan, K., Salimans, T. & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training.
3. Sennrich, R., Haddow, B. & Birch, A. (2015). Neural Machine Translation of Rare Words with Subword Units. arXiv. https://doi.org/10.48550/arxiv.1508.07909
4. Hill, F., Bordes, A., Chopra, S. & Weston, J. (2015). The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations. arXiv. https://doi.org/10.48550/arxiv.1511.02301
5. Paperno, D., Kruszewski, G., Lazaridou, A., Phamh, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G. & Fernández, R. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context. arXiv. https://doi.org/10.48550/arxiv.1606.06031
6. Levesque, H., Davis, E. & Morgenstern, L. (2012, May 17). The winograd schema challenge. Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning.
7. Reddy, S., Chen, D. & Manning, C. D. (2018). CoQA: A Conversational Question Answering Challenge. arXiv. https://doi.org/10.48550/arxiv.1808.07042
8. Nallapati, R., Zhou, B., santos, C. N. dos, Gulcehre, C. & Xiang, B. (2016). Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond. arXiv. https://doi.org/10.48550/arxiv.1602.06023
9. Fan, A., Lewis, M. & Dauphin, Y. (2018). Hierarchical Neural Story Generation. arXiv. https://doi.org/10.48550/arxiv.1805.04833
10. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q. & Petrov, S. (2019). Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics, 7, 453–466. https://doi.org/10.1162/tacl_a_00276