After GPT-2, it became clear that language models could do much more than researchers originally expected. Simply training a model to predict the next word had already started producing surprising abilities like translation, summarization, and question answering without task-specific training.
But there was still a major limitation. Even though GPT-2 could generalize a...