2022-10-20 · preprint · Topic Modeling · en
highly citedfrom the source record
Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models.
Claims drawn from cited facts, not live model generation.
This work is a preprint classified under topic modeling that concerns itself with instruction finetuning of language models. Because it is a preprint, it has not yet cleared peer review, so its findings should be given correspondingly limited weight.
4 cited facts
This work was produced by a small team of three researchers.
1 cited fact
The work has received 1,440 citations, a figure that marks it as an already established reference point in its field. It cites 0 prior works, indicating a reference list of zero.
3 cited facts
Scaling Instruction-Finetuned Language Models is a preprint, focusing on Topic Modeling, published in 2022.
Scaling Instruction-Finetuned Language Models has 1,440 citations in our record, placing it among the highly cited works we track.
Our record lists 3 authors for Scaling Instruction-Finetuned Language Models. Author names aren't in our data yet — the source link on the page lists the full byline.
Source facts, citations, and refresh stamp for this record.
Sources: openalex_works