Transfer Learning

Backlinks

• [[meta-learning]]
  • relates to [[transfer-learning]], except you’re trying to do an even better kind of pre-training, that generalizes across all tasks
• [[bitter-lesson]]
  • I’m not as familiar with the AI game literature, but at least in the context of NLP/image classification and DL, there seems to be an ideal sweet-spot in terms of finding the right kind of architecture that’s powerful enough to learn, but is simple enough that you can run it on an extreme scale. I think that’s partly why we still continue to innovate on the model side. If we hadn’t done so, then we wouldn’t have gotten [[transfer-learning]], which has been a boon for NLP.
• [[gpt3]]
  • It turns out that GPT-2 required things to be fine-tuned in order to be able to be applied to various tasks. In the spirit of [[transfer-learning]], the GPT-2 can be thought of the base model, and then for a given task (with training data), you then train on that specific dataset.
• [[pediatric-transfer-learning]]
  • Overall, this is very much like [[transfer-learning]].