transfer learning is the process of training a model on a large-scale dataset and then using that pretrained model to conduct learning for another downstream task (i.e., target task) Transfer learning was popularized in the field of computer vision thanks to the Imagenet Dataset

1. A LIGHT INTRODUCTION TO
TRANSFER LEARNING FOR NLP

2. NATURAL LANGUAGE PROCESSING
• NLP INVOLVES MACHINE OR ROBOTS TO UNDERSTAND
• IT PROCESS THE LANGUAGE THAT HUMAN SPEAK.

3. TRANSFER LEARNING
• TRANSFER LEARNING IS THE PROCESS OF TRAINING A MODEL ON A LARGE-
SCALE DATASET AND THEN USING THAT PRETRAINED MODEL TO CONDUCT
LEARNING FOR ANOTHER DOWNSTREAM TASK (I.E., TARGET TASK)
• TRANSFER LEARNING WAS POPULARIZED IN THE FIELD OF COMPUTER VISION
THANKS TO THE IMAGENET DATASET

4. TRANSFER LEARNING FOR NLP
• TRAINING AND TEST DISTRIBUTIONS ARE DIFFERENT
• DIFFERENT TEXT TYPES.
• DIFFERENT ACCENTS/AGES.
• DIFFERENT TOPICS/CATEGORIES.

5. PRETRAINED NLP MODELS
• ULMFIT
• ELMO
• GLOMO
• OPENAI TRANSFORMER

6. ULMFIT
• ULMFIT WAS PROPOSED AND DESIGNED BY FAST.AI’S JEREMY HOWARD AND
DEEPMIND’S SEBASTIAN RUDER. YOU COULD SAY THAT ULMFIT WAS THE
RELEASE THAT GOT THE TRANSFER LEARNING PARTY STARTED LAST YEAR.
• UMFIT STANDS FOR UNIVERSAL LANGUAGE MODEL FINE TUNING FOR TEXT
CLASSIFICATION

7. ELMO
• ELMO WORD REPRESENTATIONS, OR EMBEDDINGS FROM
LANGUAGE MODELS
• PRETRAINING THE ENTIRE MODEL WITH DEEP
CONTEXTUALIZED REPRESENTATIONS THROUGH STACKED
NEURAL LAYERS
• ELMO IS A NOVEL WAY OF REPRESENTING WORDS IN
VECTORS AND EMBEDDINGS. THESE ELMO WORD
EMBEDDINGS HELP US ACHIEVE STATE-OF-THE-ART
RESULTS ON MULTIPLE NLP TASKS

8. GLOMO
• UNSUPERVISED LEARNED RELATIONAL GRAPHS AS TRANSFERABLE
REPRESENT0ATIONS
• MODERN DEEP TRANSFER LEARNING APPROACHES HAVE MAINLY FOCUSED ON
LEARNING GENERIC FEATURE VECTORS FROM ONE TASK THAT ARE
TRANSFERABLE TO OTHER TASKS, SUCH AS WORD EMBEDDINGS IN LANGUAGE
AND PRETRAINED CONVOLUTIONAL FEATURES IN VISION.
• THESE APPROACHES USUALLY TRANSFER UNARY FEATURES AND LARGELY
IGNORE MORE STRUCTURED GRAPHICAL REPRESENTATIONS.
• THIS WORK EXPLORES THE POSSIBILITY OF LEARNING GENERIC LATENT
RELATIONAL GRAPHS THAT CAPTURE DEPENDENCIES BETWEEN PAIRS OF DATA
UNITS (E.G., WORDS OR PIXELS)

9. OPEN AI TRANSFORMER
• IMPROVING LANGUAGE UNDERSTANDING WITH UNSUPERVISED LEARNING
• THE TRANSFORMER ARCHITECTURE IS AT THE CORE OF ALMOST ALL THE
RECENT MAJOR DEVELOPMENTS IN NLP. IT WAS INTRODUCED IN 2017 BY
GOOGLE. BACK THEN, RECURRENT NEURAL NETWORKS (RNN) WERE BEING USED
FOR LANGUAGE TASKS, LIKE MACHINE TRANSLATION AND QUESTION
ANSWERING SYSTEMS.
• GOOGLE RELEASED AN IMPROVED VERSION OF TRANSFORMER LAST YEAR
CALLED UNIVERSAL TRANSFORMER.