There has been remarkable progress in the capabilities of computers to process and generate language in the last few years.  This course is about learning and applying Deep Learning approaches in Natural Language Processing (NLP), from word vectors to Transformers, including GPT-3, GPT-4 and chatGPT.    It teaches the fundamentals of neural-network-based NLP and gives students the opportunity to pursue a unique project.

The course lecture material begins with the basics of word-level embeddings – their properties and training. These form the basis of neural-network-based classifiers employed to do classification of sentiment, named entity recognition and many other language tasks. A significant part of the course is about the Transformer architecture – its structure, and training. This will include the use of the transformer as a classifier, but also as in generative mode, in which language is produced in response to input language, a field which is now known as prompt engineering. Much of the learning will be applied in three or four hands-on programming assignments. Students will also do a major project of their own choosing to make use of these capabilities. 

Instructor: 

Jonathan Rose (Jonathan.Rose@ece.utoronto.ca) - Department of Electrical and Computer Engineering.  Office: Engineering Annex, Room 319.

Teaching Assistants:

Mohamed Abdelwahab

Jiading Zhu

Text 

Speech and Language Processing, 3rd Edition Draft (January 2023 version) 

https://web.stanford.edu/~jurafsky/slp3/ed3book_jan72023.pdf

This text is missing a first chapter, but you can grab a reasonable version of the first chapter from the second edition, here: https://github.com/rain1024/slp2-pdf

Lectures  

Lecture 0 - Introduction to the Course and Pre-requisites, Course Structure Notes  Video

Lecture 1 - Word Embedding/Vector Properties and Meanings Notes Video

Lecture 2 - Training of Word Embeddings Notes Video

Lecture 3 - Classification of Language using Word Embeddings Notes Videos

Lecture 4 - Introduction to Transformers and Project Structure Notes Part 1 Notes Parts 2 Video

Lecture 5 - The Core Mechanism of Transformers: Attention Notes Video

Lecture 6 - Language Generation Using Language Models and Project Ideation Notes 1 Notes 2 Video

Lecture 7 - Understanding Transformers & Tokenization Notes  Video  

            Chinchilla Paper  Chain of Thought Paper  Llama2 Paper  MediumPostOnPromptEngineering

Lecture 8 - Proposal Presentations (including evening/next evening) No Notes or Video

Lecture 9 - How GPT-4 is Trained to Respond to Human Intent  Notes  Video

    OpenAI_RLHFPaper

Lecture 10 - Logistics of Final Deliverables and Consulting  Video

Lecture 11 - Consultations on Project

Lecture 12 - Project Presentations

Assignments (tentative)

#    Date Assigned               Assignment                                                                                                Due

1     12-Sep                        Word Embeddings – Properties, Meaning and Training                            25-Sept

2     26-Sep                        Classification of Subjective/Objective Text                                               10-Oct 

3     10-Oct                         Understanding, Training and Using Transformers for Classification         23-Oct

4    24-Oct                          Generation Probability Trees, Prompt Engineering                                   15-Nov