Deep Learning by Quoc Le (not focused on NLP but foundations of Deep Learning)

1. Part 1 (PDF): Nonlinear Classifiers and The Backpropagation Algorithm
2. Part 2 (PDF): Autoencoders, Convolutional Neural Networks and Recurrent Neural Networks
3. Videos (5 hrs):
   … Lecture 1 (1 hr)
   … Lectures 2 & 3 (2 hr)
   … Lectures 4 & 5 (2 hr)
   

   Part of MLSS 2014 Lectures (40+ hours) and Material. Referenced by this article.

Books:

1. A Course in Machine Learning (free e-book) by Prof Hal Daume (2017).
2. Deep Learning (2016, free e-book) by Ian Goodfellow, Yoshua Bengio and Aaron Courville. Slides — Printed copy at Amazon.
3. Speech and Language Processing by Jurafsky and Martin: 3rd Edition drafts. This edition covers the latest developments after the deep learning revolution that has invigorated NLP.
4. Neural Network Methods in Natural Language Processing (310 pages) by Yoav Goldberg (2017). GitHub Link.

   This book is based on an earlier writeup: A Primer on Neural Network Models for Natural Langauge Processing by Yoav Goldberg, Nov 2015. Appeared in J Artificial Research, Vol 57, 2016.

Courses

1. Fast.ai: A Code-First Introduction to NLP — Videos (16 hrs).
2. CS124: From Languages to Information by Jurafsky (2019): website. This course is based on the 3rd Edition of Jurafski and Martin's book which covers Deep Learning. — Lecture Videos (2018, 9 hrs 47 mins)
   — Week 1: Basic Text Processing & Edit Distance (1:44)
   — Week 2: Language Modeling & Spelling Correction (1:42)
   — Week 3: Text Classification & Sentiment Analysis (2:03)
   — Week 4: Information Retrieval (2:17)
   — Week 5: Relation Extraction & Question Answering (1:18)
   — Week 7: Recommendation Systems & Vector Semantics (21 mins)
   — Week 8: PageRank (24 mins)
3. CS224N: Natural Language Processing with Deep Learning by Chris Manning: Website — Videos (2019 by Manning, 27 hrs) — Slides & Assignments — Videos (2017 by Manning and Socher, 25 hrs, 3 mins) — Slides.
4. Top 10 Courses in NLP (2019, KDNuggets)
5. Applied Natural Language Processing (2019, IIT Madras)

Research Papers:

1. NMT (2016)
2. Attention (2017).

   Explanations: Video (27 mins, 2017, Yannic Kilcher) — Video (10 mins, 2018, Andrew Ng).

3. BERT (2018)

   Video (40 mins, 2019, Yannic Kilcher).

Big Picture:

1. Language Model Overview: From word2vec to BERT by James King
2. Embed, encode, attend, predict (2016) by Matthew Honnibal — Video (27 mins, Matthew Honnibal) and Explanatory Talk (2017, Sujit Pal).

Interesting Websites

1. Papers with Code.
2. NLP Progress.
3. Ruder.io.

Books:

1. NLP Notes by Jacob Eisenstein (2018 draft).
2. Information Retrieval by Manning, Raghavan and Schutze (2009).

Longer list of papers:

1. (2003) A Neural Probabilistic Language Model (NNML) by Bengio, Ducharme, Vincent, Jauvin, J Machine Learning Research 3, p 1137-1155.
2. (2013) Efficient Estimation of Word Representations in Vector Space by Mikolov, Chen, Corrado & Dean, ICLR 2013.
3. (2013) Distributed Representations of Words and Phrases and their Compositionality by Mikolov, Sutskever, Chen, Corrado & Dean, NIPS 2013.
4. (2014) GloVe: Global Vectors for Word Representation by Pennington, Socher & Manning, EMNLP 2014. — GloVe Homepage
5. (2014) Distributed Representations of Sentences and Documents by Le and Mikolov, ICML 2014.
6. (2015) Skip-Thought Vectors by Kiros, Zhu, Salakhutdinov, Zemel, Torralba, Urtasun & Fidler, NIPS 2015. — GitHub — Blog article
7. (2018) An Efficient Framework For Learning Sentence Representations by Logeswaran & Lee, ICLR 2018 — GitHub (S2V) — OpenReview.
8. (2018) Supervised Learning of Universal Sentence Representations from Natural Language Inference Data by Conneau, Kiela, Schwenk, Barrault & Bordes, EMNLP 2017 — GitHub (InferSent) — GitHub (SentEval) — Blog article 1 — Blog article 2 — Blog article 3
9. (2018) Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning by Subramanian, Trischler, Bengio & Pal, ICLR 2018 — GitHub
10. (2018) Universal Sentence Encoder by Cer, Yang, Kong, Hua, Limtiaco, John, Constant, Guajardo-Cespedes, Yuan, Tar, Sung, STrope & Kurzweil, EMNLP 2018 — TFHub — CoLab
11. (2017) Learned in Translation: Contextualized Word Vectors by McCann, Bradbury, Xiong & Socher, NIPS 2017 — GitHub — Blog article 1 — Blog article 2
12. (2018) Deep contextualized word representations by Peters, Neumann, Iyyer, Gardner, Clark, Lee & Zettlemoyer, ACL 2018 — Elmo — GitHub
13. (2018) Universal Language Model Fine-tuning for Text Classification (ULMFit) by Howard & Ruder, ACL 2018 — GitHub
14. Improving Language Understanding by Generative Pre-Training (GPT) by Radford, Narasimhan, Salimans & Sutskever — GitHub — Blog article 1
15. Attention Is All You Need — GitHub — Blog Article (The Illustrated Transformer by J Alammar)
16. Generating Wikipedia by Summarizing Long Sequences — GitHub
17. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — GitHub — Blog article: BERT Illustrated — Blog article: Introduction to the World of BERT — Blog article 3: BERT Explained — Blog article: BERT Explained - FAQ — Blog article: BERT EXplained
18. XLNet (Google)
19. RoBERTa (Google / CMU)
20. DistilBERT (HuggihgFace)
21. CTRL (Salesforce)
22. GPT-2 (OpenAI)
23. ALBERT (Google)
24. Megatron (NVIDIA)
25. (2019) Release Strategies and the Social Impacts of Language Models by Solaiman et al.

Blog articles:

(2016) Word2Vec Tutorial - The Skip-Gram Model by Chris McCormick.

(2018) NLP's ImageNet moment has arrived

(2019) Visualizing A Neural Machine Translation Model by Jay Alammar. Referencd by this lecture by Lex Fridman.

Deep Learning State of the Art (2020) by Lex Fridman.

Over 200 of the Best Machine Learning, NLP, and Python Tutorials — 2018 Edition by Robbie Allen

Software:

Write with Transformer by HuggingFace.

Dialog Systems:

(2019) Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems by Wu, Madotto, Hosseini-Asl, Xiong, Socher & Fung, ACL 2019).

(2019) by Rajani, McCann & Socher, ACL 2019.