COSC 7336
Advanced Natural Language Processing

Introduction to Deep Learning for Text Analysis and Understanding
Fall 2017



Course description

Instructors

Thamar Solorio
The University of Houston

Fabio A. González
Universidad Nacional de Colombia

Course Syllabus

Syllabus

Course goal

The goal of the course is to study deep learning models, i.e. neural networks with several layers, and their application to solve challenging natural language analysis problems. The course will cover the foundations of deep learning models as well as the practical issues associated with their design, implementation, training and deployment. A hands-on approach will be used through the course focused on solving different text analysis and understanding tasks motivated by real world problems.

Prerequisites

The course assumes students have taken COSC 6336 or an equivalent itroductory course to NLP. The course assumes knowledge and understanding of machine learning basic concepts, such as those studied in an introductory machine learning or data mining class, as well as knowledge of fundamental concepts of linear algebra and probability theory. The course also requires familiarity with programming in Python, as there will be several practical assignments.



Course topics

The course has two main axes: the first has to do with the problem which is text analysis and understanding, and the second with the methods to address this problem which are based on neural networks in general, and deep learning in particular. The concrete topics that we plan to address during the course, on both axes, are the following:

  • Deep learning (DL):
    • Review of machine learning and neural networks fundamental concepts
    • Computational frameworks for neural network implementation
    • DL models:
      • Convolutional neural networks
      • Recurrent neural networks (RNN): including LSTM, GRU, sequence to sequence RNN, bidirectional RNNs.
      • Attention models
      • Other models: generative adversarial networks, memory neural networks.
  • Text analysis and understanding:
    • Review of natural language processing and analysis fundamental concepts.
    • Word level semantics
    • Text classification: sentiment analysis, author profiling, author identification, text categorization
    • Language model: OCR output correction
    • Conditional language models: summarization
    • Text Similarity: community question answering


Evaluation and grading policy

  • Assignments 45% (3 X 15%)
  • Midterm 20%
  • Paper presentation 10%
  • Final project 25%

Grades



Course resources

Computing Resources

Thanks to the generous sponsorship of Microsoft Research, the course will have access to the cloud platform Azure to support experimentation for the assignments. More information about this will become available soon.

Courses

Tools

References

  • Text Book: [Goodfellow2016] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
  • Text Book: [JM2017] Jurafsky, D. and Martin, J. Speech and Language Processing, 3rd edition draft Chapters.
  • Text Book: [JM2008] Jurafsky, D. and Martin, J. Speech and Language Processing, 2nd edition.
  • [AMLS17] Aguilar, G., Maharjan, S., Lopez Monroy, A.P., and Solorio, T. A Multi-task Approach for Named Entity Recognition in Social Media Data. Proceedings of the 3rd Workshop on Noisy User-generated Text. Copenhagen, Denmark, pp 148-153. 2017. (paper)
  • [DNEL17] Derczynski, L., Nichols, E., van Erp, M. & Limsopatham, N. Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition. In Proceedings of the 3rd Workshop on Noisy, User-generated Text (W-NUT) at EMNLP (paper)
  • [K2014] Yoon Kim. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Confernece on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751. (paper)
  • [MAMGS17] Suraj Maharjan, John Arevalo, Manuel Montes and Fabio A. Gonzalez and Thamar Solorio. A Multi-task Approach to Predict Likability of Books. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp 1217-1227. Valencia, Spain, 2017. (paper)
  • [MSCCD13] Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J. Distributed Representations of Words and Phrases and their Compositionality. arXiv:1310.4546 (paper)
  • [PSM14] Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543). (paper)
  • [R2014] X. Rong, (2014). Word2Vec Parameter learning explained. arXiv:1411.2738 (paper)
  • [SSGRMS17] Shrestha, P., Sierra, S., Gonzalez, F., Rosso, P., Montes y Gomez, M. and Solorio, T. Convolutional Neural Networks for Authorship Attribution of Short Texts. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp 669-674. Valencia, Spain, 2017. (paper)
  • [YYDHSH16] Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. and Hovy, E. Hierarchical Attention Networks for Document Classification. Proceedings of NAACL-HLT 2016, pages 1480-1489. San Diego, California, June 2016. (paper)


Course schedule

Date Topic Material Assignments
Sep 8th Introduction to DL and NLP Lecture 1 slides
NN notebook
Reading material:
[GBC2016] Chap 1, 2, 3
Assignment 0
Due date: Sept. 22nd
Sep 15th Approximating semantics
Neural embedding models
Lecture 2 slides
Word2vec Demo notebook Reading material:
[JM2017] Chap 15, 16
[R2014], [MSCCD13], [AMLS17]
Sep 22nd ML background
Neural network training
Lecture 3 slides
Perceptron Training Notebook
Reading material:
[GBC2016] Chap 5, 6
Assignment 1
Assign 1 Notebook
Due date: Oct. 13th
Sep 29th Deep learning frameworks Lecture 4 slides
TensorFlow Handout Notebook
Azure VM Handout
Keras Handout Notebook
Reading material:
Check tools section in Resources
In-class Assignment 2 Notebook
Oct 6th Text classification
Convolutional neural networks
Lecture 5 slides
CNN Sentence Classification Handout Notebook
Reading material:
[GBC2016] Chap 9
[JM2017] Chap 6
[DNEL17], [YYDHSH16], [K2014],[SSGRMS17]
In-class Assignment 3 Notebook
Oct 13th Language models
Recurrent Neural Networks
Lecture 6 slides
LSTM Language Model Handout Notebook
Reading material:
[GBC2016] Chap 10
[JM2017] Chap 4, 8
Assignment 2
Due date: Nov 9th
Oct 20th Cancelled due to Mid-Semester Bash
Oct 27th Midterm Exam
Nov 3rd Machine Translation
Conditional language model
Neural Attention models
Lecture 7 slides
Seq2Seq models slides
Seq2Seq Translator Handout Notebook
Reading material:
[GBC2016] Chap 10
[JM2008] Chap 25
Useful Links:
How to make a text summarizer
Nov 10th Paper presentations Final project proposal
Due date: Nov 10th
Extra Credit Assignment
Due date: Dec. 1st
Nov 17th Paper Presentations
Dec 1st Multimodal learning
Dec 8th Final project presentations
Poster madness (5:00pm - 8:00pm)
Final project report
Due date: Dec 12th