COSC 7336
Advanced Natural Language Processing

Introduction to Deep Learning for Text Analysis and Understanding
Fall 2017


Course description

Instructors

Thamar Solorio
The University of Houston

Fabio A. González
Universidad Nacional de Colombia

Course Syllabus

Syllabus

Course goal

The goal of the course is to study deep learning models, i.e. neural networks with several layers, and their application to solve challenging natural language analysis problems. The course will cover the foundations of deep learning models as well as the practical issues associated with their design, implementation, training and deployment. A hands-on approach will be used through the course focused on solving different text analysis and understanding tasks motivated by real world problems.

Prerequisites

The course assumes students have taken COSC 6336 or an equivalent itroductory course to NLP. The course assumes knowledge and understanding of machine learning basic concepts, such as those studied in an introductory machine learning or data mining class, as well as knowledge of fundamental concepts of linear algebra and probability theory. The course also requires familiarity with programming in Python, as there will be several practical assignments.


Course topics

The course has two main axes: the first has to do with the problem which is text analysis and understanding, and the second with the methods to address this problem which are based on neural networks in general, and deep learning in particular. The concrete topics that we plan to address during the course, on both axes, are the following:

  • Deep learning (DL):
    • Review of machine learning and neural networks fundamental concepts
    • Computational frameworks for neural network implementation
    • DL models:
      • Convolutional neural networks
      • Recurrent neural networks (RNN): including LSTM, GRU, sequence to sequence RNN, bidirectional RNNs.
      • Attention models
      • Other models: generative adversarial networks, memory neural networks.
  • Text analysis and understanding:
    • Review of natural language processing and analysis fundamental concepts.
    • Word level semantics
    • Text classification: sentiment analysis, author profiling, author identification, text categorization
    • Language model: OCR output correction
    • Conditional language models: summarization
    • Text Similarity: community question answering

Evaluation and grading policy

  • Assignments 45% (3 X 15%)
  • Midterm 20%
  • Paper presentation 10%
  • Final project 25%

Grades


Course resources

Computing Resources

Thanks to the generous sponsorship of Microsoft Research, the course will have access to the cloud platform Azure to support experimentation for the assignments. More information about this will become available soon.

Resources

References

  • Text Book: [Goodfellow2016] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
  • Text Book: [JM2017] Jurafsky, D. and Martin, J. Speech and Language Processing, 3rd edition draft Chapters.
  • [AMLS17] Aguilar, G., Maharjan, S., Lopez Monroy, A.P., and Solorio, T. A Multi-task Approach for Named Entity Recognition in Social Media Data. Proceedings of the 3rd Workshop on Noisy User-generated Text. Copenhagen, Denmark, pp 148-153. 2017. (paper)
  • [DNEL17] Derczynski, L., Nichols, E., van Erp, M. & Limsopatham, N. Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition. In Proceedings of the 3rd Workshop on Noisy, User-generated Text (W-NUT) at EMNLP (paper)
  • [MAMGS17] Suraj Maharjan, John Arevalo, Manuel Montes and Fabio A. Gonzalez and Thamar Solorio. A Multi-task Approach to Predict Likability of Books. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp 1217-1227. Valencia, Spain, 2017. (paper)
  • [MSCCD13] Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J. Distributed Representations of Words and Phrases and their Compositionality. arXiv:1310.4546 (paper)
  • [PSM14] Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543). (paper)
  • [R2014] X. Rong, (2014). Word2Vec Parameter learning explained. arXiv:1411.2738 (paper)
  • [YYDHSH16] Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. and Hovy, E. Hierarchical Attention Networks for Document Classification. Proceedings of NAACL-HLT 2016, pages 1480-1489. San Diego, California, June 2016. (paper)

Course schedule

Date Topic Material Assignments
Sep 8th Introduction to DL and NLP Lecture 1 slides
NN notebook
Reading material:
[GBC2016] Chap 1, 2, 3
Assignment 0
Due date: Sept. 22nd
Sep 15th Approximating semantics
Neural embedding models
Lecture 2 slides
Word2vec Demo notebook Reading material:
[JM2017] Chap 15, 16
[R2014], [MSCCD13], [AMLS17]
Sep 22nd ML background
Neural network training
Lecture 3 slides
Reading material:
Perceptron Training Notebook
Reading material:
[GBC2016] Chap 5, 6
Assignment 1
Assign 1 Notebook
Due date: Oct. 13th
Sep 29th Text classification
Convolutional neural networks
Reading material:
[GBC2016] Chap 9
[JM2017] Chap 6
[MAMGS17], [DNEL17], [YYDHSH16]
Oct 6th Deep learning frameworks
Oct 13th Language models
Recurrent Neural Networks
Reading material:
[GBC2016] Chap 10
[JM2017] Chap 8
Assignment 2
Due date: Nov 3rd
Oct 20th Conditional language model
Neural Attention models
Reading material:
[GBC2016] Chap 10
[JM2017] Chap 8
Oct 27th Textual similarity
Neural textual similarity models
Nov 3rd Midterm Exam Assignment 3
Due date: Nov. 17th
Nov 10th Reinforcement learning Final project proposal
Due date: Nov 10th
Nov 17th Multimodal learning
Dec 11th Final project presentations
Poster madness (5:00pm - 8:00pm)
Final project report
Due date: Dec 11th