COSC 7336
Advanced Natural Language Processing

Instructors

Thamar Solorio
The University of Houston

Fabio A. González
Universidad Nacional de Colombia

Course Syllabus

Syllabus

Course goal

The goal of the course is to study deep learning models, i.e. neural networks with several layers, and their application to solve challenging natural language analysis problems. The course will cover the foundations of deep learning models as well as the practical issues associated with their design, implementation, training and deployment. A hands-on approach will be used through the course focused on solving different text analysis and understanding tasks motivated by real world problems.

Prerequisites

The course assumes students have taken COSC 6336 or an equivalent itroductory course to NLP. The course assumes knowledge and understanding of machine learning basic concepts, such as those studied in an introductory machine learning or data mining class, as well as knowledge of fundamental concepts of linear algebra and probability theory. The course also requires familiarity with programming in Python, as there will be several practical assignments.

The course has two main axes: the first has to do with the problem which is text analysis and understanding, and the second with the methods to address this problem which are based on neural networks in general, and deep learning in particular. The concrete topics that we plan to address during the course, on both axes, are the following:

• Deep learning (DL):
  • Review of machine learning and neural networks fundamental concepts
  • Computational frameworks for neural network implementation
  • DL models:
    • Convolutional neural networks
    • Recurrent neural networks (RNN): including LSTM, GRU, sequence to sequence RNN, bidirectional RNNs.
    • Attention models
    • Other models: generative adversarial networks, memory neural networks.
• Text analysis and understanding:
  • Review of natural language processing and analysis fundamental concepts.
  • Word level semantics
  • Text classification: sentiment analysis, author profiling, author identification, text categorization
  • Language model: OCR output correction
  • Conditional language models: summarization
  • Text Similarity: community question answering

• Assignments 45% (3 X 15%)
• Midterm 20%
• Paper presentation 10%
• Final project 25%

Grades

Computing Resources

Thanks to the generous sponsorship of Microsoft Research, the course will have access to the cloud platform Azure to support experimentation for the assignments. More information about this will become available soon.

Courses

• [NLPDL-Stanford] CS224n: Natural Language Processing with Deep Learning, Stanford University, Spring 2017
• [DLNLP-Oxford] Deep Learning for Natural Language Processing: 2016-2017, University of Oxford, 2016-2017
• [DLNLP-UCSB] CS292F: Deep Learning for NLP, University of California at Santa Barbara, Spring 2017

Tools

• TensorFlow: An open-source software library for Machine Intelligence
  • Installation
  • Getting started
  • TensorBoard
  • Programmer’s Guide
  • Vector Representations of Words
  • Recurrent NNs
  • Sequence-to-Sequence Models
• Keras: The Python Deep Learning library
  • Installation
  • Getting started
  • Text preprocessing
  • Pre-trained word embeddings in Keras
  • Classifying Yelp Reviews
  • Keras language modeling
  • Using Keras as part of a TensorFlow workflow
  • More Keras resources

References

• Text Book: [Goodfellow2016] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
• Text Book: [JM2017] Jurafsky, D. and Martin, J. Speech and Language Processing, 3rd edition draft Chapters.
• Text Book: [JM2008] Jurafsky, D. and Martin, J. Speech and Language Processing, 2nd edition.
• [AMLS17] Aguilar, G., Maharjan, S., Lopez Monroy, A.P., and Solorio, T. A Multi-task Approach for Named Entity Recognition in Social Media Data. Proceedings of the 3rd Workshop on Noisy User-generated Text. Copenhagen, Denmark, pp 148-153. 2017. (paper)
• [DNEL17] Derczynski, L., Nichols, E., van Erp, M. & Limsopatham, N. Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition. In Proceedings of the 3rd Workshop on Noisy, User-generated Text (W-NUT) at EMNLP (paper)
• [K2014] Yoon Kim. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Confernece on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751. (paper)
• [MAMGS17] Suraj Maharjan, John Arevalo, Manuel Montes and Fabio A. Gonzalez and Thamar Solorio. A Multi-task Approach to Predict Likability of Books. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp 1217-1227. Valencia, Spain, 2017. (paper)
• [MSCCD13] Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J. Distributed Representations of Words and Phrases and their Compositionality. arXiv:1310.4546 (paper)
• [PSM14] Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543). (paper)
• [R2014] X. Rong, (2014). Word2Vec Parameter learning explained. arXiv:1411.2738 (paper)
• [SSGRMS17] Shrestha, P., Sierra, S., Gonzalez, F., Rosso, P., Montes y Gomez, M. and Solorio, T. Convolutional Neural Networks for Authorship Attribution of Short Texts. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp 669-674. Valencia, Spain, 2017. (paper)
• [YYDHSH16] Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. and Hovy, E. Hierarchical Attention Networks for Document Classification. Proceedings of NAACL-HLT 2016, pages 1480-1489. San Diego, California, June 2016. (paper)

Date	Topic	Material	Assignments
Sep 8th	Introduction to DL and NLP	Lecture 1 slides NN notebook Reading material: [GBC2016] Chap 1, 2, 3	Assignment 0 Due date: Sept. 22nd
Sep 15th	Approximating semantics Neural embedding models	Lecture 2 slides Word2vec Demo notebook Reading material: [JM2017] Chap 15, 16 [R2014], [MSCCD13], [AMLS17]
Sep 22nd	ML background Neural network training	Lecture 3 slides Perceptron Training Notebook Reading material: [GBC2016] Chap 5, 6	Assignment 1 Assign 1 Notebook Due date: Oct. 13th
Sep 29th	Deep learning frameworks	Lecture 4 slides TensorFlow Handout Notebook Azure VM Handout Keras Handout Notebook Reading material: Check tools section in Resources	In-class Assignment 2 Notebook
Oct 6th	Text classification Convolutional neural networks	Lecture 5 slides CNN Sentence Classification Handout Notebook Reading material: [GBC2016] Chap 9 [JM2017] Chap 6 [DNEL17], [YYDHSH16], [K2014],[SSGRMS17]	In-class Assignment 3 Notebook
Oct 13th	Language models Recurrent Neural Networks	Lecture 6 slides LSTM Language Model Handout Notebook Reading material: [GBC2016] Chap 10 [JM2017] Chap 4, 8	Assignment 2 Due date: Nov 9th
Oct 20th	Cancelled due to Mid-Semester Bash
Oct 27th	Midterm Exam
Nov 3rd	Machine Translation Conditional language model Neural Attention models	Lecture 7 slides Seq2Seq models slides Seq2Seq Translator Handout Notebook Reading material: [GBC2016] Chap 10 [JM2008] Chap 25 Useful Links: How to make a text summarizer
Nov 10th	Paper presentations		Final project proposal Due date: Nov 10th Extra Credit Assignment Due date: Dec. 1st
Nov 17th	Paper Presentations
Dec 1st	Multimodal learning
Dec 8th Dec 11th	Final project presentations Poster madness (5:00pm - 8:00pm)		Final project report Due date: Dec 12th