Link Search Menu Expand Document


Advanced Natural Language Processing


Jonathan May

Office Hours: Mondays and Wednesdays 2:00–3:00 pm, SAL 311, or by appointment

Elan Markowitz

Office Hours: Mondays and Wednesdays, 12:00pm–1:00pm, location Zoom (see Piazza)


Monday and Wednesday 3:30–5:20 pm, KAP 166


Required: Natural Language Processing - Eisenstein – or free version

Required: Selected papers from NLP literature, see (evolving) schedule

Optional: Introduction to Deep Learning - Charniak – first three chapters here

Optional: Speech and Language Processing 3rd edition - Jurafsky, Martin


10% - In class participation
10% - Posted questions before each in-class selected paper presentation
10% - In-class selected paper presentation
30% - Three Homeworks (10% each)
40% - Project, comprising proposal (5%), first version of report (5%), in-class presentation (10%), and final report (20%). Done in small groups.
Final report is due December 13, 2021, 4:00 PM PST

Contact us

On Piazza, Slack, or in class/office hours. Please do not email (unless notified otherwise).


(subject to change per instructor/class whim) (will not be presented in this order):

Linguistic Stack (graphemes/phones - words - syntax - semantics - pragmatics - discourse)

Corpora, Corpus statistics, Data cleaning and munging
Annotation and crowdwork
Models/approaches: rule-based, automata/grammars, perceptron, logistic regression, neural network models
Effective written and oral communication
Language Models
Syntax: POS tags, constituency tree, dependency tree, parsing
Semantics: lexical, formal, inference tasks
Information Extraction: Named Entities, Relations, Events
Generation: Machine Translation, Summarization, Dialogue, Creative Generation

Schedule of Classes

Aug 23
intro, applications
Eisenstein 1
project assignment out (due 9/1)
Aug 25
end of intro
Aug 30
probability basics, ethics, text processing
Goldwater probability tutorial, Nathan Schneider’s unix notes, Unix for poets, sculpting text
Sep 1
Naive Bayes, Perceptron, Logistic Regression
Eisenstein 2.2, 2.3, 2.4, Charniak 1.
Jon – Preregistering NLP research
project proposal due
Sep 6
Sep 8
Perceptron, Logistic Regression, Nonlinear classifiers
Eisenstein 3
Julie – Adversarial Learning for Zero-Shot Stance Detection on Social Media
HW1 out (due 9/29)
Sep 13
Nonlinear classifiers, backpropagation, gradient descent
Eisenstein 6, 18.1
Jiageng – A Disentangled Adversarial Neural Topic Model for Separating Opinions from Plots in User Reviews
Sep 15
Sep 20
language models: ngram, feed-forward
Eisenstein 18.2, 18.3, 19.1, 19.2
Preni – The Importance of Modeling Social Factors of Language: Theory and Practice
Sep 22
recurrent LM, MT history
Attention is all you need, Illustrated Transformer
Anirudh – End-to-end ASR to jointly predict transcriptions and linguistic annotations
Sep 27
MT evaluation, Statistical MT
Illustrated BERT, ElMo, and co.
Taufeq – Smoothing and Shrinking the Sparse Seq2Seq Search Space
Sep 29
Neural Machine Translation, Transformers
Eisenstein 7
Zhuochen – Continual Learning for Neural Machine Translation
HW1 due
Oct 4
Large Contextualized Language Models (ElMo, BERT, GPT-N, etc.), POS tags, HMMs, constituencies, cky, treebank
Eisenstein 9.2, 10
Fei – Counterfactual Data Augmentation for Neural Machine Translation
HW2 out (due 10/25)
Oct 6
restructuring, dependencies, shift-reduce
Eisenstein 11
Josh – A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Oct 11
arc-eager, evaluation, human annotation
Fiona – Causal Effects of Linguistic Properties OR
Adam – D2S: Document-to-Slide Generation Via Query-Based Text Summarization
Oct 13
semantics: word sense, propbank, amr, distributional
Eisenstein 13, 14.
Kartik – Enhancing Factual Consistency of Abstractive Summarization
Oct 18
Blade Runner NLP/Bertology
Hanchen – Noisy Self-Knowledge Distillation for Text Summarization
Oct 20
Information Extraction: Entity/Relation, CRF
Eisenstein 17.1, 17.2
Souvik – Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!
Oct 25
Information Extraction: Events, Zero-shot
Eisenstein 17.3
Zhaoxu – Better Feature Integration for Named Entity Recognition
HW2 due
Oct 27
Question Answering and Asking
Eisenstein 17.5
Jincheng – Abstract Meaning Representation Guided Graph Encoding and Decoding for Joint Information Extraction
HW3 out (due 11/17)
Nov 1
Eisenstein 19.3
Hassan – A Frustratingly Easy Approach for Entity and Relation Extraction
Project Report Version 1 due
Nov 3
Power and Ethics
Pithayuth – Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems
Nov 8
Knowledge Graphs (Guest Lecture Elan Markowitz)
Liqiu – Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs
Nov 10
Text Games and Reinforcement Learning (Guest Lecture Jesse Thomason)
Aleksei – How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds
Nov 15
How to write a paper
Neubig slides on Piazza
Haoming – Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation
Nov 17
Generalization and Robustness (Guest Lecture Robin Jia)
Yi – Generating An Optimal Interview Question Plan Using A Knowledge Graph And Integer Linear Programming
HW3 due
Nov 22
Jayanth – Knowledge Router: Learning Disentangled Representations for Knowledge Graphs
Hitesh – Dynamically Disentangling Social Bias from Task-Oriented Representations with Adversarial Attack
Nov 24
Nov 29
Project presentations
James – QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
Dec 1
Project presentations
Abhinav – Robust Question Answering Through Sub-part Alignment