CS662

Advanced Natural Language Processing

Staff

Jonathan May

Office Hours: Mondays and Wednesdays 2:00–3:00 pm, SAL 311, or by appointment

Elan Markowitz

esmarkow@usc.edu

Office Hours: Mondays and Wednesdays, 12:00pm–1:00pm, location Zoom (see Piazza)

Lectures

Monday and Wednesday 3:30–5:20 pm, KAP 166

Textbook

Required: Natural Language Processing - Eisenstein – or free version

Required: Selected papers from NLP literature, see (evolving) schedule

Optional: Introduction to Deep Learning - Charniak – first three chapters here

Optional: Speech and Language Processing 3rd edition - Jurafsky, Martin

Grading

10% - In class participation
10% - Posted questions before each in-class selected paper presentation
10% - In-class selected paper presentation
30% - Three Homeworks (10% each)
40% - Project, comprising proposal (5%), first version of report (5%), in-class presentation (10%), and final report (20%). Done in small groups.
Final report is due December 13, 2021, 4:00 PM PST

Contact us

On Piazza, Slack, or in class/office hours. Please do not email (unless notified otherwise).

Topics

(subject to change per instructor/class whim) (will not be presented in this order):

Linguistic Stack (graphemes/phones - words - syntax - semantics - pragmatics - discourse)

Tools:: Corpora, Corpus statistics, Data cleaning and munging; Annotation and crowdwork; Evaluation; Models/approaches: rule-based, automata/grammars, perceptron, logistic regression, neural network models; Effective written and oral communication; Components/Tasks/Subtasks:; Language Models

Syntax: POS tags, constituency tree, dependency tree, parsing: Semantics: lexical, formal, inference tasks; Information Extraction: Named Entities, Relations, Events; Generation: Machine Translation, Summarization, Dialogue, Creative Generation

Schedule of Classes

Aug 23

intro, applications: Eisenstein 1

project assignment out (due 9/1)

Aug 25

end of intro

Aug 30

probability basics, ethics, text processing: Goldwater probability tutorial, Nathan Schneider’s unix notes, Unix for poets, sculpting text

Sep 1

Naive Bayes, Perceptron, Logistic Regression

Eisenstein 2.2, 2.3, 2.4, Charniak 1.: Jon – Preregistering NLP research

project proposal due

Sep 6

LABOR DAY NO CLASS

Sep 8

Perceptron, Logistic Regression, Nonlinear classifiers

Eisenstein 3: Julie – Adversarial Learning for Zero-Shot Stance Detection on Social Media

HW1 out (due 9/29)

Sep 13

Nonlinear classifiers, backpropagation, gradient descent

Eisenstein 6, 18.1: Jiageng – A Disentangled Adversarial Neural Topic Model for Separating Opinions from Plots in User Reviews

Sep 15

YOM KIPPUR NO CLASS

Sep 20

language models: ngram, feed-forward

Eisenstein 18.2, 18.3, 19.1, 19.2: Preni – The Importance of Modeling Social Factors of Language: Theory and Practice

Sep 22

recurrent LM, MT history

Attention is all you need, Illustrated Transformer: Anirudh – End-to-end ASR to jointly predict transcriptions and linguistic annotations

Sep 27

MT evaluation, Statistical MT

Illustrated BERT, ElMo, and co.: Taufeq – Smoothing and Shrinking the Sparse Seq2Seq Search Space

Sep 29

Neural Machine Translation, Transformers

Eisenstein 7: Zhuochen – Continual Learning for Neural Machine Translation

HW1 due

Oct 4

Large Contextualized Language Models (ElMo, BERT, GPT-N, etc.), POS tags, HMMs, constituencies, cky, treebank

Eisenstein 9.2, 10: Fei – Counterfactual Data Augmentation for Neural Machine Translation

HW2 out (due 10/25)

Oct 6

restructuring, dependencies, shift-reduce

Eisenstein 11: Josh – A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

Oct 11

arc-eager, evaluation, human annotation: Fiona – Causal Effects of Linguistic Properties OR; Adam – D2S: Document-to-Slide Generation Via Query-Based Text Summarization

Oct 13

semantics: word sense, propbank, amr, distributional

Eisenstein 13, 14.: Kartik – Enhancing Factual Consistency of Abstractive Summarization

Oct 18

Blade Runner NLP/Bertology: Hanchen – Noisy Self-Knowledge Distillation for Text Summarization

Oct 20

Information Extraction: Entity/Relation, CRF

Eisenstein 17.1, 17.2: Souvik – Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!

Oct 25

Information Extraction: Events, Zero-shot

Eisenstein 17.3: Zhaoxu – Better Feature Integration for Named Entity Recognition

HW2 due

Oct 27

Question Answering and Asking

Eisenstein 17.5: Jincheng – Abstract Meaning Representation Guided Graph Encoding and Decoding for Joint Information Extraction

HW3 out (due 11/17)

Nov 1

Dialogue

Eisenstein 19.3: Hassan – A Frustratingly Easy Approach for Entity and Relation Extraction

Project Report Version 1 due

Nov 3

Power and Ethics: Pithayuth – Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems

Nov 8

Knowledge Graphs (Guest Lecture Elan Markowitz): Liqiu – Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs

Nov 10

Text Games and Reinforcement Learning (Guest Lecture Jesse Thomason): Aleksei – How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds

Nov 15

How to write a paper

Neubig slides on Piazza: Haoming – Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation

Nov 17

Generalization and Robustness (Guest Lecture Robin Jia): Yi – Generating An Optimal Interview Question Plan Using A Knowledge Graph And Integer Linear Programming

HW3 due

Nov 22

TBD: Jayanth – Knowledge Router: Learning Disentangled Representations for Knowledge Graphs; Hitesh – Dynamically Disentangling Social Bias from Task-Oriented Representations with Adversarial Attack

Nov 24

THANKSGIVING BREAK; NO CLASS

Nov 29

Project presentations: James – QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering

Dec 1

Project presentations: Abhinav – Robust Question Answering Through Sub-part Alignment