Link Search Menu Expand Document

CS662

Advanced Natural Language Processing

Staff

Jonathan May

Office Hours: Mondays and Wednesdays 2:00–3:00 pm, SAL 311, or by appointment

Elan Markowitz

esmarkow@usc.edu

Office Hours: Mondays and Wednesdays, 12:00pm–1:00pm, location TBD

Lectures

Monday and Wednesday 3:30–5:20 pm, KAP 166

Textbook

Required: Natural Language Processing - Eisenstein – or free version

Required: Selected papers from NLP literature, see (evolving) schedule

Optional: Introduction to Deep Learning - Charniak – first three chapters here

Optional: Speech and Language Processing 3rd edition - Jurafsky, Martin

Grading

10% - In class participation
10% - Posted questions before each in-class selected paper presentation
10% - In-class selected paper presentation
30% - Three Homeworks (10% each)
40% - Project, comprising proposal (5%), first version of report (5%), in-class presentation (10%), and final report (20%). Done in small groups.
Final report is due December 13, 2021, 4:00 PM PST

Contact us

On Piazza, Slack, or in class/office hours. Please do not email (unless notified otherwise).

Topics

(subject to change per instructor/class whim) (will not be presented in this order):

Linguistic Stack (graphemes/phones - words - syntax - semantics - pragmatics - discourse)

Tools:
Corpora, Corpus statistics, Data cleaning and munging
Annotation and crowdwork
Evaluation
Models/approaches: rule-based, automata/grammars, perceptron, logistic regression, neural network models
Effective written and oral communication
Components/Tasks/Subtasks:
Language Models
Syntax: POS tags, constituency tree, dependency tree, parsing
Semantics: lexical, formal, inference tasks
Information Extraction: Named Entities, Relations, Events
Generation: Machine Translation, Summarization, Dialogue, Creative Generation

Schedule of Classes

Aug 23
intro, applications
Eisenstein 1
project assignment out (due 9/1)
Aug 25
end of intro
Aug 30
probability basics, ethics, text processing
Goldwater probability tutorial, Nathan Schneider’s unix notes, Unix for poets, sculpting text
Sep 1
Naive Bayes, Perceptron, Logistic Regression
Eisenstein 2.2, 2.3, 2.4, Charniak 1.
Jon – Preregistering NLP research
project proposal due
Sep 6
LABOR DAY NO CLASS
Sep 8
Perceptron, Logistic Regression, Nonlinear classifiers
Eisenstein 3
Julie – Adversarial Learning for Zero-Shot Stance Detection on Social Media
HW1 out (due 9/29)
Sep 13
Nonlinear classifiers, backpropagation, gradient descent, language models: ngram, feed-forward, recurrent Machine Translation history, evaluation
Eisenstein 6, 18.1
Jiageng – A Disentangled Adversarial Neural Topic Model for Separating Opinions from Plots in User Reviews
Sep 15
YOM KIPPUR NO CLASS
Sep 20
Statistical, Neural Machine Translation, summarization, generation
Eisenstein 18.2, 18.3, 19.1, 19.2
Preni – The Importance of Modeling Social Factors of Language: Theory and Practice
Sep 22
Transformers
Attention is all you need, Illustrated Transformer
Anirudh – End-to-end ASR to jointly predict transcriptions and linguistic annotations
Sep 27
Large Contextualized Language Models (ElMo, BERT, GPT-N, etc.)
Illustrated BERT, ElMo, and co.
Taufeq – Smoothing and Shrinking the Sparse Seq2Seq Search Space
Sep 29
POS tags, HMMs
Eisenstein 7
Zhuochen – Continual Learning for Neural Machine Translation
HW1 due
Oct 4
constituencies, cky, treebank
Eisenstein 9.2, 10
Fei – Counterfactual Data Augmentation for Neural Machine Translation
HW2 out (due 10/25)
Oct 6
restructuring, dependencies, shift-reduce
Eisenstein 11
Josh – A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Oct 11
arc-eager, evaluation, human annotation
Fiona – Causal Effects of Linguistic Properties OR
Adam – D2S: Document-to-Slide Generation Via Query-Based Text Summarization
Oct 13
semantics: word sense, propbank, amr, distributional
Eisenstein 13, 14.
Kartik – Enhancing Factual Consistency of Abstractive Summarization
Oct 18
Blade Runner NLP/Bertology
Hanchen – Noisy Self-Knowledge Distillation for Text Summarization
Oct 20
Information Extraction: Entity/Relation, CRF
Eisenstein 17.1, 17.2
Souvik – Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!
Oct 25
Information Extraction: Events, Zero-shot
Eisenstein 17.3
Zhaoxu – Better Feature Integration for Named Entity Recognition
HW2 due
Oct 27
Question Answering and Asking
Eisenstein 17.5
Jincheng – Abstract Meaning Representation Guided Graph Encoding and Decoding for Joint Information Extraction
HW3 out (due 11/17)
Nov 1
Dialogue
Eisenstein 19.3
Hassan – A Frustratingly Easy Approach for Entity and Relation Extraction
Project Report Version 1 due
Nov 3
Power and Ethics
Pithayuth – Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems
Nov 8
Knowledge Graphs (Guest Lecture Elan Markowitz)
Liqiu – Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs
Nov 10
Text Games and Reinforcement Learning (Guest Lecture Jesse Thomason)
Aleksei – How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds
Nov 15
How to write a paper
Neubig slides on Piazza
Haoming – Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation
Nov 17
Generalization and Robustness (Guest Lecture Robin Jia)
Yi – Generating An Optimal Interview Question Plan Using A Knowledge Graph And Integer Linear Programming
HW3 due
Nov 22
TBD
Jayanth – Knowledge Router: Learning Disentangled Representations for Knowledge Graphs
Nov 24
THANKSGIVING BREAK; NO CLASS
Nov 29
Project presentations
James – QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
Dec 1
Project presentations
Abhinav – Robust Question Answering Through Sub-part Alignment