Deep Learning for Natural Language Processing

Course Content

The lecture will cover the following topics:

• Basics: Text representation, normalization, regular expressions, tokenization, stemming, lemmatization, Bag-of-Words, weighting schemes (e.g., TF-IDF)

• Language models: Probabilistic language models, N-grams, perplexity, smoothing

• Semantics and embeddings: Lexical semantics, lexical databases (WordNet), distance measures, sparse and dense embeddings, word2vec, GloVe

• Neural network basics: Architectures, document classification, word sense disambiguation, evaluation metrics (precision, recall, F1)

• Sequence models: Machine translation, encoder-decoder, Seq2Seq, attention mechanisms, decoding strategies (beam search, stochastic)

• Transformers and pre-training: Self-attention, multi-head attention, Masked and Causal Language Models, efficient fine-tuning (PEFT, LoRA)

• Post-training and alignment: Modern LLMs, RoPE, FlashAttention, MoE, supervised fine-tuning, RLHF, DPO, tool use, RAG

• Reasoning and computation: Test-time reasoning, reinforcement learning (PPO, GRPO), parallel computation (DDP, ZeRO, Ring Attention)

• AI safety and ethics: Alignment failures, jailbreaks, prompt injection, red-teaming, bias mitigation

In the final project, students will work on applied deep learning projects (teamwork) that address complex natural language processing tasks using the programming language Python and NHR GPU clusters.

Learning Objectives

After successfully completing the course, students should be able to:

• Explain state-of-the-art methods to tackle NLP sub-problems, such as text representation, language modeling, sequence-to-sequence generation, and LLM alignment

• Determine the conceptual requirements and training paradigms of specific NLP tasks

• Assess the strengths, limitations, and safety implications of modern neural NLP approaches

• Devise solutions for complex NLP problems by implementing, fine-tuning, and adapting suitable neural architectures and distributed training methods

• Evaluate NLP methods and models quantitatively and qualitatively using standard benchmarks and alignment criteria

The course provides a good foundation for a master’s thesis in our group. Check this page for our current theses proposals.

Requirements

• Knowledge of at least one object-oriented programming language, preferably Python, is required.
• Python is used as part of the course.
• Basic knowledge of neural networks is desired to participate in this course. We recommend that participants are familiar with basic neural network architectures, hidden layers, activation functions, derivatives, classification, training and test strategies, precision, recall, backpropagation, gradients, and other foundational topics in machine learning and artificial neural networks. For participants who are unfamiliar with these topics, an integrated and fast-paced introduction focused on the use case of natural language processing will be provided. At the University of Göttingen’s computer science department, the course B.Inf.1236 Machine Learning provides an excellent foundation for this course.
• Be aware that the course module is now B.Inf.1250 and also open to Bachelor students (previous code was M.Inf.2202)

Exam

• Applied research project (project structure, creativity, results) – 60% of the final grade
• Written test (90 min.) or oral exam (approx. 20 min.) on the lecture content – 40% of the final grade

Teaching Team

• Prof. Dr. Bela Gipp & Dr. Terry Ruas & Dr. Jan Philip Wahle

Time schedule