NLP: Self-supervised Models (or: "How to grow language models") - Lecture 4 - Fall 2026

NLP: Self-supervised Models (or: "How to grow language models") - Lecture 4 - Fall 2026

by Center for Language & Speech Processing (CLSP), JHU · View the channel on PSI

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQLNatural language and machine translation
🎙 Center for Language & Speech Processing (CLSP), JHU 👥 4K 📅 September 10, 2026 ⏱ 39 min 👁 0 📄 tutorial 🧭 2026-09-10
Available in: English (current) Français

Keywords

loss functioncross entropygradient descentstochastic gradient descentempirical risk

Summary

This lecture from a university course on self-supervised models introduces the foundational concepts needed for training neural networks. It starts by framing machine learning as an optimization problem: given inputs and outputs, find a parameterized function that minimizes an empirical risk based on a loss function. The instructor discusses loss functions, beginning with regression losses (L1 and L2) and contrasting their behavior, particularly how squared error is more sensitive to outliers. Then, the lecture moves to classification losses, focusing on cross-entropy. The instructor derives binary cross-entropy and explains why a negative sign is present, why the loss is always positive, and why a perfect model yields zero loss. It also connects cross-entropy to language modeling, where the goal is to maximize the probability of the next token. The second half covers optimization via gradient descent. Using an analogy of climbing down a mountain with closed eyes, the instructor explains the role of derivatives and gradients in finding the direction of steepest descent. The lecture discusses gradient descent with different initializations, illustrates local minima, and highlights the importance of step size. Finally, it introduces stochastic gradient descent as a more efficient alternative that uses random batches of data. The lecture concludes by summarizing the three key elements: training data, a predictive model, and a loss function, and sets the stage for backpropagation.

221 words

Critical Evaluation

Value of the Information & Strength of the Argument

The value of this lecture lies in its pedagogical clarity. It systematically builds from a general machine learning problem to specific loss functions and optimization, making complex topics accessible. The use of interactive questions helps students develop intuition, such as why cross-entropy loss is positive or why larger errors produce larger losses. The argumentation is sound and mathematically correct, with clear derivations (e.g., binary cross-entropy as negative log probability). However, the content is mainly introductory and does not present new research or advance novel arguments; it is a well-structured review of established concepts.

Scientific Rigor, Source Quality, Title Accuracy

The lecture is scientifically rigorous in its mathematical presentation, but it does not cite external literature—it relies on instructor knowledge and standard ML definitions. The course website is the only provided source. The title adequately represents the lecture’s place within a course on self-supervised models, though the specific content on loss functions and gradient descent is more foundational than ‘self-supervised’ per se. No viewer comments were provided, so no public reception analysis is possible.

182 words

Title / Content Match

The title suggests a broad overview of self-supervised NLP models, but the lecture focuses specifically on loss functions, gradient descent, and optimization—essential building blocks for such models. It is a reasonable fit as a lecture in the series, though not exactly on self-supervised architecture.

Quality & Reliability

8/10

Lecture from an academic institution (JHU CLSP) presenting foundational machine learning concepts clearly, with interactive Q&A and accurate mathematical definitions. Lacks explicit citations and is introductory, but the material is rigorous within its scope.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

Contribution & Novelties

This lecture is a didactic recapitulation of standard techniques, offering no novel contributions to the field. Its value lies in pedagogy: it clearly frames training neural nets as an optimization problem and builds intuition through interactive questioning. It effectively explains why cross-entropy is used for language modeling and how gradient descent functions in high-dimensional spaces.

Pour aller plus loin :

  • Gradient descent — Overview of the optimization algorithm central to the lecture.
  • Cross entropy — Mathematical background on the loss function for classification and language modeling.
  • Stochastic gradient descent — Practical optimization method described in the lecture as faster than full-batch gradient descent.
  • Empirical risk minimization — The general learning principle discussed at the start of the lecture.
  • Backpropagation — The natural next step after computing gradients, as hinted at the end of the lecture.

135 words

Radar Profile

The profile shows strong quality information and good reliability for an introductory lecture, with moderate technical depth and a moderate amount of content. The lecture is clearly a solid pedagogical base rather than an advanced research overview.

Reliability 8/10