Chapter 3 questions

John6666
Sun, 07 Dec 2025 20:26:03 +0000

Maybe:

The loss function is torch.nn.CrossEntropyLoss (for num_labels > 1), which PyTorch implements internally as LogSoftmax + NLLLoss. That is why you see grad_fn=NllLossBackward0. Mathematically, for single-label classification, this is exactly a negative log-likelihood loss.

1. What the model code uses

For BertForSequenceClassification, the loss logic is:

If num_labels == 1: regression, uses MSELoss.

If num_labels > 1: classification, uses CrossEntropyLoss.

So in the typical classification case, the actual loss module is:

torch.nn.CrossEntropyLoss on the logits and the integer class labels.

That is the high-level answer.

2. Why you see NllLossBackward in grad_fn

PyTorch defines CrossEntropyLoss as a combination of two operations:

1. Apply a log-softmax to the logits.

2. Apply a negative log-likelihood loss to those log-probabilities.

Conceptually:

Let logits be a vector z \in \mathbb{R}^C for C classes.

Softmax probabilities are
p_i = \frac{e^{z_i}}{\sum_{j=1}^C e^{z_j}}.

Log-softmax is
\log p_i = z_i - \log \left( \sum_{j=1}^C e^{z_j} \right).

Internally, CrossEntropyLoss does something equivalent to:

Compute \log p_i via a log-softmax.

Feed those log-probabilities into NLLLoss (negative log-likelihood loss).

Autograd tracks the low-level ops. The last primitive that creates the scalar loss is the NLL part, so the gradient function shows up as something like:

grad_fn=<NllLossBackward0>

even though at the API level you called CrossEntropyLoss.

So there is no contradiction: the HF model calls CrossEntropyLoss, and PyTorch’s implementation of that is what produces NllLossBackward in the computation graph.

3. Cross-entropy vs negative log-likelihood

The course text calling it a “negative log-likelihood loss” is also correct. For standard single-label classification with one correct class per example, cross-entropy equals negative log-likelihood.

Assume:

There are C classes.

Model outputs logits z \in \mathbb{R}^C.

Softmax probabilities:
p_i = \frac{e^{z_i}}{\sum_{j=1}^C e^{z_j}}.

True label index is y \in {0, \dots, C-1}.

One-hot target vector t has t_y = 1 and t_i = 0 for i \neq y.

Cross-entropy

The cross-entropy between target t and prediction p is

H(t, p) = -\sum_{i=1}^C t_i \log p_i.

For a one-hot target, only t_y = 1 contributes, so

H(t, p) = -\log p_y.

Negative log-likelihood

If we view p_y as the model’s probability for the true class, then the log-likelihood of the data point is

\log p_y,

and the negative log-likelihood (NLL) is

\text{NLL} = -\log p_y.

So for single-label classification,

\text{cross-entropy} = \text{NLL}.

This is exactly what NLLLoss is computing when you supply it with log-probabilities and the class index.

Since CrossEntropyLoss = log-softmax + NLL, for the single-label case it is just a convenient wrapper for computing that negative log-likelihood from raw logits.

4. How this ties back to your observation

Putting it all together:

The Hugging Face model code:

Calls CrossEntropyLoss for classification.

PyTorch implementation:

CrossEntropyLoss internally does log_softmax then NLLLoss.

The backward node is therefore named NllLossBackward.

The math:

For a one-hot target, cross-entropy equals negative log-likelihood of the true class.

So:

The HF course saying “negative log-likelihood loss”

The Transformers code using CrossEntropyLoss

The grad_fn=NllLossBackward0

are three consistent descriptions of the same underlying loss for classification.

Short bullet summary

For BertForSequenceClassification:

num_labels > 1 ⇒ torch.nn.CrossEntropyLoss on logits and labels.

num_labels == 1 ⇒ torch.nn.MSELoss for regression.

PyTorch’s CrossEntropyLoss is implemented as log-softmax + NLLLoss.

Autograd exposes the low-level op, so you see grad_fn=NllLossBackward0.

For single-label classification, cross-entropy equals negative log-likelihood, so calling it an NLL objective is mathematically accurate.