Lesson 3 established the classical picture: more model capacity relative to your data means more room to fit noise instead of signal, and test error eventually turns upward — the U-shaped curve. By that logic, a network with, say, a million weights trained on ten thousand examples should be deep in overfitting territory: plenty of capacity to memorize the training set exactly, including its noise, and generalize badly.
What actually happens, empirically, surprisingly often: train such a network with plain stochastic gradient descent (Lesson 6, 19’s mechanism for computing how to update each weight), and test performance is frequently still good — sometimes it even keeps improving as you add more parameters past the point where the model can perfectly fit the training data, which is the opposite of what the classical U-shaped curve predicts.
This isn’t a small footnote — it’s one of the things that genuinely puzzled the field, and reconciling it with classical statistical learning theory is still an active area of research. But there’s a leading explanation that fits the puzzle well enough to build real intuition from.
Your task: among the four explanations above, which one best matches how modern overparameterized networks, trained with SGD, actually behave?