Welcome back
Last time, we sat in a classroom and learned to split our examples three ways: training data to learn from, validation data to check our progress, and test data kept sealed for an honest final exam. We also heard a warning bell: the moment when training loss keeps falling but validation loss begins to rise.
Today we’ll give that warning bell a name. To understand it, let’s visit a tailor.
Two tailors
Imagine a clothing company asks two tailors to design a coat pattern. Both are given the same model to measure: a man named Henry.
The first tailor measures Henry’s shoulders, chest, waist and arm length. She cuts a coat that fits Henry well, and also fits most men of a similar build. It’s a good pattern. The company can sell it to thousands.
The second tailor is more ambitious. He measures everything: the slight slope of Henry’s left shoulder, the bump where Henry once broke his wrist, the way Henry’s wallet makes one pocket bulge. His coat fits Henry like a second skin, better than the first tailor’s ever could.
But when the company tries the second coat on anyone else, it’s a disaster. It pulls in odd places and sags in others. The tailor fitted Henry’s quirks, not the shape of a person.
Overfitting
A neural network can become that second tailor. This is called overfitting.
Our training data is like Henry. It contains real patterns, the general shape we want to learn. But it also contains quirks: noise, accidents, coincidences that just happen to be in this particular set of examples. A network with lots of weights is very capable, and if we train it long enough, it can start fitting everything, quirks included.
When that happens, the network does beautifully on its training data, often near-perfectly. But on new examples, it stumbles, because those quirks aren’t there. It has memorised Henry instead of learning what a person looks like.
The opposite problem also exists. A network that’s too simple, or hasn’t trained long enough, is like a tailor who only knows “small, medium, large.” It fits nobody well, not even Henry. That’s called underfitting. What we’re after is the sweet spot between the two.
Spotting it
This is where the validation data from last lesson earns its keep. Remember the two loss curves?
- Early in training, both fall together. The network is learning the real shape.
- At some point, the validation loss stops improving and starts to climb, while training loss keeps dropping.
That turning point is where the network starts to overfit. It’s the moment the tailor stops measuring the person and starts measuring the wallet.
A first, simple remedy
The simplest cure is to stop at the right time. If we watch the validation loss and halt training when it stops improving, we keep the network from wandering into memorisation. This is called early stopping, and it’s used everywhere.
Another cure is more data. A tailor who measures a hundred men instead of one can’t fit any single man’s quirks, so he’s pushed toward the general shape. The same is true for networks.
What to carry with you
Overfitting is when a network learns its training examples too closely, quirks and all, so it fails on new ones. We spot it when validation loss rises while training loss keeps falling.