Welcome back
Last time, we balanced the light with normalisation, keeping every signal on an even scale so none glared and none hid in shadow. Our network now trains steadily from the first layer to the last.
Today we turn to the very end of the network, to the moment it gives its answer. Many of the questions we ask a network are really sorting questions. Is this photo a cat, a dog, or a rabbit? Is this email spam or not? Which digit is this, 0 through 9? This is called classification.
The sorting room
Picture the back room of a small-town post office before dawn. A long wall of canvas sacks: one for each town on the route. A clerk picks up a letter, glances at the smudged address, and has to decide which sack it goes in.
Sometimes it’s obvious. Sometimes the handwriting is awful, and she thinks, “Probably Millbrook. Could be Milford. Definitely not Ashby.”
That honest, graded hunch is exactly what we want from a network.
One score for every sack
For classification, the final layer gets one output neuron for each possible class: one for cat, one for dog, one for rabbit.
Each of those neurons produces a raw number called a score (people also call it a logit). A higher score means “more like this one.” But raw scores are awkward. They can be negative, they can be huge, and they don’t add up to anything meaningful. A score of 4.1 for cat and 2.3 for dog doesn’t tell us how confident the network is.
We want something clearer: a set of probabilities, each between 0 and 1, all adding up to exactly 1. “70% cat, 25% dog, 5% rabbit.”
Softmax: turning hunches into shares
The tool that does this is called softmax. It works in two gentle steps:
- Make every score positive and stretch the gaps. Each score is put through the exponential function, which turns any number into a positive one. It also makes bigger scores grow much faster than smaller ones, so the leader stands out.
- Share out the whole. Divide each of those positive numbers by their total. Now they all sit between 0 and 1, and together they add up to 1.
So scores like 4.1, 2.3 and 0.5 become roughly 84%, 14% and 2%. The highest score still wins, but now we can see how much it wins by. It’s called “soft” max because, rather than simply crowning the winner, it gives every class a share in proportion to its score.
How the clerk learns
During training, we compare those probabilities with the true answer. Think back to the archery of Lesson 6: we need a loss that says how far the arrow strayed. For classification, the usual choice is cross-entropy loss. Put plainly, it asks: “What probability did you give to the right answer?”
- Gave the correct sack 95%? Small loss, well done.
- Gave it only 5%? A large loss, and a strong nudge through backpropagation.
Over many letters, the network learns to lean confidently toward the right sack, and to be honestly unsure when the handwriting really is a mess.
What to carry with you
For classification, the network gives one score per class, and softmax turns those scores into probabilities that add up to one. Cross-entropy loss then rewards giving a high probability to the right answer.