Wherein Words Are Given a Place Upon the Map

Welcome back

Last time, we stood in the post office sorting room and met softmax, which turns a network’s raw scores into honest probabilities, one for each sack.

But there’s a question we’ve quietly skipped. Networks work with numbers. A chessboard, a photo, a house price are easy enough to turn into numbers. But what about a word? How do you feed “river” into a neuron?

The trouble with name tags

The simplest idea is to give every word a number: “apple” is 1, “river” is 2, “trumpet” is 3. But that tells the network something false. It suggests “river” sits halfway between “apple” and “trumpet,” which is nonsense.

A slightly better idea is to give each word its own switch. Picture a row of 50,000 switches, one per word in the vocabulary, with only the switch for “river” turned on. This is honest, but it’s a poor map. Every word is exactly as far from every other word. “River” is no closer to “stream” than to “trumpet.” The network learns nothing about how words relate.

Drawing a map of meaning

Imagine you were a map-maker asked to chart words instead of towns.

You’d put “river,” “stream” and “brook” in one little valley together. “Cat,” “kitten” and “dog” would settle on a nearby hill, a short walk from “pet.” “Trumpet” and “violin” would live in a far-off musical county. Distance on the map would mean difference in meaning. Close means similar; far means unrelated.

That map is exactly what an embedding is. Each word gets a position: a short list of numbers, like coordinates. A paper map uses two coordinates, east and north. Word embeddings use many more, often a few hundred, because meaning has many more directions than a sheet of paper. We can’t picture a 300-dimensional map, but the idea is just the same: nearby points, similar meanings.

Who draws the map?

Here’s the lovely part. Nobody draws it by hand.

The coordinates for each word are simply weights, and the network learns them during training with the same loss, gradient descent and backpropagation you already know. We start every word at a small, random position (remember the garden). Then, as the network practises a task such as guessing a missing word in a sentence, each word’s position gets nudged.

Words that keep appearing in similar surroundings (“I swam in the ___”, “We fished in the ___”) get nudged toward each other. Slowly, the map arranges itself. The old saying fits well: you shall know a word by the company it keeps.

Directions that mean something

When embeddings are learned well, even the directions on the map can carry meaning. A famous example: the step from “man” to “woman” points in roughly the same direction as the step from “king” to “queen.” Start at “king,” take the man-to-woman step, and you land near “queen.”

It doesn’t always work so neatly, but it shows that the map has captured real relationships, not just closeness.

What to carry with you