Attention
A 2017 paper replaced recurrence with a comparison of every token to every other.
Vaswani and co-authors, 2017, "Attention Is All You Need"
In 2017 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin published Attention Is All You Need. Several of them were at Google. The paper described a model, the Transformer, that dropped the recurrent step. Recurrent networks read a sequence in order. They pass a hidden state from one position to the next, which means a long document has to survive a long chain of overwrites, and which means training does the same chain in time. The new model lets each position look at the other positions directly. Order is added back in as a signal, not as the schedule of the computation. The schedule is parallel.
The look is attention. Scaled dot-product attention gives each position a query, and gives the sequence a set of keys and a set of values. The query is compared with the keys. The comparisons are scores. The scores are divided by the square root of the key depth, so that a high-dimensional dot product does not explode into a softmax that has already made up its mind. Softmax turns the scores into weights that add to one. The weights mix the values. The position receives a blend of every value, tilted toward the keys it matched. Nothing in that blend is a recurrence. The same blend can be computed for every position at once.
Multi-head attention does the blend several times in parallel. Each head has its own projections, so one head can learn to watch the previous word while another watches a distant verb, or a name, or whatever the training actually requires. The heads are concatenated and projected again, back to the width the rest of the layer expects. Around that core the paper puts the usual supports of a deep network: a feed-forward block, residual connections, normalization, and a positional encoding so the model is not blind to order. The bars in this catalog are not the whole model. They are an original sketch of the two functions, the scaled comparison and the multiple heads.
The paper is copyrighted. The bars are not a figure from it, and they are not its equations set in type. They were written for this catalog so the shape could be shown without copying the paper. A lawyer cleared that listing. The notice on the product page says so in one sentence: original sketch, not a figure, cleared. This essay uses words for the same shape, query and key and value, because a history that refused those words would be hiding the idea in order to look careful. Care is not copying the figure. Care is not pretending the mug is the paper.
What followed is why the 2017 date is on the piece. The architecture became the default for language models. Later systems changed the training data, the objective, the depth, the context length, and the way a person is invited to talk to the result. They kept the comparison of tokens. A model that writes a paragraph is not "the paper." The paper is eight authors and a result. The model is a descendant that industry scaled. Confusing the two is how a product caption gets ahead of the history. The caption on this site calls the paper the core of current AI, which is a claim about influence, not a claim that the PDF contains a chatbot.
The sketch is the right object for a mug because the paper is the wrong one. A paper has a layout, a set of figures, and a copyright line. Printing the layout would be printing the paper. Printing a new drawing of the two functions is closer to what the other pieces do when they turn a listing into bars. The difference, and it is the difference the notice exists to protect, is that here there was no listing to abstract. The authors published mathematics and experiments. The catalog drew a diagram of the operation they named. The diagram's author is the catalog. The operation's authors are the eight names on the credit.
Read the bars as nested repetition. One block is a comparison and a weighted sum. Several of those blocks sit side by side: the heads. The colors mark the same kinds of tokens the rest of the catalog marks, so a sketch written for the site looks like a sibling of the listings, not like a poster of a neural network with arrows and a brain. There is no brain on the shirt. There is a function, drawn twice in spirit, once for the product and once, in sentences, here. The paper remains in the proceedings. The clearance remains on the product page.