Transformers are behind many of the most exciting recent developments in machine learning. However they are difficult to understand and most attempts to do so tried to dissect trained models. The goal here is the opposite: we will put the weights into the models by hand, so that we know precisely what they do.

‍

The family of models we have chosen here is very similar to GTP-2, but of course much smaller. They will be used to complete text, and generate patterns that are more and more complex as we go.

‍

The machine we are going to tweak is the following, where every orange bit is a parameter that can be changed.

‍

Pour avoir une idée de ce à quoi ressemblerait ces paramètres, lire : Induction heads - illustrated - LessWrong.