This post is converted from a Jupyter Notebook. To view the original interactive version, check out the Colab notebook.

Monthly Algorithmic Challenge (November 2024): Trigrams

Last week, I worked through the monthly Mechanistic Interpretability challenge from Callum McDougall’s ARENA course.

(A huge shoutout to Callum and the entire ARENA team for all the work they do!)

The challenge was to interpret how a simple neural net - in this case, a 1 layer 1 head transformer (with MLP) - solves a problem. The problem at hand was to predict the next token in a sequence of random tokens. As the model was trained with cross-entropy loss, training on a completely random dataset would lead the model to always uniformly predict all tokens in the vocabulary.

Learning about Information Theory can be fun! Join me as we explore what entropy and information mean, intuitively.

This post was inspired by, and heavily leans upon, the structure set out in “A Mathematical Framework for Transformer Circuits.” In it, the authors present a compelling way to decompose the transformer architecture into individual, more interpretable pieces.

If you haven’t read it yet, I’d recommend doing so. Most of what I present won’t make sense without context.

Motivation

A transformer’s ability to process long sequences of text is facilitated by multiple attention layers, which we can decompose into multiple attention heads per layer.

Hyperspaces are unintuitive, strange, and fascinating places. Let’s explore them together, interactively.

What are GANs?

Much ink has already been spilled on the class of machine learning networks called GANs, or Generative Adversarial Networks, so I will only summarize it here.

If you’re interested in learning more, this short course is a great resource.

Although replaced in contemporary applications by diffusion models for tasks like image generation, GANs provide a unique opportunity to study the interplay of two tightly-coupled systems, each seeking a different goal.