Homer Simpson brain scan ( source link )
Debugging One layer Transformer
Read this story for Free: Link
Quick Note — We will train a one-layer transformer + sparse autoencoder based small million parameter LLM, and then debug its thinking to see how similar the LLM’s thinking is to human thinking.
You can skip to the results section to avoid the training code and theory
Large language models like ChatGPT, Claude, or Gemini are incredibly powerful, but the question still hasn’t been answered yet.
how do LLMs think?
The approach since the release of open-source LLMs is simple, which is to read their architecture, as every LLM is made up of neurons. To study them means understanding what neurons are firing for a certain input, for example, if the user asks, “What is definition of sound?” what certain neurons are being fired for an answer corresponding to that question, so we can actually look at the thinking of an LLM.
To answer this, we will train a one-layer transformer with a sparse autoencoder-based LLM that at least speaks with proper punctuation and grammar. After that, we can begin exploring its inner architecture to see which neurons are activated for certain input…