OpenAI has developed a tool to identify which parts of a large language model (LLM) are responsible for which of its behaviors. The tool uses a language model to figure out the functions of the components of other, architecturally simpler LLMs, specifically OpenAI’s own GPT-2. The tool attempts to simulate the behaviors of neurons in an LLM. It runs text sequences through the model being evaluated and waits for cases where a particular neuron activates frequently. It then shows GPT-4, OpenAI’s latest text-generating AI model, these highly active neurons and has GPT-4 generate an explanation. The researchers were able to generate explanations for all 307,200 neurons in GPT-2, which they compiled in a data set that’s been released alongside the tool code. Tools like this could one day be used to improve an LLM’s performance, for example, to cut down on bias or toxicity. The tool was confident in its explanations for about 1,000 of those neurons, a small fraction of the total. The tool identifies neurons activating across layers in the LLM. The researchers hope that this will open up a promising avenue to address interpretability in an automated way that others can build on and contribute to.
