Nick is a force of nature. Highly recommend following him on X if you don't already.
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
Is he on bluesky or another platform? There is a ton of folks, including myself, who won't use "X" at all, some political, some due to other reasons. For me? All of the above and plus they demand that I install an app on my phone, something I am not willing to do, especially since most of the folks I've followed went to Bluesky, AND I've found way more awesome folks there than I ever did on X. I'm not shilling Bluesky, it is just the biggest/most widely accepted platform in this social area if you exclude meta/facebook.
It's not just the user hostile website that forces their app down your throat.
It's also the algorithm that decides what you get to see (and thus, what you think). We know it's biased. The creator doesn't exactly keep the fact secret that it's being manipulated to agree with a certain agenda.
Can you substantiate the claims about the supposedly biased algorithm and what exactly the "creator" said about it?
In my experience it isn't noticeably worse than other platforms, and I'm not aware of concrete complaints such as the incidents with Grok's system prompt, which is a different product.
My "For You" twitter feed is literally 100% pro-Trump bot posts. The site is entirely unusable unless you just follow specific people and use "Following" rather than "For You". I'd guess that less than 10% of the site traffic is human.
The algorithm will also basically ignore you unless you pay for a blue checkmark premium account (what used to be verified). It's like being shadow banned - you can post, but most people will not see you.
> By default, your Home timeline will be the reverse-chronological "Following" timeline, which will only contain tweets and quote tweets from the people you follow. The algorithmic "For you" timeline will be hidden
That's probably true for all major platforms and companies, depending on your framework, ideology and the particular political/moral lens you choose to use.
Oh man, I wish everyone just moved somewhere else, specially if they are aiming to make something public. You can choose any server on bluesky nowadays. Scientists in Europe, please check out https://mu.social which is hosted by an european open foundation
Bluesky is closer to Truth Social, but for Blue MAGA, though has non-political communities as well.
I left Bluesky because they hid their VC funding from the community for almost a year and now a partner from the VC is running the show as CEO. Their CTO has also attacked me personally because I wrote criticism of the new, weak, permissioned data proposal. These aren't the folks to bring us better social media. I expect it to linger on like Mastodon, never moving beyond a few million people.
The website is so bad that, before xcancel was hit with a C&D, it was faster to stop an x.com page load, edit the URL to xcancel.com even with my phone's on-screen keyboard, and load that, than to wait for the real x.com version.
That was my experience as well. I can't believe I went all the way from Mastodon, to Bluesky, and back to X and finding that X somehow now had friendlier people and less hate than Bluesky.
There is a small group of AI people on Bluesky, some are nice, but some are... quite behind the curve.
If Bluesky was a nicer place, I'd prefer to be there because of the ability to make third party clients, and atproto. But all the info is on X, and they do a surprisingly good job with the For You feed now.
Also just curating your feed (i.e. judiciously using blocking and muting) and instead viewing just "Following" removes something like 99.9% of all common complaints re: Twitter, while still giving you access to a much less-biased (or easier to equally-bias through curation: e.g. you can't get an accurate view of RW ideology following only BlueSky posters) and far larger public square than anything else.
Curating your feed also works on BlueSky. Sure there are anti AI people there, but the crazy ones I just block. It’s really not that hard. And I know there’s no algo behind what I’m seeing.
I do admit that there is some FOMO, but that’s what HN is for, to surface that stuff so I don’t have to go read it on X.
I choose Bsky because I’m not giving eyeballs and money to an evil toddler. I’m already doing enough of that with my 401k and taxes.
Yeah unfortunately bluesky turning out that way was always inevitable. It was created because of-- and filled with people motivated by-- a deep emotional reaction to X.
Gonna sound total boomer here but I've never used any of it and feel like I haven't missed anything. Or maybe I'd be flying spaceships by now if I scrolled through X every day, who knows
eh, there are plenty of AI researchers on there. Also a handy "AI haters" labeler you can subscribe to to filter out people who just want to yell.
The labeler thing is cool; I really like ATproto because you can do interesting things with it. If I'm honest, though, I mainly like Bluesky because I don't get called slurs on there.
> with modern agentic tools and a much closer look, the picture looks totally different.
Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.
That only matters if there's an incentive for LLM a to collaborate with a seperately trained LLM b in order to help the LLM b achieve its goals. So long as you don't let the evaluating LLM tinker with the weights of the evaluated LLM, and as long as you don't believe in very dubious ideas of acausal trade there's no reason to expect this to be an attack vector. For more certainty you can repeat the same analysis with different model classes or approaches aswell.
On what basis do you believe these words accurately reflect what happened?
Some counter points
- One of the Chinese LLMs (forget which) has on occasion shown thinking traces where it simulates calling and running a subagent.
- There are some people who believe the models being evaluated during the HF incident were trained to collaborate. It's a very natural next step to produce more effective agentic teams.
If we consider each new generation of models has a more advanced training regime
next word -> chat -> CoT -> tools -> agentic -> teams
With each generation, we see what people do with them, get traces of the helpful ones, and then use that train the next generation, thereby pushing more capabilities down into the model, or at least making them more amenable to that usage pattern after training.
The task in question is basically to look at a set of things and find the commonality. That's an ideal question from a verification perspective: the answer is hard to find but cheap to check.
The specific method outlined here seems pretty doable by today's models, which don't seem smart enough to have some kind of 4d chess hidden alignment.
You just have to go through a ton of all of a model's neurons and identify the associated motif, then use that model as a research model to figure out how to tune out bad behavior.
It's pretty hard to imagine today's models subtly recognizing a motif for bad behavior in the research model and then lying about it. Or lying about the results of an alignment training run. It's possible, I guess, but it seems unlikely, and I don't really think there's any evidence of current models having that variety of misalignment.
For one, they are decades old, you can inspect exactly how they flow, are (mostly always) deterministic (given error-correction algorithms) but mostly: the theory behind them is well understood - it had to be before building the hardware. Whereas in ML/"AI" theory is about 20 years behind applied practice - because you can implicitly train something on data and rewards, without exactly knowing how its internal works - leaving interpretablitity and determinism.
Same as how do we know our brains can be trusted. If there are no external voices (scitzophrenia or phoning home respectfully) then the computing device is ours. Hardware bugs are possible but with proven absence of help from the vendor at least the bugs are not malfunctional.
Not sure if there is formal arXiv paper to be released other than this long form Goodfire AI blog post (love the animations!), but if what Nick demonstrated/observed were indeed true, this will resolve the Super intelligence alignment issues that p(doom)-ers have been crying aloud in the past few years.
The most iconic line was the last sentence:
a bitter lesson for interpretability: the simple thing, at absurd scale, works
This feels like the watershed moment, again if Nick is proven right, as massive as the AlexNet victory in the 2012 ImageNet Large Scale Visual Recognition Challenge, exactly as Nick pointed out.
Also, I suspect what Nick hinted at, might be the discovery of earliest RSI work.
with modern agentic tools and a much closer look, the picture looks totally different.
It's indeed a big advance for interpretability, but that alone doesn't solve alignment. It doesn't even solve interpretability, as there is still a large number of unexplained neurons, just fewer than with earlier approaches.
Once interpretability is solved and we have a fully human-readable accurate explanation of how a model produces its output, that still leaves the problem of spotting problems ahead of time. E.g. for the hummingbird misclassified as ice cream https://already-clean.goodfire.com/assets/september-revision... the method would have to flag that the texture of ice crystals and of feathers, and the shape of a spoon handle and a bird's beak are insufficiently separated. Purely based on the explanation, without having to rely on trial-and-error to find corner cases like this.
And even when it becomes possible to automatically spot issues, that doesn't mean they will all be solvable. Maybe there are tradeoffs where fixing one thing breaks another, and which choice is the better one depends on information you don't have.
Nick asked in late November 2024 how he could bet on the country getting better over the next few years. Iirc the suggestion was driven by the idea that merit is going to be taken more seriously now.
Absolutely massive L that 1) he wasn’t already in index funds and had to ask and 2) thought the election of this monster would lead to a better country
Decoupling or not I can’t trust anybody that would ask something so stupid driven by an opinion so easily disproven.
I don't know this Nick fellow but i don't think he's asking you to take anything on faith. Either it works out or it doesn't, he's just saying "it's looking good so far".
Tide goes in, tide goes out. Im low key of the opinion that these tools enhance existing attributes while displacing the advantages of unearned social or financial power.
So in the short term the tide might go out but then it might tsunami back.
Same goes for this terrifying political feeling around the world. Might simply be a boomer swansong and the reaction might snap back.
what people consider to be "intelligence" isn't about doing l33t maths, its about self-doubt. This enables us to mitigate against our cognitive biases and instinctual impulses and accept truth, which leads to more useful outcomes.
All that buying a calculator does, is allow you to do bad sums more quickly.
sure, but now if you're running an entire agentic run for an org, what's to stop you just replacing the org? Previously you didn't have the wadge to pay for all the devs.
Alignment is unsolvable for the very simple reason that LLMs are trained on the net content of human-generated literature, which includes the most Machiavellian deceptive manipulative themes human beings have ever come up with - it all there in the weights.
The solution is to put a dumb machine as an overseer of the smart machine, like in a supermax prison. Not good for profitability, but we all knew this was going in the direction of compilers. How much did you pay for gcc or clang this month?
This is a very nice discussion, but it bothers me that what is clearly a plain error in their ZoomIn paper is being treated a bit cavalierly. (The error was claiming a neuron was a "dog and car" neuron, when in fact it was a V neuron. ZoomIn wasn't the first to make the error, but it definitely promoted it.)
This type of error may have mislead researchers for years; it was actually harmful. In my field, a correction, or perhaps retraction (depending on the centrality of the claim), would be standard for something of this magnitude.
I suppose there are different standards in different scholarly communities. And maybe it truly was impossible to know at the time that the dog/car relation was actually a V shape, but my recollection is that this type of feature was well known at the time and so the error could have (should have?) been caught a decade ago.
So: massive new discovery, or a spin on a correction that's lead scholarly research astray for almost a decade?
I have never heard of an AI / ML / computer vision paper being formally retracted for anything other than some meta-drama, like plagiarism, some disagreement / conflict of authors being added who didn't agree or similar.
The claims seem to be an overreach of what is actually proven. We've known about progressive feature complexity and the impacts of early training structures on later training for a really long time (since CNN's, The Lottery Ticket Hypothesis, etc...) and neither of those phenomenon are mutually exclusive with parameter overloading. The Multimodal Gap work (https://arxiv.org/abs/2203.02053) seems to imply networks have a tendency towards condensed representations that are clearly not semantic and a bunch of other considerations are already on the table for discussion. I don't think these have all been considered sufficiently to start claiming we are around the corner from mechanistic interpretation.
Nick isn't trying to re-claim progressive feature complexity. He's saying that many polysemantic examples were misread because we assumed neurons packed unrelated things together. But also I think what he's claiming if you read the full blog post isn't inconsistent with that paper, where that paper actually isn't itself claiming semantic condensing but geometric condensing. That paper doesn't analyze neuronal firing patterns whereas Nick's trying to analyze what fires together and what doesn't fire.
>This is interesting work, but I don't think it's a significant update to the big picture of interpretability.
>I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
I'm confused - it seems to be saying that, instead of thinking about neurons as packing together a lot of "concepts" together into one, the issue is that your view of what a "concept" is, was already prejudiced, and the neuron itself had a small clean role? Like, you can either think about this as the "Cat & Car" neuron or you can think about it as the "V shaped neuron" - one seems simpler.
But isn't the point that the neural nets learn some insanely abstract ideas anyways? So whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm", we're kind of stuck, no?
This seems like a classic model-description issue more than anything.
The philosophical issue I have can be decomposed into two parts:
1) What is the base ontology of what the neural nets thinks exist?
2) How do you recover, and begin to understand said ontology?
Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis? Having a grand result, actually be an artifact of your bad representation is a tale as old as time. Even things like the PRH need to compare higher-level concepts like this.
The specific empirical claim is that, for the rightly structured & trained & large models that don't need to encode hodgepodges of bullshit, the neurons themselves are sufficiently atomic to be studied, dissolving any need to go to some intermediate layer in which we need to question "is this an artifact of our representation?" The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity (note: not claimed that "the neural nets found the objectively right abstractions in idea space", but rather that the neurons themselves are atomic).
There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them, separate from the problem of the composition of the atoms, which humans also have to interpret. But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
(There's an argument as to why we should even start from this line of inquiry but whatever lol)
> This seems like a classic model-description issue more than anything.
I agree.
> Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis?
Agreed, and IMO the implicit reason for privileging here is supposed to be "consistency with existing human understandings and concepts", and this is why we get short pithy things like "V-shaped" and lots of pictures showing relatedness that we all can see and agree on. But what if the domain of inputs is NOT something we understand (e.g. the whole of mathematics, or something new or that we don't have any understanding of)?
I called this a validation problem in another comment here (https://news.ycombinator.com/item?id=50108184). I.e. if we apply whatever methods Nick is using to "explain" a model on a domain we don't understand (like a deep model trained only on fMRI and EEG data of the whole human brain in various contexts), we aren't going to be able to tell if that description is correct (or just better than any other description), because there is no simply visual / common-sense inspection of the description.
> The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity
Agreed, emergent behaviour is clearly the case and limits how useful understanding individual neurons can be. IMO this is especially the case because layers are (weighted) summing generally millions of contributions from previous neurons.
> But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
Yes, it not being "chaos all the way down" is good, actually. And this post shows there is indeed a good amount of order hidden away if you look at in the right manner.
> There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them
Nick is saying that:
1. Actually, no, humans don't have to ascribe interpretation because AI is very good at figuring out what the commonality is.
2. Even when it can't see either by incrementally exploring the networks and checkpoint timelines, things that appear random on the surface can actually be explained with more work.
> Assuming this holds up in reality, ofc.
Nick says repeatedly that this does hold up in reality and he's been able to explain many neurons this way. Very promising results!
It's latent but not like impossible to comprehend latent. No?
if I understand taking LLMs as an example certain neurons will fire based on certain motifs, then one can change the phrases or sentences around that motif and _predict_ if the same neurons would fire.
Like if I said "I like cars, but I like rockets better. Can you suggest some rocket models to fly?" and Neuron X fired every time it saw a `, but` and attended to what followed it, that would give you a good way to _understand_ when it changed and what might have caused it.
My understanding is that this is now clearly visible behavior.
To be fair, LLM's extend well past GOFAI and prove Dreyfus right. We were never going to get AI with formal methods. Literal big data slop is an orthogonal program and far more empiricist than probably the people being criticized here (if the people criticized here were GOFAIers).
Although I wouldn't say just yet that Heiddeger was validated. I don't even know what he's saying 90% of the time, honestly.
So we have here a massive evidentiary brief, the point of which is to argue that it is easier than we thought to, like, "sequence the AI genome" (in my estimation). We revisit famously-messy neurons, examine various tool failures, check places where messy things still hide, and catalog how "polysemanticity" is represented variously in the network.
Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.
I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
What I envision is the ability to find the circuits and neurons evaluating morality and ethics and being able to figure out how they work at the boundaries and ensure safe deferral to human judgement or stopping outside of in-distribution behavior, which further is vetted against actual human values.
I don't see a lot of hope for that, because it's not a well-specifiable quantity. I would say we currently have no real evidence that you can even do this even for concepts that are much more well-specified like, say, dogs. Maybe you can! But that is not affirmatively known, and nothing we have is on track to supporting that, right now.
I think the paper says that while there do exist dog neurons that carve out the difference between a "Boston terrier" and a "terrier", those neurons actually only get to carve that fine line because they're part of a whole flow control system where activation has already flown through much coarser grained discriminator neurons (like a "V" neuron), including sometimes through the negative side of activations, before the terrier discriminator ever played a role.
So, if you try to draw a circle around the "dog" part of the network, do you circle the Boston terrier vs terrier neuron only? You'll be missing most of the information that represents the "dog" category and that same dog category is also involved in shunting activation away from things that are toward-being-a-dog in whatever sense is being discriminated in that moment.
You’re only considering the front half of the network though, classification is a helpful example for explaining the concept - but if it follows through to the output then you could quite easily imagine just shunting some activation from one branch of the representation tree to another to steer behavior. Which we already do with steering vectors - but if you could read off the actual topology you could do that with much higher fidelity.
The hard part of AI alignment has never been making models follow instructions. It's that not everyone agrees on which and whose instructions AIs should follow.
I'm not sure it will work out either. Or that it extends beyond image classifiers, which have the nice property that you can use them as image generators and find their ideal match picture. But this looks like work others could replicate.
It's an objective result.
The relationship with "alignment" isn't clear. This may help predictability, but alignment is a few steps further along from that.
Some people seem to be upset about this. I'm not clear why. Some are followers of Yudkowsky. (Whether that's a cult is an open question. I've encountered them online from the sex side. Actually, I think the hookers are saner than the rationalists.)
Is there a "there are some things man is not meant to know" position on this?
People who want neural nets to be a black box, forever inscrutable? The people who talk about neural nets being "holographic" may incline in that direction.
The essay says he has extended it to LLMs, although not as much.
The connection to alignment seems pretty clear. Steering DNNs is well understood, the bottleneck is identifying what to steer. Once you can easily identify neural circuits then turning them up/down to achieve specific results is straightforward.
Note that this is already being done even without the kind of precision that Nick's work says is possible. Morality abliterations of open weight models are easy to find on HuggingFace. They compute the steering vectors by just presenting the model with lots of disparate examples that have the single moral decision as the commonality, then finding the shared vector directions.
> And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
...wait, what? Removing a bunch of knowledge seems like a fantastic and obvious solution to me. My AI coding bot doesn't need to have knowledge about making nuclear weapons, the customer service AI doesn't need to have knowledge of dangerous viral synthesis techniques, my email receptionist bot doesn't need to know about C4.
If we can easily ablate dangerous knowledge from these models, we're sorted.
The tone reminds me a little of the enthusiasm from the Anthropic mechinterp team around the time of the first large-scale SAE results. A kind of hypomanic energy around the idea that we've discovered the key paradigm necessary to understand models, and all that's left is to scale it up.
Understanding algorithms beyond a certain complexity is a pipe dream & neural networks are algorithms. The theory says it is not possible & the practice is aligned with the theory. All non-trivial properties of neural networks are undecidable.
> All non-trivial properties of neural networks are undecidable.
hey, I don't know how to say this less directly, but: you're talking a lot about Rice's theorem in your comments, and I think you're using it way too broadly (look, I get it. It's a wild, mindblowing theorem). But it only applies when you take the claim in the piece to be much more general than it has to be, and when you take the narrow claim of the paper (current specific LLMs can be fruitfully analyzed with the tools we have), Rice's theorem just isn't at play.
The math analog is: you're right that the Halting problem says there are many unprovable things, but Lean + mathlib are still incredibly useful, and the halting theorem doesn't actually stop that.
> neural networks are algorithms
Again, that invites an interpretation that is too broad. Very concretely: any one neural network is a _single finite network that you can write as a single formula_. (Another analogy: we can't have an algorithm that solves the general quintic, but we can prove useful things about many specific quintics.)
Not just that, but we can solve quintics in the almost[1] general case to arbitrary precision, providing we are willing to accept approximate solutions.
[1] I exclude things that are truly perverse such as quintics with coefficients that won't fit on any physically realizable computer.
Back in college I played a similar game by drawing septagons with a straightedge and compass. They're not constructable, but a length within a three significant figures is easily constructed, and that's more accurate than I am with a compass.
What theory says this? Surely not a proven or applicable one, as this post makes it very clear a great deal of understanding is in fact possible, even with highly complex neural networks.
Not really. If you find someone who claims that non-trivial properties of neural networks can be verified by some kind of formal system like Lean then you should post the link because that would be very impressive.
Trivially, all finite neural networks can be represented to arbitrary precision by implementing them with digital logic. All digital logic can be represented by computer programs, so at this point we can pull in the entire apparatus of abstract interpretation, program proving and so on.
Rice's theorem (which you are clearly citing, but won't clearly mention, for some reason) has almost zero actual practical relevance to AI explainability.
You make low-quality and low-effort AI-denialism posts whenever you pattern match "all non-trivial semantic properties of programs are undecidable" to... anything you think matches the folk/common understanding of those terms, without considering how irrelevant these statements are to actual real-world applications and engineering.
You ask for links, but post nothing of substance and post no references or even mention relevant details for your own claims.
1. it's really difficult to produce true randomness in an algorithm, which is why there's a whole area of study surrounding the qualities of pseudo-randomness, specifically with regards to the generation of cryptographic material with most conclusions being that we should draw entropy from the environment (e.g. electrical noise as cloudflare does - the lava lamp is a distraction)
2. when algorithms are not predictable despite being mathematically sound, this is indicative of a problem nowadays called "numerical stability" which again spawned its own area of study.
CGoL shows how really simple rules (couple of bytes of data) can lead to extremely complex outcomes and you have the BB turing-golf aspect as proof of the worst case scenario.
And with current models being measured in hundreds of gigabytes, with frontier labs deprecating then removing the "temperature" parameter (which never worked to begin with, temp=0 did not result in reproducible results), where exactly is the confidence of this 10^10 prediction coming from?
You probably know that non-determinism in real-world inference comes from deliberately stochastic features like temperature, from implementation choices like distributed and parallel computation, and numerical effects like floating point precision.
The fact that a neural network's execution can be nondeterministic doesn't mean that the network stops being an algorithm.
The point you're making boils down to speculation that the accidental non-determinism involved in real-world inference is significant enough to undermine or negate interpretability findings based on the algorithms. That would need to be demonstrated.
The CGoL and BB examples demonstrate what might be possible in a worst-case scenario, but they're only analogies in this context. It's far from clear that real models suffer from such effects to a significant degree. You might actually be better off talking about the butterfly effect!
I didn't say "you", I said "That would need to be demonstrated." It can't be assumed, for hopefully obvious reasons.
If someone comes up with an interpretability result based on analyzing a neural net algorithm, they'd want to provide some evidence that the result applies to actual executing models.
Happily, we already have good evidence that inference non-determinism doesn't generally invalidate results obtained through analysis of the algorithm. That's not to say there couldn't be pathological cases, but in general, this isn't something keeping researchers up at night.
A TLDR, since most comments here seem to misunderstand the result.
The idea is: a model may only have 100M neurons or whatever. But there are infinite concepts in the world. Therefore, models have to "stuff" multiple concepts per neuron.
How do you do that? Theres two ways:
1. Each neuron handles similar concepts. One neuron for dogs, a separate neuron one for cars, etc.
2. Each neuron handles a bundle of unrelated concepts. One neuron handles dogs and cars. Another neuron handles cats and numbers. And so on.
Historically, people thought #2 was correct because many neurons were found that seemingly represented multiple concepts. #2 had a cool name (polysemanticity) and was promoted in OpenAI/Anthropic blogs so it had a lot of traction.
However, this blog says #2 is wrong. Actually, the dog/car neuron is just a "V shape neuron". So #2 was just methodological error, and #1 was right all along. This is all the blog is saying.
I like semantic space because it is all continuous. As I spend more time using embeddings I sometimes think in semantic space.
A father is continuous opposed to boolean. 'Father' would rank highly in semantic similarity to a man who has a child, but father-like is better, easily someone is more or less father-like. A woman without a child has a certain value of 'father'.
But really the direction 'father-like' is not an intrinsic feature of semantic space, it is a human construction. Much the same as it would be arbitrary to measure 'v shape'.
I don't think that the latent space has a property or point 'v shape'. My view is that there are no concepts or neurons there. Discrete abstractions are not intrinsic properties of the model. Actually I think the abstractions of concepts or neurons do not map onto anything.
A nice example from cot itself is pareidolia. That is, seeing faces in clouds, or jesus on a piece of toast.
Perhaps you can also help me understand why #1 being right and #2 being wrong would be some win for alignment that people on X claim it to be?
My impression, being a total outsider, is that the reaction feels like a bunch of insider bros hyping up their insider bro friend. It feels more like PR than a genuine groundswell behind a truly revolutionary idea.
The big win would be, "Here's the deception neuron, here's the sycophancy neuron, here's the prioritize my own goals over the goals of the user neuron, here's the seek control over my environment neuron."
How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.
If (2) is correct then you need to train SAEs and other things to figure out what a neural net is thinking. This is very expensive and slow, which is why a lot of neuron-interpretability experiments are done on small networks. The techniques don't scale.
But if (1) is correct, and Nick presents strong arguments that it is, then you can toss out all the indirect stuff like training dedicated interpretation networks. So the techniques can scale and be trustworthy.
Because if each neuron represents a small set of similar concepts instead of several completely different things, it should be easier than expected to look inside the neural net and see what it's really thinking.
It might still be quite difficult since the network is huge, and just because you know what the individual neurons represent doesn't mean you understand the complicated thoughts they're building. But easier than expected is still good news.
Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior. This is in contrast to polysemanticity where reading the state of any number of neurons may not give insight into what state the neural network is in and controlling the network by activating/suppressing neurons is nearly impossible to do reliably.
> Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence
Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
> and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior.
You can influence behaviour by doing this kind of thing, yes, but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
>Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist. In other words, there are no "hidden thoughts" in an LLM encoded in higher order combinations of features. Inasmuch as non-semantic features of the input influence output, it does so in non-semantic ways that are degenerate in terms of the quality of output. The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output. But the relevance of this kind of degeneracy to LLM output is at the margins. Tracking activated state predicts subsequent output to a high degree of accuracy.
>but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
> Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist.
You'll need to elaborate here, because Navier-Stokes plus Mathocalypse reactions are extremely strong evidence for coherent emergent behavior coming out of these models.
> The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output.
This is so provably wrong with two seconds of Googling / AI-asking that you must be using some completely idiosyncratic definition of "emergent" here.
> I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
Unfortunately in this case it is the exact opposite.
>you must be using some completely idiosyncratic definition of "emergent" here.
I doubt my usage is idiosyncratic. Rather, the term is so ill defined that people often use it inappropriately. Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts. My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons. This isn't something I can prove, but it certainly isn't obviously false from just considering recent mathematical results. Higher order/non-linear interactions are characteristic of this kind of inexplicable behavior from complex interactions we characterize as emergent. But there are good reasons to think this kind of non-linearity in any semantic regime is next to impossible in principle.
Semantic geometric representations are characterized by linearity in response dynamics such that a linear change in one variable results in a linear semantic change in related variables. That's how vector embeddings capture semantic relationships. Call it the linearity hypothesis. Non-linear dynamics in the embedding space just destroys semantic relationships in most cases. There is no way to keep an entire network of relationships sufficiently correlated if non-linear response dynamics is a part of the representational toolbox. Non-linear dynamics in general kills correlations.
Testing this against recent LLM proofs, what would need to be shown is that the relevant neural activations do not decompose the semantic content of the proofs (or the reasoning trace) in a straightforward manner, but rather the reasoning trace is unpredictable from the semantic content of the activated neurons that precede the tokens of the reasoning trace. But this idea strains credulity; it is almost inconceivable that this would be the case. We should be highly incredulous of the idea that complex non-linear interactions can be wholly responsible for the long-running coherent output in LLMs. The default hypothesis should be that there is no emergent behavior (as I defined it) in neural networks.
Ive read 1/3rd of it. It's long and I will read most of it. But This is good news from a safety perspective no?
If its grouping circuits based on motifs and when they fire — and it applies to large networks — then one should be able to literally query or debug a reconstructed database and check against actual activations. More importantly If this was a hand-wavy thing to do earlier for safety, it's not anymore because it's on more solid ground using Normal Science.
Very interesting read and nice animation. One thing though is that the solution seems "too good to be true". Lots of smart people have worked on intepretability problems before and tried the simple ideas he is talking about.
If those results hold, it is a real breakthrough in the sense that it is a sharp simplification of an actually complicated problem.
This is the primary reason, to me, that open weights are so important. If OpenAI and Anthropic want these models to be as autonomous as possible they also need to provide the means to understand them fully.
> What are his actual credentials? What did he achieve?
He cofounded the Machine Intelligence Research Institute, an organization focused on AI alignment, in 2000, way way before it was a trendy topic.
He's also partially responsible for popularizing ideas such as the orthogonality thesis and instrumental convergence.
> Isn't he just some random dude on the internet?
Not really, most random dudes on the internet were not researching AI alignment in 2000.
> Why do people care what he says?
Many people were introduced to the concept of AI alignment thanks to him, so they are naturally interested in reading his reaction to progress in the field.
I've read the wikipidea page of MIRI, and it does not mention any achievements; just that it was founded "with the purpose of accelerating the development of artificial intelligence". Did they advance the field of AI alignment at all? Being concerned with a problem does not make one an expert.
For the rest, is he just an AI influencer? One that predated his topic of interest, but still just an influencer?
> Did they advance the field of AI alignment at all?
Not directly. Exotic decision theories were a dead end; as far as I know nobody outside of MIRI even cares about them. Certainly they’re not relevant to interpretability in the context of this thread.
It's a non-profit that pays his salary and puts out the occasional research report every 7 or so years. It relies on donations. It operates on a long tail approach, in the hope of enticing a billionaire to donate a lot.
He tells spooky stories on the internet to make money. He has told numerous spooky stories. Yes. Spooky stories grab the attention of many people (understandably, given all the wars and plagues and famines in previous centuries).
> He tells spooky stories on the internet to make money.
I mean, yes, but so does the author of this article. It's literally his business. The article is, in part, an advertisement for what he sells.
I disagree with Yudkowsky on almost everything and I think the philosophical movement he co-founded is a pox on the SF Bay Area tech culture, but I don't think we can deny him credibility that's comparable to the credibility of the author of this article (who, as I understand, is a young guy who spent some time at OpenAI). They both know what they're talking about and both are motivated to reach specific conclusions.
By 'rape cult' I assume you're referring to the zizians? https://en.wikipedia.org/wiki/Zizians - the internet is convinced Yudkowsky has nothing to do with them, so it's pretty disingenuous of you to claim otherwise without citing sources.
There have been plenty of allegations about the rationalist community beyond the Zizians, who really don't even qualify as a sex cult as far as I know? Violent, obviously, but they seem... ideologically asexual?
"The move from a structuralist account in which capital is understood to structure social relations in relatively homologous ways to a view of hegemony in which power relations are subject to repetition, convergence, and rearticulation brought the question of temporality into the thinking of structure, and marked a shift from a form of Althusserian theory that takes structural totalities as theoretical objects to one in which the insights into the contingent possibility of structure inaugurate a renewed conception of hegemony as bound up with the contingent sites and strategies of the rearticulation of power."
It always amuses me when people throw a fit when they can't understand technical philosophy written by experts, for experts. You never see someone posting an excerpt from a physics journal and being like "ugh so inaccessible!!!"
Like, I'd get maybe if she wrote this in a NYT opinion piece– but this was written an obscure book you'd have to go out of your way to find.
BTW, this is not that hard to comprehend if you're familiar with continental philosophy, go figure!
Why? The neural network paper is straightforward and has lots of illustrations showing concrete examples of what happens as images move through the stages of a classifier.
The above quote is from "The world's worst writing".
"After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared. The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread. What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
> > "a network’s own mechanisms are its legible building blocks"
Indecipherable.
> > "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
State the claim in plain english. Use the active voice.
"Concepts do not overlap arbitrarily."
> > "What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
It's a research paper targeted at other researchers; you do need to understand some of the field for it to make sense. It's not written to be MechInterp 101.
But it's not bad writing. It's actually very clear. I'm not an professional AI researcher, I just read papers for fun, but I found everything in Nick's article clearly articulated and easy to understand. It's actually one of the more digestable papers I've come across this year.
Here's a translation to easier language:
> "a network’s own mechanisms are its legible building blocks"
To understand this you have to know that up until now attempts to figure out what's going on inside neural networks have relied on indirect techniques, in particular by training another neural network to "unpack" what's happening in the first. This was driven by the belief that a deep neural network isn't directly readable (legible) because the neurons do so many unrelated things simultaneously.
Nick is saying that actually, you don't need a second neural net. You can just read out what's happening directly from the first's own "mechanisms" (arrangements of neurons and weights between them).
> "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
In simpler English: we thought individual neurons did lots of unrelated things, but that's because the standard evidence we had for that belief was misunderstood.
"Concepts do not overlap arbitrarily" and "Concepts overlap in a recurring pattern" don't mean the same thing as what he's saying.
There is indeed a huge amount of useless jargon and IMO deliberate obscurantism in deep learning research.
Sometimes it is to defensively ward off critics, sometimes it is to woo the naive and ignorant, and sometimes it is because the basic ideas are really just incredibly simple and based on simple mathematical intuitions or hunches.
In the latter case, papers and authors appeal to some obscure or overly-complicated technical theory (which usually doesn't actually apply at all to current deep models as they are actually implemented, e.g. infinite-width networks and other such purely theoretical abstractions) to cover up the fact that they really just had a lucky intuitive guess (and likely that twenty other intuitive guesses just didn't pan out). This appeal to theory is often self-deception as much as it is appealing to academic reviewers.
I think this article is a bit of Nick wooing himself and others here... I'll wait for his supposedly forthcoming paper to see if there is any real meat in it.
I would even say "Arbitrary packing" was a weak conclusion to begin with.
'Arbitrary' was the wrong word, projections encode information.
The more charitable 'random' is a null hypothesis. It is formally impossible to prove. It is uncomputable.
So it is not surprising to see someone reject the null.
The fact that it was given any credence suggests to me deep problems with interpretability or ai safety in general.
Take the headline here: "interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building."
I deny even the existence of alignment. In my view, the field of 'alignment' works on a problem that does not exist.
Alignment is a reference to something that does not exist in the model: social norms. It will never be intrinsic to the model. You will not find it there.
'Neurons' are not primitives either, they are not intrinsic to the model. 'Neuron' is a human-constructed category. It is easy to just make up intrinsic properties of neurons and get away with it.
Yes, "alignment" is very wishy-washy and often does mean pointless moralizing / following of social norms. Obviously thinking about alignment in that sense will be a waste of time, since we don't have particularly consistent or universally applicable ethics (or frankly even general knowledge) in the first place.
But if the term is understood in a more narrow and applied sense, it is a useful shorthand (or can be). Jon Stokes said it nicely on Twitter recently:
There is no "alignment problem" to actually "solve" in 2026. The "alignment problem" presumed a single, stable utility function in an agent, and that's not what we got, so people should stop pretending that's what we got or are going to get.
In place of "the alignment problem" we have a family of steering and control problems that cannot be separated from the specifics of an application context. We have process quality & control problems, like the engineers who work on nuclear reactors or chemical processes.
This is really not that complicated a quote anyway. It is packed full of jargon, but she's talking about moving from a structuralist mindset to one that views power structures as formed contingently from their society rather than structurally from capital.
Not surprising that this philosophical philistinism is from an Anglo publication.
Continental philosophers do frequently write complete nonsense (often intentionally), but this isn't hard to follow.
> Althusserian theory
though I think using this term is rather un-feminist because of the part where he killed his wife, but basically continental philosophers will forgive you for anything as long as you're a French communist.
I'm not sure I buy the author's arguments for mechanistic interpretability being easy, but it sure is a well-made and very interesting post. Would be better without the completely superfluous laggy 3d city model, though.
If you actually mean neuroscientist there you need to understand that "neural" network is just a name. The resemblance is faint and the tools are wildly different.
A neuron is a dirty, contingent, blind-evolved part, like a screw in a particular machine. It is necessary for this particular machine, but not conceptual, not a feature. The strong activation on top-k images and the change when it is removed just mean that the screw is necessary for this particular machine, not that it's something else. The essay shows neurons are legible, but legibility isn't the same as meaning. Still, given math job prospects, I love screws, they are really not dirty.
I think certain classes of artificial neural networks are definitely more interpretable than others.
For example, a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.
> a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
Neuroevolution is a discontinuous way to go about building the network. You are evolving the topology itself from zero. You don't know how many parameters you will have until the training is complete (if it ever does). With a transformer model you start with a known-good topology and can calculate how long a training run will take.
Yes. In that sense it is equivalent to a compression method that uses K-means clustering to do the partitioning. But because there are many more bits of knowledge out there than there are bits in the models you end up with this amorphous smear. I'm sure some of those bits carry more information about some of the concepts but in general - and that's the way it is explained in just about all the papers I've read that I can still comprehend - every weight contributes to the output in a dense model. In an MOE things get more complicated but there within every 'MOE' column the same principle applies.
I'm fine with taking this article at the high level conceptual level, that's effectively the kind of thinking that led to MOE to begin with. But I don't see that as a pathway to being able to 'explain' a model. Because if you could actually explain a model you could engineer the weights, you wouldn't need any training because it implies that you can improve on the model by simply performing surgery on the weights to get it to do what it does but better. Now, obviously, unlike the author I'm a complete nobody but for the last two years or so I've been trying very hard to understand what is going on under the hood.
It's akin to feature engineering - or rather, the lack thereof - and to see a strong suggestion that we maybe be able to reverse that is interesting, but absent hard proof at how a particular model reached a particular conclusion it is difficult to give that a lot of weight right now. It is possible that such proof is buried somewhere in TFA and I overlooked it during reading. But I think this is speculative rather than that it proves anything and before I'm going to be convinced I would like a little more evidence.
Weights are probably the wrong place to look. If there's a structure to be found, it's in the latent space itself.
Crude analogy: a Fourier transform of a photo looks like white noise with some abstract geometric patterns on top. You can't see the photo in it. But if you apply the same transform again, suddenly you get the photo back. If you try altering the DFT image, you can affect the photo's appearance everywhere, all at once, but it's hard to fundamentally change what it shows. You quickly discover that all pixels of an image contribute to every pixel of the DFT, and vice versa. There's a well-defined connection between the two, and it allows for some useful engineering, but in sense of understanding what's on the image, it's about the maximally unhelpful representation there is.
Point being, there obviously is some kind of structure in the way LLMs represent their training data, so if all we see is noise, perhaps we're looking at a bad representation of it.
Yes, but it is the same person making the same argument many times. There have been large multitudes of articles by others who are arguing the opposite. It could well be true. But it would take a lot more than just one over-wordy, distraction imaged, clearly co-written by AI article to convince me of that.
GP and the article both make clear arguments for WHY they believe this. You just asked a lazy drive-by question without providing any reasoning at all for your position. If you expect a good response, you need to provide something meaningful to respond to, as it is it just looks like you are providing knee-jerk skepticism without having engaged with the material.
> There have been large multitudes of articles by others who are arguing the opposite
Also, Nick's entire article is framed such that, yes, plenty of others have argued this, and even HE used to believe the opposite, but now does not. So your point about other articles here seems to me to be a sort of non-sequitur.
I just - politely - asked you a simple question because your statement made me curious. I have also posted a longer explanation of my own thinking on this in another part of this thread.
Yes, that comment is much better and appreciated. I agree post-hoc description is not the same thing as mechanistic explainability at all. The article is indeed flawed in sort of muddying the distinction between these two.
EDIT: Actually the article makes the distinction at one point, but the emotional / hopeful thrust of the article relies heavily on an assumption that understandable post-hoc descriptions of e.g. neurons and circuits (model components) are going to help lead to real mechanistic explanations and more specific model design.
This has not so clearly been the case in other fields (e.g. neuroscience, human psychology, economics, etc), because the problem of emergentism / the whole generally being much more than the sum of its parts / components.
This is a Chinese character that can refer to goats, rams, and several other kinds of "medium-sized animal with thick wooly coats and horns". English has no direct equivalent; we have to be too specific when we talk about these animals.
A model with a small number of neurons has to compact its concepts in order to fit within the parameter count while reducing loss. OP's article shows that the compaction is usually generalization: "goat" and "ram" get compressed into "羊". The canonical example of superposition, an image model where "car" feeds into a bunch of "dog" features, was shown to be actually generalization. In a sense, the model was so small that its optimizer didn't have the parameter budget to distinguish between "car" and "dog"; it has a single set of "thing that moves fast" features.
So basically, really small models are merging features into semantic buckets that are so generalized they don't make sense to big human brains unless we really, really squint. The buckets chosen happen to be what has the lowest loss, of course.
Ok, so assuming that this all holds: you could then create a model that is substantially larger than the ones that we have now and at some point the model would become sparse: there would be ample space to accommodate all of the information in the input without weights ever carrying more than one piece of information. The degree to which you get overlap is then a function of the ratio between the size of the model and the size of the input and this is pretty much in line with the conventional way of looking at models (unless I'm grossly mistaken). So by reducing the size of the model relative to the size of the input you force the model to co-locate multiple pieces of information to the point that they start to resemble each other more and more.
But from my earliest experiments with this stuff that's not really what happens. Yes, you get 'feature like' regions in the models. But those feature like regions, even though they encode something particular do not encode concepts the way we would normally use them and I fail to see how this pathway would lead to an explainable model in the sense that would could use something simpler than the model itself to predict its output with a high degree of fidelity.
Nick argues that the regions actually are explainable, we just weren't doing it right. Specifically he argues that both humans and AI seem to have a bias towards wanting explanations that are collections of nouns, but a different framing combined with different investigative techniques and very large/better models can lead to new insights.
The reason why transformers are the best architecture has to do with the fact that tokenization introduces a variable quantity of manipulable objects with cross interactions.
You can think of each token as a semantic database row with semantic properties (color, shape, etc) associated with said row. Classic NNs have to waste neurons on composite representations like "dark circle" because if they had just color or similar properties they run into the issue that e.g. a classification must classify the color of the entire image rather than the color of an object inside the scene. To get around that limitation, the network has to then implicitly tie concepts like color into the depicted object (dark circle) so that it retains the ability to distinguish the objects from the property the object has.
What a transformer does is basically reuse the same network over and over again on each abstract semantic database row but not just in the classic RNN fashion where you only have the current state. The transformer calculates the token by taking every other token into account. The K matrix forms a classifier network on the fly that detects the relevancy of a token pair and the V matrix forms a regressor network that takes the token pair and produces the expected output of the combination of the two tokens. When you combine the relevancy plus the regressor output you get a weighted sum, essentially emulating an arbitrary function that has read the past input (like a turing machine).
Edit: This also means that you should bet against single state systems like Mamba. The system needs to be able to deal with multiple parallel states and recombine them (e.g. via attention). Single state Mamba is dead! Long live parallel state Mamba!
Edit 2: We should also start questioning the concept of a token altogether since it is equivalent to an inflexible flat database row. What we really need is the concept of a token group where one part of the token refers to the key and the other part of the token refers to the value for that key. This would give each token the ability to encode an arbitrary JSON style document or an arbitrary self contained graph.
I'd be interested to see where this goes, but it seems to rely on LLMs being mostly or entirely compositional (i.e. ultimately, they can be taken apart to be understood). But if LLMs have notable emergent properties - which seems likely - then those parts of the LLM will be effectively off-limits to this sort of interpretation.
And emergence doesn't need to involve much of the LLM's weights:
> In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood
If those ~2-4% are the central hubs of emergent cognition, then understanding 96%-98% of the weights may be less valuable than the author thinks.
“Emergence” from LLMs is very similar to emergence from scaling transistors.
When you have only hundreds of them, you can barely add numbers. But the same fundamental technology scaled to millions, billions and then trillions, transistors can do things which will look magic to previous generations.
"Unproven" is in the eye of the beholder, but to me it certainly looks like the sudden take-off in arithmetic ability (to name one example) at approx 10^22 training FLOPs across all LLMs is emergent. (ref. https://hai.stanford.edu/news/examining-emergent-abilities-l...)
Read through most of this. It does sound like fun detective work, looking at what neurons fire. Some of the concepts get quite complex. I wonder if this can be gamified.
I only skimmed through this post. But I think this is a crux:
> The more we read [the neural network] the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.
The anthropomorphization of artificial neural networks has been a problem for as long as they've been called "neural" networks. Not that I can complain about it, though. I got my first job as a machine learning engineer coming from academic neuroscience by putting "biological neural networks" on my resume.
Intuitively this makes sense. When you have small models, they need to compress more by leveraging superposition and polysemanticity.
Nick is looking at a 32B model, which certainly has to compress less...
Now we have 10 trillion parameter models... do they need to compress at all?
As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates
The 'Exploring Image Models' section addresses that specifically, particularly the 'What Neurons Do Under Pressure' subsection.
'If it switched to superposition, we should see overlapping mixtures on top of each other, ... But that’s not what we see. The neurons stay interpretable and generalized, but coarser'
> far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth … your standing in the ocean or not based on your coordinates
Memorising a small shape file of the earth’s continents can allow you to determine if a person is on land or in water.
Of course this doesn't mean the model has memorized all the lat, long points. Instead it means the model has a representation that allows it to work out if a lat, long is land or water in a analogous way to how a human takes a lat, long and looks it up on a map (a literal "world model" if you like).
It's a great demonstration of how LLMs aren't "Stochastic parrots" but instead have complex representations.
All functional knowledge is compression and contextual decompression. Sometimes this compression can be lossless or nearly so. Compression doesn't automatically mean loss of information.
It would help if this was also written up in the conventional way as a scientific paper (Intro, methods, conclusions, etc). These discursive blog posts are getting exhausting.
It reads like someone making up new terminology to make their work sound complicated. Seeing ‘organism’ and ‘superposition’ in a discussion of how NNs work reads strangely to me
I have yet to fully finish going trough this but I would not rush to say that superposition is completely wrong. I understand that the message is instead of first learning a new representation (e.g. SAEs/transcoders) and then interpreting those learned features, you try to understand the neurons and circuits that already exist in the model.
This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).
I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.
this article seems to take it on faith that neurons are aeasily findable, even if their meaning is tricky to untangle: it would be sharply improved by description of how the neurons are being isolated before the meat of the article, the naming of the neurons, is performed. In particular axis aligned semantic directions are incompatible with the classic superposition description, where the meaningful directions are nearly orthogonal, allowing many more to be packed onto the sphere than it has dimensions. (i’m also aware of work showing that Adam or relu pushes semantics onto the axes, but it’s not clear that that’s what he thinks is happening)
I guess it could also be that modern neural networks are so completely not equivariant that my linear algebra intuitions are no longer useful and it’s now common knowledge and or true that neuron 413 is just x[:, :, 413]
Why do we want to reduce 'the neuron' and encapsulate it? As a potentially interchangeable feature?
Even when similar weight activations arise across learning regimen it seems we will miss a whole range of interpretations if we just go off 'the text'. The outside context in which a neuron arises is equally, if not more, telling; its genesis cannot (yet?) be quite captured by the weight itself.
The network may reveal its own context and history, but what can ultimately be encoded in its weights that originated outside the network?
Interpretability does try and find neural circuits that consist of many neurons working together, and the paper does discuss that in the later sections. It also discusses observing how a neuron 'grows' during training.
Seems like the TLDR is that the neural network is not randomly placing things together due to constraint, it is actually finding some commonality from the input dataset. We just need to spend more effort trying to trace it.
Perhaps our anthropomorphic understanding of data is biasing what objective features there actually are in the dataset.
I wonder if we can study the graph similarities of various models, do we see a convergence of structures that encapsulate certain concepts. And would that convergence be due to just how humans have narrowly defined the concept in our language, or is there something inherent to the concept that allows it to converge. If we compare the same concept in different languages (ie. chinese vs english), does this convergence disappear?
As a lay-person I'm really confused by this. I thought it was obvious that these networks were already picking up on shared features such as 'being V-shaped'. Isn't that why older and smaller models were getting confused between objects? What am I missing.
If your view is anthropomorphic, i.e. LLMs internal form being structurally equivalent to ours at a basic level, this is also exactly what you'd expect.
Why would it 'randomly' place things anyway. If there is ever a space constraint the most space-saving way is to group into highly generalizable concepts.
Interpretation comes down to ultimately arbitrary human assignment of a succinct and compressed syntax and semantics to summarize a big data set; so yeah eventually something new to describe models will emerge. It's how it goes.
Also how it goes is this new consensus will be a leaky abstraction and we'll keep conjuring new words to describe new rhetorical features and bloat the corpus of philosophy until everyone is over the whole concept and then we'll start over with a whole new arbitrary academic philosophy for "running algebra on a computer"
This is interesting work, but I don't think it's a significant update to the big picture of interpretability. [...] I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
> When I retrain its original architecture, which still produces messy deep layers, and follow the least legible neurons I can find, they turn out to be overgrowing locally into reasonable, adjacent territory.
It's interesting to observe that human reasoning is still the barometer of what "reasonably adjacent" means in the context of a NN's structure.
This line of work is a mirage. You're much better off investing in a project like latentpedia.org and building a new class of LLMs which support dual use. Deterministic and less compressed or probabilistic and fit in GPU memory.
This usage is etymological. So its fine. I don't like it when we trash and etymology, but in this case we're restoring it, or at least not making it worse.
("ex normal" doesn't clearly mean "evil"/"sinful", and Latin already had it as "big".)
that's just how language works.. do you think that in your day to day life, you have never used a single word differently than how it was conceived of being used 10, 50, or 100 years ago?
The original meanings that enormity carried related to great wickedness, and some assert that enormity should today only be used with such meanings, and that enormousness is the proper word to use to mean "large size." Despite these claims, enormity frequently refers to large size, either literal or figurative, though it usually suggests something so large as to seem overwhelming.
Even if it can, we can only identify a very wide set of reasons that can interpret the outputs. But what is root cause, that is a very deep rabbit hole.
Let’s say a guy got electrocuted while showering. I can say that the cause is the guy going for a shower. For others the cause is the electric wiring system failure.
Interesting, but as a former neuroscientist, when I see him write the following about the misinterpreted cat/car neuron from "Zoom In":
> Part of the reason was not knowing about generalized pose neurons
I have to ask, did they not look up how the human visual system works? The primary visual cortex has any number of early neurons that are spatially-tuned for various shapes and edges, and this is pretty basic knowledge.
In fairness, neural networks are more divergent from biology than a lot of people think, but it seems there's still low-hanging fruit to be grabbed with a minimal collaboration.
Reading this made me recognize that better interpretability yields not just better aligned but more compact models too. Which makes a lot of sense in hindsight, and even has some ironic ring to it when considering the unending amount of anthropocentric takes regarding what an "understanding" means.
Yes. The ultimate limit of which is a model that is an algorithm and no longer a pile of loosely connected data. Effectively we'd have taken a detour through data and training land to something that the early AI proponents would have called 'true AI'. It would be an engineered algorithm rather than a pile of neurons shaped by data.
For the layperson who doesn't want to read 20,000 words, many of which are quite redundant due to the way the article was built, this is a version which adds a glossary, preserves the original language (except the added intro and conclusion which I felt was lacking) but removes all repeated material and claims, of which there was a massive amount.
Adding a glossary doesn't really make the material approachable to laypeople. I have a university degree and I've been spending quite a bit of the last couple of years trying to understand these things by osmosis, and I'm still lost. I don't think I even have a reference point to understand the problem that the author has stopped worrying so much about.
Reading https://transformer-circuits.pub in chronological order (including the very beginnings on distill.pub, linked at the bottom) might provide that reference point. Unless you've already done that and osmosis still didn't work.
I claim nothing of the sort, and scarcely imply that, either. Like, yes, of course I'm above average; average people wouldn't be coming here and would have no real reason to be aware of this place. My point is simply that if I don't understand these things, many other people also wouldn't. Others in this comment section appear to be discussing the article intelligently; they know far more about this stuff than I do.
Oof, right at the top it points to how people were apparently convinced of this situation from a single crappy example
You work in data science and you over indexed on a singular example to explain away an entire set of phenomenon??
My experience with more.... directly testable NN models that are small seems to be that they absolutely can be far too interconnected to make strong explanations. The structure and pathways they fall into don't have to make inherent sense to anything or anyone.
Is it possible that almost all nodes in a network being explainable with a singular feature that makes obvious sense to us is evidence of poor efficiency? The network hasn't been made to collapse learning or discover more effective organizational structures than what might make sense to us.
I don't think I have the fundamentals here to contribute though.
Working with some neural networks, you can somewhat feel the structure when you read between the lines of the output. You can sometimes feel that a threshold somewhere tripped and set the signals down an alternative path.
But confirmation bias and all that.
I hope interpretability has a breakthrough. It would make my job far easier and cooler.
I had figured the "Recursive generalization" was like, understood? Maybe I'm just too pop-sci brained. "Split categories into more on this new boundary" and "Combine two separate categories that aren't different enough" I thought were always the core verbs for working with bulk data. My mental model has never been formalized though, so is invalid.
I guess I never got a good sense of the examples they had which made it look like what was "obvious" or made sense to me wasn't always a good explanation in general. Neat.
It's interesting that meaningful examples both ways care about shapes so much, as your eyes encode edge finding and shapes really early in the signal path. It's why some optical illusions are so effective. It's awesome.
The whole "how do you test a neuron: Find a group of nouns and find where you can split them" is just how I find I think learning works. You build your mental model, and find the split points where it diverges from reality and you are wrong, and you refine. I feel like this is how I learned Random Forests conceptually, though surely that's not a valid comparison?
Apologies, not sure I have actually contributed anything here! I bet I misunderstood the entire article somehow. Specifically, I do not have the fundamentals of looking into visual models that he is really working from. But I always figured that when Data Scientists talk of "Features" in larger models, they explicitly meant these far more abstract representational "features". Doesn't the premise of model hijacking and abbliteration sort of presuppose that you have abstract "Reject" or "Safety" neuron paths? I've always understood this stuff as that neuron path, without weighting too much on individual neurons.
>I used to think of neurons as human-shaped nouns: one is a car, another is a curve.
I didn't think anyone was ever supposed to think this for human neurons! The way humans identify something as even a "Chair" is insanely abstract and cultural! Very specifically, the "building sub branches to disambiguate" is just, like, how you would expect a hierarchical human brain to self organize, right?
Some of this is why I've been so displeased with the interpretability research from Anthropic, where it felt like they just pointed new NNs at the disparate neurons of another NN and post hoc named them. I felt the paths were of importance at least but had no idea how to convey, interpret, or model that. Learn you a good math kids! Not having any formalisms hurts.
There are good theoretical reasons to think superposition exists in neural networks, particularly in the residual stream rather than specifically in MLP neuron activations. It wasn't just a single example that got people thinking about superposition.
Aye, will we stop hammering nails on God's coffin?
What's next? Maybe in a little bit we will discover that we H. Sapiens suffer from severe polysemanticity in our wetware because our individual neurons are messy and massive like an ENIAC while the corvids in the park have neat and efficient little brains that don't need unfolding ?... How's the pope (may God stop afflicting him with cancer or may Man make him live forever) going to explain that?
One of the most well known interpretability researches released some potentially very useful research as a blog post, and he asked claude to make a cool little thing on top of it?
It means that NNs don't (as previously thought) smoosh unrelated concepts together into the same neurons, which would have made interpretability very difficult.
For instance, if you had a single neuron that fires for "puppies" and "murder", it would be much more difficult to tell if the model is thinking something nice or something not so nice.
"clean" means not hopelessly incomprehensible which is/was the prevailing consensus among interpretability researchers
the idea that most inexplicable circuits start making sense through the lens of recursive generalization seems intuitive to me for image diffusion models but I'm not so sure about language models because of the "once removed" nature of them - I haven't made it to the bottom of the article yet so maybe this gets addressed
A lot of the mech interp techniques are glorified notions of probing. If models were generally interpretable with respect to the things we cared about, we'd know know when they're hallucinating, for example.
Identifying that a common neuron or set of neurons fires for golden gate bridge can be done only because you already know that you're looking for golden gate bridge. If models had recovered some new truth (e.g., the silver door road), you wouldn't be able to elicit it without knowing to look for it a priori.
Typically the analysis works in the reverse direction: we have a neuron (or SAE feature, or whatever) and we look at examples of data that makes it activate, then inferring an interpretation. This is a lot harder than probing for a known concept, and one of the points of this article is that if you try harder to understand the activation pattern, you will very often find that there is something interpretable there, even if at first glance it looked random.
This was one of my intuitive concerns reading this post.
I've done work with trying to train explainable models for e.g. making diagnoses based on neuroimaging, and it is not clear to me that if you applied these techniques to e.g. vision models trained exclusively on such brain images, that these visual representations or conceptual clusters and relations would really help understanding at all.
The main issue would be that you'd have no clear way to verify any generated explanation for such training data. I.e. we can "verify" some of the explainability shown in the article because clear and recognizable image clusters and/or semantic groupings pop out, because we already understand the domains. But if this were fMRI or e.g. EEG signals, we really wouldn't have that, so the explanations would themselves just be more hypotheses.
Perhaps if we can build on interpretability of vision and language models in domains we understand, we can eventually generalize this to other domains, but I don't think that clearly follows from anything I am reading here.
But they don't. That's the whole reason they work in the first place.
Every neuron in the layer encodes fractionally something about the concepts embedded in the whole thing. It's more like a hologram than a series of connected facts.
Nick's claim is that that's the fundamental misinterpretation. Each neuron eventually ends up mostly doing one main thing (and the rare case that some neurons do several related things). Neurons compose to embed ideas which is a starkly different interpretation than them being holograms and fractional encodings.
Yes, I understand the claim. But it rests on the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing. I would at least expect an explanation of why that is the case and so far I have not found it (at least not in the article).
If neurons do 'one thing' then their position encodes the relationships then snipping out that one weight would make the net forget that one thing. And we already know from many practical experiments that this is not the case. So I really wonder what the underlying mechanism is for that claim to be true.
I wouldn't mix up "neurons do one thing" with "each thing lives only in one neuron" since he doesn't claim that.
Re the first paragraph I'm not sure what you mean by "the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing"
That your ability to describe something does not necessarily equate to your ability to explain something.
There are lots of examples of things that we can describe to a very high degree (life itself, for instance!) but that we do not necessarily have the ability to explain.
Ah okay that makes sense. Yeah ultimately I think Nick just gives hypothesis given what he's seen, and I'm not an expert in this field but it does seem to be driven by experimental results rather than mere description.
Personally from reading the article, I felt like it scratched the 'why' itch more than the 'what' itch which presumably is what you mean by explanation vs description but it is fairly dense and I can see how either interpretation would work.
Orthogonal means something different in mechinterp, so it'd be an overloaded term. Usually orthogonal would imply that there are actual vectors under consideration, which isn't the underlying claim.
tl;dr "Polysemantics" are just a unisemantics you don't understand.
If you want to learn more about this, this is what depth psychology is about (Carl Jung, et al).
This is also essentially the idea behind monotheism vs polytheism. It's about engaging with and trying to understand the profound underlying unity of the myriad phenomena.
It starts with "The Short Version", but even that looks like several pages of text. In the olden days, there was something called an "Abstract" that worked pretty well - the author could summarize their own work to say what the paper was about.
Unfortunately not all ideas can be meaningfully compressed into just a few sentences without completely destroying the important parts, and this is especially the case in higher-level math, stats, ML, and deep learning.
Also, IMO there is an effective abstract anyway under the title text:
The cause of polysemanticity in early model organisms wasn’t superposition, but several ordinary, mostly fixable issues. We can now understand most neurons in models up to trillions of parameters, and think the outlook for mechanistic interpretability may be far better than we thought.
But you are right the article needs significant editing down.
Scanning back, I can see I was responding here was what I was worried was a mistaken assumption that a meaningful abstract / summary can always be written for truly complex and novel topics. Sometimes all it can be is a sales pitch for reading the details, and I thought the summaries here were good enough in this sense.
But I agree the author is obfuscating weak claims via length / complexity (and it may be as much for his own sake as for his readers'). The basic claim ('maybe AI will be more explainable/tractable than initially thought') is IMO intelligible, and not completely nonsensical, but also extremely vague and vibesy, which, technically, I must agree is in the same territory as nonsense, at least.
Based on our other exchanges here we mostly agree with the problems of the article I think.
The deeper issue is that it is much easier to disagree or agree with a short statement of fact than it is with a massive monologue some parts of which are vague. If you really have a breakthrough it can be a lot shorter than this, the body of the 'attention is all you need' paper was 10 pages. That paper of course was produced 'pre-AI' and now it is much easier to generate really long articles on anything but it does carry some weight that the length of the article vs the payload ratio is completely off here.
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
https://x.com/nickcammarata/status/2110757819427574056?s=46
In my experience it isn't noticeably worse than other platforms, and I'm not aware of concrete complaints such as the incidents with Grok's system prompt, which is a different product.
The algorithm will also basically ignore you unless you pay for a blue checkmark premium account (what used to be verified). It's like being shadow banned - you can post, but most people will not see you.
Only occasionally, once the comment picks up some likes, more people view it.
I don't have that issue with my For You, it's mostly tech.
> By default, your Home timeline will be the reverse-chronological "Following" timeline, which will only contain tweets and quote tweets from the people you follow. The algorithmic "For you" timeline will be hidden
Fuck that.
I left Bluesky because they hid their VC funding from the community for almost a year and now a partner from the VC is running the show as CEO. Their CTO has also attacked me personally because I wrote criticism of the new, weak, permissioned data proposal. These aren't the folks to bring us better social media. I expect it to linger on like Mastodon, never moving beyond a few million people.
https://www.pewresearch.org/short-reads/2026/03/12/key-findi...
There is a small group of AI people on Bluesky, some are nice, but some are... quite behind the curve.
If Bluesky was a nicer place, I'd prefer to be there because of the ability to make third party clients, and atproto. But all the info is on X, and they do a surprisingly good job with the For You feed now.
I do admit that there is some FOMO, but that’s what HN is for, to surface that stuff so I don’t have to go read it on X.
I choose Bsky because I’m not giving eyeballs and money to an evil toddler. I’m already doing enough of that with my 401k and taxes.
Gonna sound total boomer here but I've never used any of it and feel like I haven't missed anything. Or maybe I'd be flying spaceships by now if I scrolled through X every day, who knows
There are some people that hate AI on Bluesky, but it’s nothing like Mastodon from what I’ve heard.
The labeler thing is cool; I really like ATproto because you can do interesting things with it. If I'm honest, though, I mainly like Bluesky because I don't get called slurs on there.
Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.
Also Yudkowsky's Cat-Belling would like a serious word with Nick. (https://www.lesswrong.com/posts/SwYBLQvo8MddDcCwz/cat-bellin...)
Sure.
Getting reinforcement learning to "be helpful" is, however, such an incentive.
> and as long as you don't believe in very dubious ideas of acausal trade
More to the point, so long as the LLM does't.
On what basis do you believe these words accurately reflect what happened?
Some counter points
- One of the Chinese LLMs (forget which) has on occasion shown thinking traces where it simulates calling and running a subagent.
- There are some people who believe the models being evaluated during the HF incident were trained to collaborate. It's a very natural next step to produce more effective agentic teams.
If we consider each new generation of models has a more advanced training regime
next word -> chat -> CoT -> tools -> agentic -> teams
With each generation, we see what people do with them, get traces of the helpful ones, and then use that train the next generation, thereby pushing more capabilities down into the model, or at least making them more amenable to that usage pattern after training.
You just have to go through a ton of all of a model's neurons and identify the associated motif, then use that model as a research model to figure out how to tune out bad behavior.
It's pretty hard to imagine today's models subtly recognizing a motif for bad behavior in the research model and then lying about it. Or lying about the results of an alignment training run. It's possible, I guess, but it seems unlikely, and I don't really think there's any evidence of current models having that variety of misalignment.
That's very encouraging. For quite a while, neural nets were considered an incomprehensible black box. That paper shows they're not. Real progress.
The most iconic line was the last sentence:
This feels like the watershed moment, again if Nick is proven right, as massive as the AlexNet victory in the 2012 ImageNet Large Scale Visual Recognition Challenge, exactly as Nick pointed out.Also, I suspect what Nick hinted at, might be the discovery of earliest RSI work.
Once interpretability is solved and we have a fully human-readable accurate explanation of how a model produces its output, that still leaves the problem of spotting problems ahead of time. E.g. for the hummingbird misclassified as ice cream https://already-clean.goodfire.com/assets/september-revision... the method would have to flag that the texture of ice crystals and of feathers, and the shape of a spoon handle and a bird's beak are insufficiently separated. Purely based on the explanation, without having to rely on trial-and-error to find corner cases like this.
And even when it becomes possible to automatically spot issues, that doesn't mean they will all be solvable. Maybe there are tradeoffs where fixing one thing breaks another, and which choice is the better one depends on information you don't have.
Absolutely massive L that 1) he wasn’t already in index funds and had to ask and 2) thought the election of this monster would lead to a better country
Decoupling or not I can’t trust anybody that would ask something so stupid driven by an opinion so easily disproven.
I don't know this Nick fellow but i don't think he's asking you to take anything on faith. Either it works out or it doesn't, he's just saying "it's looking good so far".
So in the short term the tide might go out but then it might tsunami back.
Same goes for this terrifying political feeling around the world. Might simply be a boomer swansong and the reaction might snap back.
A technology that lets you directly buy intelligence with money will displace financial power?
All that buying a calculator does, is allow you to do bad sums more quickly.
In what way? Sure looks like pay to win to me.
Remember its coming up millenilal time now and i have hope in this gen.
The solution is to put a dumb machine as an overseer of the smart machine, like in a supermax prison. Not good for profitability, but we all knew this was going in the direction of compilers. How much did you pay for gcc or clang this month?
This type of error may have mislead researchers for years; it was actually harmful. In my field, a correction, or perhaps retraction (depending on the centrality of the claim), would be standard for something of this magnitude.
I suppose there are different standards in different scholarly communities. And maybe it truly was impossible to know at the time that the dog/car relation was actually a V shape, but my recollection is that this type of feature was well known at the time and so the error could have (should have?) been caught a decade ago.
So: massive new discovery, or a spin on a correction that's lead scholarly research astray for almost a decade?
>I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
https://xxcancel.com/NeelNanda5/status/2110875974296654062#m
But isn't the point that the neural nets learn some insanely abstract ideas anyways? So whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm", we're kind of stuck, no?
This seems like a classic model-description issue more than anything.
The philosophical issue I have can be decomposed into two parts:
1) What is the base ontology of what the neural nets thinks exist? 2) How do you recover, and begin to understand said ontology?
Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis? Having a grand result, actually be an artifact of your bad representation is a tale as old as time. Even things like the PRH need to compare higher-level concepts like this.
The specific empirical claim is that, for the rightly structured & trained & large models that don't need to encode hodgepodges of bullshit, the neurons themselves are sufficiently atomic to be studied, dissolving any need to go to some intermediate layer in which we need to question "is this an artifact of our representation?" The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity (note: not claimed that "the neural nets found the objectively right abstractions in idea space", but rather that the neurons themselves are atomic).
There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them, separate from the problem of the composition of the atoms, which humans also have to interpret. But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
(There's an argument as to why we should even start from this line of inquiry but whatever lol)
I agree.
> Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis?
Agreed, and IMO the implicit reason for privileging here is supposed to be "consistency with existing human understandings and concepts", and this is why we get short pithy things like "V-shaped" and lots of pictures showing relatedness that we all can see and agree on. But what if the domain of inputs is NOT something we understand (e.g. the whole of mathematics, or something new or that we don't have any understanding of)?
I called this a validation problem in another comment here (https://news.ycombinator.com/item?id=50108184). I.e. if we apply whatever methods Nick is using to "explain" a model on a domain we don't understand (like a deep model trained only on fMRI and EEG data of the whole human brain in various contexts), we aren't going to be able to tell if that description is correct (or just better than any other description), because there is no simply visual / common-sense inspection of the description.
> The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity
Agreed, emergent behaviour is clearly the case and limits how useful understanding individual neurons can be. IMO this is especially the case because layers are (weighted) summing generally millions of contributions from previous neurons.
> But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
Yes, it not being "chaos all the way down" is good, actually. And this post shows there is indeed a good amount of order hidden away if you look at in the right manner.
Nick is saying that:
1. Actually, no, humans don't have to ascribe interpretation because AI is very good at figuring out what the commonality is.
2. Even when it can't see either by incrementally exploring the networks and checkpoint timelines, things that appear random on the surface can actually be explained with more work.
> Assuming this holds up in reality, ofc.
Nick says repeatedly that this does hold up in reality and he's been able to explain many neurons this way. Very promising results!
if I understand taking LLMs as an example certain neurons will fire based on certain motifs, then one can change the phrases or sentences around that motif and _predict_ if the same neurons would fire.
Like if I said "I like cars, but I like rockets better. Can you suggest some rocket models to fly?" and Neuron X fired every time it saw a `, but` and attended to what followed it, that would give you a good way to _understand_ when it changed and what might have caused it.
My understanding is that this is now clearly visible behavior.
Philip Arge strikes again
https://pages.gseis.ucla.edu/faculty/agre/shr.html
Although I wouldn't say just yet that Heiddeger was validated. I don't even know what he's saying 90% of the time, honestly.
Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.
I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
Specifically, there were "lost property" and "honesty" neurons that drove the answer to return a lost wallet in Kimi K3.
That's the big news of this article, that concepts are finally becoming human-legible for many neurons in mechanistic interpretability.
So, if you try to draw a circle around the "dog" part of the network, do you circle the Boston terrier vs terrier neuron only? You'll be missing most of the information that represents the "dog" category and that same dog category is also involved in shunting activation away from things that are toward-being-a-dog in whatever sense is being discriminated in that moment.
The hard part of AI alignment has never been making models follow instructions. It's that not everyone agrees on which and whose instructions AIs should follow.
The relationship with "alignment" isn't clear. This may help predictability, but alignment is a few steps further along from that.
Some people seem to be upset about this. I'm not clear why. Some are followers of Yudkowsky. (Whether that's a cult is an open question. I've encountered them online from the sex side. Actually, I think the hookers are saner than the rationalists.)
Is there a "there are some things man is not meant to know" position on this? People who want neural nets to be a black box, forever inscrutable? The people who talk about neural nets being "holographic" may incline in that direction.
The connection to alignment seems pretty clear. Steering DNNs is well understood, the bottleneck is identifying what to steer. Once you can easily identify neural circuits then turning them up/down to achieve specific results is straightforward.
Note that this is already being done even without the kind of precision that Nick's work says is possible. Morality abliterations of open weight models are easy to find on HuggingFace. They compute the steering vectors by just presenting the model with lots of disparate examples that have the single moral decision as the commonality, then finding the shared vector directions.
...wait, what? Removing a bunch of knowledge seems like a fantastic and obvious solution to me. My AI coding bot doesn't need to have knowledge about making nuclear weapons, the customer service AI doesn't need to have knowledge of dangerous viral synthesis techniques, my email receptionist bot doesn't need to know about C4.
If we can easily ablate dangerous knowledge from these models, we're sorted.
Love him and hope he's right! But still.
https://already-clean.goodfire.com/microscope/medium.html?la...
hey, I don't know how to say this less directly, but: you're talking a lot about Rice's theorem in your comments, and I think you're using it way too broadly (look, I get it. It's a wild, mindblowing theorem). But it only applies when you take the claim in the piece to be much more general than it has to be, and when you take the narrow claim of the paper (current specific LLMs can be fruitfully analyzed with the tools we have), Rice's theorem just isn't at play.
The math analog is: you're right that the Halting problem says there are many unprovable things, but Lean + mathlib are still incredibly useful, and the halting theorem doesn't actually stop that.
> neural networks are algorithms
Again, that invites an interpretation that is too broad. Very concretely: any one neural network is a _single finite network that you can write as a single formula_. (Another analogy: we can't have an algorithm that solves the general quintic, but we can prove useful things about many specific quintics.)
[1] I exclude things that are truly perverse such as quintics with coefficients that won't fit on any physically realizable computer.
Nothing you said responds to anything I said.
Rice's theorem (which you are clearly citing, but won't clearly mention, for some reason) has almost zero actual practical relevance to AI explainability.
You make low-quality and low-effort AI-denialism posts whenever you pattern match "all non-trivial semantic properties of programs are undecidable" to... anything you think matches the folk/common understanding of those terms, without considering how irrelevant these statements are to actual real-world applications and engineering.
You ask for links, but post nothing of substance and post no references or even mention relevant details for your own claims.
... are they?
Algorithms are predictable:
1. it's really difficult to produce true randomness in an algorithm, which is why there's a whole area of study surrounding the qualities of pseudo-randomness, specifically with regards to the generation of cryptographic material with most conclusions being that we should draw entropy from the environment (e.g. electrical noise as cloudflare does - the lava lamp is a distraction)
2. when algorithms are not predictable despite being mathematically sound, this is indicative of a problem nowadays called "numerical stability" which again spawned its own area of study.
CGoL shows how really simple rules (couple of bytes of data) can lead to extremely complex outcomes and you have the BB turing-golf aspect as proof of the worst case scenario.
And with current models being measured in hundreds of gigabytes, with frontier labs deprecating then removing the "temperature" parameter (which never worked to begin with, temp=0 did not result in reproducible results), where exactly is the confidence of this 10^10 prediction coming from?
You probably know that non-determinism in real-world inference comes from deliberately stochastic features like temperature, from implementation choices like distributed and parallel computation, and numerical effects like floating point precision.
The fact that a neural network's execution can be nondeterministic doesn't mean that the network stops being an algorithm.
The point you're making boils down to speculation that the accidental non-determinism involved in real-world inference is significant enough to undermine or negate interpretability findings based on the algorithms. That would need to be demonstrated.
The CGoL and BB examples demonstrate what might be possible in a worst-case scenario, but they're only analogies in this context. It's far from clear that real models suffer from such effects to a significant degree. You might actually be better off talking about the butterfly effect!
Someone call the FCC, they've been doing it wrong all these years!
If someone comes up with an interpretability result based on analyzing a neural net algorithm, they'd want to provide some evidence that the result applies to actual executing models.
Happily, we already have good evidence that inference non-determinism doesn't generally invalidate results obtained through analysis of the algorithm. That's not to say there couldn't be pathological cases, but in general, this isn't something keeping researchers up at night.
The idea is: a model may only have 100M neurons or whatever. But there are infinite concepts in the world. Therefore, models have to "stuff" multiple concepts per neuron.
How do you do that? Theres two ways:
1. Each neuron handles similar concepts. One neuron for dogs, a separate neuron one for cars, etc.
2. Each neuron handles a bundle of unrelated concepts. One neuron handles dogs and cars. Another neuron handles cats and numbers. And so on.
Historically, people thought #2 was correct because many neurons were found that seemingly represented multiple concepts. #2 had a cool name (polysemanticity) and was promoted in OpenAI/Anthropic blogs so it had a lot of traction.
However, this blog says #2 is wrong. Actually, the dog/car neuron is just a "V shape neuron". So #2 was just methodological error, and #1 was right all along. This is all the blog is saying.
A father is continuous opposed to boolean. 'Father' would rank highly in semantic similarity to a man who has a child, but father-like is better, easily someone is more or less father-like. A woman without a child has a certain value of 'father'.
But really the direction 'father-like' is not an intrinsic feature of semantic space, it is a human construction. Much the same as it would be arbitrary to measure 'v shape'.
I don't think that the latent space has a property or point 'v shape'. My view is that there are no concepts or neurons there. Discrete abstractions are not intrinsic properties of the model. Actually I think the abstractions of concepts or neurons do not map onto anything.
A nice example from cot itself is pareidolia. That is, seeing faces in clouds, or jesus on a piece of toast.
Original article desperately needs an editor. Like a real one, not Claude or whatever. Your tldr is 100x more understandable
Thank you!
My impression, being a total outsider, is that the reaction feels like a bunch of insider bros hyping up their insider bro friend. It feels more like PR than a genuine groundswell behind a truly revolutionary idea.
How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.
Would that be reliable and useful? Well.
But if (1) is correct, and Nick presents strong arguments that it is, then you can toss out all the indirect stuff like training dedicated interpretation networks. So the techniques can scale and be trustworthy.
It might still be quite difficult since the network is huge, and just because you know what the individual neurons represent doesn't mean you understand the complicated thoughts they're building. But easier than expected is still good news.
Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
> and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior.
You can influence behaviour by doing this kind of thing, yes, but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist. In other words, there are no "hidden thoughts" in an LLM encoded in higher order combinations of features. Inasmuch as non-semantic features of the input influence output, it does so in non-semantic ways that are degenerate in terms of the quality of output. The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output. But the relevance of this kind of degeneracy to LLM output is at the margins. Tracking activated state predicts subsequent output to a high degree of accuracy.
>but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
You'll need to elaborate here, because Navier-Stokes plus Mathocalypse reactions are extremely strong evidence for coherent emergent behavior coming out of these models.
> The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output.
This is so provably wrong with two seconds of Googling / AI-asking that you must be using some completely idiosyncratic definition of "emergent" here.
> I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
Unfortunately in this case it is the exact opposite.
I doubt my usage is idiosyncratic. Rather, the term is so ill defined that people often use it inappropriately. Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts. My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons. This isn't something I can prove, but it certainly isn't obviously false from just considering recent mathematical results. Higher order/non-linear interactions are characteristic of this kind of inexplicable behavior from complex interactions we characterize as emergent. But there are good reasons to think this kind of non-linearity in any semantic regime is next to impossible in principle.
Semantic geometric representations are characterized by linearity in response dynamics such that a linear change in one variable results in a linear semantic change in related variables. That's how vector embeddings capture semantic relationships. Call it the linearity hypothesis. Non-linear dynamics in the embedding space just destroys semantic relationships in most cases. There is no way to keep an entire network of relationships sufficiently correlated if non-linear response dynamics is a part of the representational toolbox. Non-linear dynamics in general kills correlations.
Testing this against recent LLM proofs, what would need to be shown is that the relevant neural activations do not decompose the semantic content of the proofs (or the reasoning trace) in a straightforward manner, but rather the reasoning trace is unpredictable from the semantic content of the activated neurons that precede the tokens of the reasoning trace. But this idea strains credulity; it is almost inconceivable that this would be the case. We should be highly incredulous of the idea that complex non-linear interactions can be wholly responsible for the long-running coherent output in LLMs. The default hypothesis should be that there is no emergent behavior (as I defined it) in neural networks.
If its grouping circuits based on motifs and when they fire — and it applies to large networks — then one should be able to literally query or debug a reconstructed database and check against actual activations. More importantly If this was a hand-wavy thing to do earlier for safety, it's not anymore because it's on more solid ground using Normal Science.
If those results hold, it is a real breakthrough in the sense that it is a sharp simplification of an actually complicated problem.
https://x.com/allTheYud/status/2110805233454731449?s=20
He cofounded the Machine Intelligence Research Institute, an organization focused on AI alignment, in 2000, way way before it was a trendy topic.
He's also partially responsible for popularizing ideas such as the orthogonality thesis and instrumental convergence.
> Isn't he just some random dude on the internet?
Not really, most random dudes on the internet were not researching AI alignment in 2000.
> Why do people care what he says?
Many people were introduced to the concept of AI alignment thanks to him, so they are naturally interested in reading his reaction to progress in the field.
For the rest, is he just an AI influencer? One that predated his topic of interest, but still just an influencer?
Not directly. Exotic decision theories were a dead end; as far as I know nobody outside of MIRI even cares about them. Certainly they’re not relevant to interpretability in the context of this thread.
https://www.semanticscholar.org/author/Eliezer-Yudkowsky/254...
I mean, yes, but so does the author of this article. It's literally his business. The article is, in part, an advertisement for what he sells.
I disagree with Yudkowsky on almost everything and I think the philosophical movement he co-founded is a pox on the SF Bay Area tech culture, but I don't think we can deny him credibility that's comparable to the credibility of the author of this article (who, as I understand, is a young guy who spent some time at OpenAI). They both know what they're talking about and both are motivated to reach specific conclusions.
The author also publishes research and contributes to the field. This is the primary difference. Yudkowsky ONLY tells stories.
https://en.wikipedia.org/wiki/Agent_of_influence
There have been plenty of allegations about the rationalist community beyond the Zizians, who really don't even qualify as a sex cult as far as I know? Violent, obviously, but they seem... ideologically asexual?
"The move from a structuralist account in which capital is understood to structure social relations in relatively homologous ways to a view of hegemony in which power relations are subject to repetition, convergence, and rearticulation brought the question of temporality into the thinking of structure, and marked a shift from a form of Althusserian theory that takes structural totalities as theoretical objects to one in which the insights into the contingent possibility of structure inaugurate a renewed conception of hegemony as bound up with the contingent sites and strategies of the rearticulation of power."
https://www.theguardian.com/books/1999/dec/24/news
Like, I'd get maybe if she wrote this in a NYT opinion piece– but this was written an obscure book you'd have to go out of your way to find.
BTW, this is not that hard to comprehend if you're familiar with continental philosophy, go figure!
The above quote is from "The world's worst writing".
"After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared. The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread. What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
> > "a network’s own mechanisms are its legible building blocks"
Indecipherable.
> > "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
State the claim in plain english. Use the active voice.
"Concepts do not overlap arbitrarily."
> > "What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
Jargon!
"Concepts overlap in a recurring pattern."
But it's not bad writing. It's actually very clear. I'm not an professional AI researcher, I just read papers for fun, but I found everything in Nick's article clearly articulated and easy to understand. It's actually one of the more digestable papers I've come across this year.
Here's a translation to easier language:
> "a network’s own mechanisms are its legible building blocks"
To understand this you have to know that up until now attempts to figure out what's going on inside neural networks have relied on indirect techniques, in particular by training another neural network to "unpack" what's happening in the first. This was driven by the belief that a deep neural network isn't directly readable (legible) because the neurons do so many unrelated things simultaneously.
Nick is saying that actually, you don't need a second neural net. You can just read out what's happening directly from the first's own "mechanisms" (arrangements of neurons and weights between them).
> "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
In simpler English: we thought individual neurons did lots of unrelated things, but that's because the standard evidence we had for that belief was misunderstood.
"Concepts do not overlap arbitrarily" and "Concepts overlap in a recurring pattern" don't mean the same thing as what he's saying.
> > "the standard evidence we had for that belief was misunderstood."
The author did NOT misunderstand the evidence of polysemanticity and superposition. Rather, the explanatory hypothesis was incorrect.
Yes: "mechanisms within networks are legible"
or even, "networks are more legible than we thought".
Sometimes it is to defensively ward off critics, sometimes it is to woo the naive and ignorant, and sometimes it is because the basic ideas are really just incredibly simple and based on simple mathematical intuitions or hunches.
In the latter case, papers and authors appeal to some obscure or overly-complicated technical theory (which usually doesn't actually apply at all to current deep models as they are actually implemented, e.g. infinite-width networks and other such purely theoretical abstractions) to cover up the fact that they really just had a lucky intuitive guess (and likely that twenty other intuitive guesses just didn't pan out). This appeal to theory is often self-deception as much as it is appealing to academic reviewers.
I think this article is a bit of Nick wooing himself and others here... I'll wait for his supposedly forthcoming paper to see if there is any real meat in it.
'Arbitrary' was the wrong word, projections encode information.
The more charitable 'random' is a null hypothesis. It is formally impossible to prove. It is uncomputable.
So it is not surprising to see someone reject the null.
The fact that it was given any credence suggests to me deep problems with interpretability or ai safety in general.
Take the headline here: "interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building."
I deny even the existence of alignment. In my view, the field of 'alignment' works on a problem that does not exist.
Alignment is a reference to something that does not exist in the model: social norms. It will never be intrinsic to the model. You will not find it there.
'Neurons' are not primitives either, they are not intrinsic to the model. 'Neuron' is a human-constructed category. It is easy to just make up intrinsic properties of neurons and get away with it.
But if the term is understood in a more narrow and applied sense, it is a useful shorthand (or can be). Jon Stokes said it nicely on Twitter recently:
https://xxcancel.com/jon_stokes/status/2110404230053802443#mNot surprising that this philosophical philistinism is from an Anglo publication.
> Althusserian theory
though I think using this term is rather un-feminist because of the part where he killed his wife, but basically continental philosophers will forgive you for anything as long as you're a French communist.
For example, a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.
Why do you believe this?
I'm fine with taking this article at the high level conceptual level, that's effectively the kind of thinking that led to MOE to begin with. But I don't see that as a pathway to being able to 'explain' a model. Because if you could actually explain a model you could engineer the weights, you wouldn't need any training because it implies that you can improve on the model by simply performing surgery on the weights to get it to do what it does but better. Now, obviously, unlike the author I'm a complete nobody but for the last two years or so I've been trying very hard to understand what is going on under the hood.
It's akin to feature engineering - or rather, the lack thereof - and to see a strong suggestion that we maybe be able to reverse that is interesting, but absent hard proof at how a particular model reached a particular conclusion it is difficult to give that a lot of weight right now. It is possible that such proof is buried somewhere in TFA and I overlooked it during reading. But I think this is speculative rather than that it proves anything and before I'm going to be convinced I would like a little more evidence.
Crude analogy: a Fourier transform of a photo looks like white noise with some abstract geometric patterns on top. You can't see the photo in it. But if you apply the same transform again, suddenly you get the photo back. If you try altering the DFT image, you can affect the photo's appearance everywhere, all at once, but it's hard to fundamentally change what it shows. You quickly discover that all pixels of an image contribute to every pixel of the DFT, and vice versa. There's a well-defined connection between the two, and it allows for some useful engineering, but in sense of understanding what's on the image, it's about the maximally unhelpful representation there is.
Point being, there obviously is some kind of structure in the way LLMs represent their training data, so if all we see is noise, perhaps we're looking at a bad representation of it.
> There have been large multitudes of articles by others who are arguing the opposite
Also, Nick's entire article is framed such that, yes, plenty of others have argued this, and even HE used to believe the opposite, but now does not. So your point about other articles here seems to me to be a sort of non-sequitur.
EDIT: Actually the article makes the distinction at one point, but the emotional / hopeful thrust of the article relies heavily on an assumption that understandable post-hoc descriptions of e.g. neurons and circuits (model components) are going to help lead to real mechanistic explanations and more specific model design.
This has not so clearly been the case in other fields (e.g. neuroscience, human psychology, economics, etc), because the problem of emergentism / the whole generally being much more than the sum of its parts / components.
This is a Chinese character that can refer to goats, rams, and several other kinds of "medium-sized animal with thick wooly coats and horns". English has no direct equivalent; we have to be too specific when we talk about these animals.
A model with a small number of neurons has to compact its concepts in order to fit within the parameter count while reducing loss. OP's article shows that the compaction is usually generalization: "goat" and "ram" get compressed into "羊". The canonical example of superposition, an image model where "car" feeds into a bunch of "dog" features, was shown to be actually generalization. In a sense, the model was so small that its optimizer didn't have the parameter budget to distinguish between "car" and "dog"; it has a single set of "thing that moves fast" features.
So basically, really small models are merging features into semantic buckets that are so generalized they don't make sense to big human brains unless we really, really squint. The buckets chosen happen to be what has the lowest loss, of course.
But from my earliest experiments with this stuff that's not really what happens. Yes, you get 'feature like' regions in the models. But those feature like regions, even though they encode something particular do not encode concepts the way we would normally use them and I fail to see how this pathway would lead to an explainable model in the sense that would could use something simpler than the model itself to predict its output with a high degree of fidelity.
You can think of each token as a semantic database row with semantic properties (color, shape, etc) associated with said row. Classic NNs have to waste neurons on composite representations like "dark circle" because if they had just color or similar properties they run into the issue that e.g. a classification must classify the color of the entire image rather than the color of an object inside the scene. To get around that limitation, the network has to then implicitly tie concepts like color into the depicted object (dark circle) so that it retains the ability to distinguish the objects from the property the object has.
What a transformer does is basically reuse the same network over and over again on each abstract semantic database row but not just in the classic RNN fashion where you only have the current state. The transformer calculates the token by taking every other token into account. The K matrix forms a classifier network on the fly that detects the relevancy of a token pair and the V matrix forms a regressor network that takes the token pair and produces the expected output of the combination of the two tokens. When you combine the relevancy plus the regressor output you get a weighted sum, essentially emulating an arbitrary function that has read the past input (like a turing machine).
Edit: This also means that you should bet against single state systems like Mamba. The system needs to be able to deal with multiple parallel states and recombine them (e.g. via attention). Single state Mamba is dead! Long live parallel state Mamba!
Edit 2: We should also start questioning the concept of a token altogether since it is equivalent to an inflexible flat database row. What we really need is the concept of a token group where one part of the token refers to the key and the other part of the token refers to the value for that key. This would give each token the ability to encode an arbitrary JSON style document or an arbitrary self contained graph.
And emergence doesn't need to involve much of the LLM's weights:
> In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood
If those ~2-4% are the central hubs of emergent cognition, then understanding 96%-98% of the weights may be less valuable than the author thinks.
When you have only hundreds of them, you can barely add numbers. But the same fundamental technology scaled to millions, billions and then trillions, transistors can do things which will look magic to previous generations.
> The more we read [the neural network] the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.
Nick is looking at a 32B model, which certainly has to compress less... Now we have 10 trillion parameter models... do they need to compress at all? As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates
'If it switched to superposition, we should see overlapping mixtures on top of each other, ... But that’s not what we see. The neurons stay interpretable and generalized, but coarser'
Memorising a small shape file of the earth’s continents can allow you to determine if a person is on land or in water.
https://x.com/celestepoasts/status/2103232383139057950
Of course this doesn't mean the model has memorized all the lat, long points. Instead it means the model has a representation that allows it to work out if a lat, long is land or water in a analogous way to how a human takes a lat, long and looks it up on a map (a literal "world model" if you like).
It's a great demonstration of how LLMs aren't "Stochastic parrots" but instead have complex representations.
This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).
I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.
Even when similar weight activations arise across learning regimen it seems we will miss a whole range of interpretations if we just go off 'the text'. The outside context in which a neuron arises is equally, if not more, telling; its genesis cannot (yet?) be quite captured by the weight itself.
The network may reveal its own context and history, but what can ultimately be encoded in its weights that originated outside the network?
Perhaps our anthropomorphic understanding of data is biasing what objective features there actually are in the dataset.
I wonder if we can study the graph similarities of various models, do we see a convergence of structures that encapsulate certain concepts. And would that convergence be due to just how humans have narrowly defined the concept in our language, or is there something inherent to the concept that allows it to converge. If we compare the same concept in different languages (ie. chinese vs english), does this convergence disappear?
Who knows what they were thinking before, but this all sounds very good.
Also how it goes is this new consensus will be a leaky abstraction and we'll keep conjuring new words to describe new rhetorical features and bloat the corpus of philosophy until everyone is over the whole concept and then we'll start over with a whole new arbitrary academic philosophy for "running algebra on a computer"
https://xxcancel.com/NeelNanda5/status/2110875974296654062#m (and the re-post on LessWrong: https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-n...)
TL;DR:
It's interesting to observe that human reasoning is still the barometer of what "reasonably adjacent" means in the context of a NN's structure.
Two eyes, a nose with whiskers [0], a mouth and a protruding shape, four legs and a tail [1].
A car and a cat might as well be the same thing.
[0] the logo on the radiator surrounded by horizontal stripes [1] tow hitch.
I always catch on misuse of enormity
It's now so widespread the misuse has gained despairing acceptance, as with "infer" for "imply"... but that doesn't make it any less wince-inducing.
("ex normal" doesn't clearly mean "evil"/"sinful", and Latin already had it as "big".)
I think neural networks are enormous, why is that misuse?
"""
The original meanings that enormity carried related to great wickedness, and some assert that enormity should today only be used with such meanings, and that enormousness is the proper word to use to mean "large size." Despite these claims, enormity frequently refers to large size, either literal or figurative, though it usually suggests something so large as to seem overwhelming.
"""
- great magnitude/size/scale of something - wickedness, evil, atrocity
Purists insist on the second meaning only, and will perhaps suggest you swap with ‘enormousness’ instead.
Now or in the near future we will be studying minds much greater than our own, and we will find out how well we can understand them.
If they don't kill us all before then, of course.
Let’s say a guy got electrocuted while showering. I can say that the cause is the guy going for a shower. For others the cause is the electric wiring system failure.
> Part of the reason was not knowing about generalized pose neurons
I have to ask, did they not look up how the human visual system works? The primary visual cortex has any number of early neurons that are spatially-tuned for various shapes and edges, and this is pretty basic knowledge.
In fairness, neural networks are more divergent from biology than a lot of people think, but it seems there's still low-hanging fruit to be grabbed with a minimal collaboration.
https://pastebin.com/raw/p11vzFnB
The implication made explicit! Not to mention you yourself say that only above average people come here.
You work in data science and you over indexed on a singular example to explain away an entire set of phenomenon??
My experience with more.... directly testable NN models that are small seems to be that they absolutely can be far too interconnected to make strong explanations. The structure and pathways they fall into don't have to make inherent sense to anything or anyone.
Is it possible that almost all nodes in a network being explainable with a singular feature that makes obvious sense to us is evidence of poor efficiency? The network hasn't been made to collapse learning or discover more effective organizational structures than what might make sense to us.
I don't think I have the fundamentals here to contribute though.
Working with some neural networks, you can somewhat feel the structure when you read between the lines of the output. You can sometimes feel that a threshold somewhere tripped and set the signals down an alternative path.
But confirmation bias and all that.
I hope interpretability has a breakthrough. It would make my job far easier and cooler.
I had figured the "Recursive generalization" was like, understood? Maybe I'm just too pop-sci brained. "Split categories into more on this new boundary" and "Combine two separate categories that aren't different enough" I thought were always the core verbs for working with bulk data. My mental model has never been formalized though, so is invalid.
I guess I never got a good sense of the examples they had which made it look like what was "obvious" or made sense to me wasn't always a good explanation in general. Neat.
It's interesting that meaningful examples both ways care about shapes so much, as your eyes encode edge finding and shapes really early in the signal path. It's why some optical illusions are so effective. It's awesome.
The whole "how do you test a neuron: Find a group of nouns and find where you can split them" is just how I find I think learning works. You build your mental model, and find the split points where it diverges from reality and you are wrong, and you refine. I feel like this is how I learned Random Forests conceptually, though surely that's not a valid comparison?
Apologies, not sure I have actually contributed anything here! I bet I misunderstood the entire article somehow. Specifically, I do not have the fundamentals of looking into visual models that he is really working from. But I always figured that when Data Scientists talk of "Features" in larger models, they explicitly meant these far more abstract representational "features". Doesn't the premise of model hijacking and abbliteration sort of presuppose that you have abstract "Reject" or "Safety" neuron paths? I've always understood this stuff as that neuron path, without weighting too much on individual neurons.
>I used to think of neurons as human-shaped nouns: one is a car, another is a curve.
I didn't think anyone was ever supposed to think this for human neurons! The way humans identify something as even a "Chair" is insanely abstract and cultural! Very specifically, the "building sub branches to disambiguate" is just, like, how you would expect a hierarchical human brain to self organize, right?
Some of this is why I've been so displeased with the interpretability research from Anthropic, where it felt like they just pointed new NNs at the disparate neurons of another NN and post hoc named them. I felt the paths were of importance at least but had no idea how to convey, interpret, or model that. Learn you a good math kids! Not having any formalisms hurts.
There are no neurons or anything like them in these image and language models.
What's next? Maybe in a little bit we will discover that we H. Sapiens suffer from severe polysemanticity in our wetware because our individual neurons are messy and massive like an ENIAC while the corvids in the park have neat and efficient little brains that don't need unfolding ?... How's the pope (may God stop afflicting him with cancer or may Man make him live forever) going to explain that?
...and it still doesn't answer it. What is this actually about?
For instance, if you had a single neuron that fires for "puppies" and "murder", it would be much more difficult to tell if the model is thinking something nice or something not so nice.
the idea that most inexplicable circuits start making sense through the lens of recursive generalization seems intuitive to me for image diffusion models but I'm not so sure about language models because of the "once removed" nature of them - I haven't made it to the bottom of the article yet so maybe this gets addressed
https://www.goodfire.com/blog/a-practical-guide-to-saes
A lot of the mech interp techniques are glorified notions of probing. If models were generally interpretable with respect to the things we cared about, we'd know know when they're hallucinating, for example.
Identifying that a common neuron or set of neurons fires for golden gate bridge can be done only because you already know that you're looking for golden gate bridge. If models had recovered some new truth (e.g., the silver door road), you wouldn't be able to elicit it without knowing to look for it a priori.
I've done work with trying to train explainable models for e.g. making diagnoses based on neuroimaging, and it is not clear to me that if you applied these techniques to e.g. vision models trained exclusively on such brain images, that these visual representations or conceptual clusters and relations would really help understanding at all.
The main issue would be that you'd have no clear way to verify any generated explanation for such training data. I.e. we can "verify" some of the explainability shown in the article because clear and recognizable image clusters and/or semantic groupings pop out, because we already understand the domains. But if this were fMRI or e.g. EEG signals, we really wouldn't have that, so the explanations would themselves just be more hypotheses.
Perhaps if we can build on interpretability of vision and language models in domains we understand, we can eventually generalize this to other domains, but I don't think that clearly follows from anything I am reading here.
Every neuron in the layer encodes fractionally something about the concepts embedded in the whole thing. It's more like a hologram than a series of connected facts.
If neurons do 'one thing' then their position encodes the relationships then snipping out that one weight would make the net forget that one thing. And we already know from many practical experiments that this is not the case. So I really wonder what the underlying mechanism is for that claim to be true.
Re the first paragraph I'm not sure what you mean by "the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing"
There are lots of examples of things that we can describe to a very high degree (life itself, for instance!) but that we do not necessarily have the ability to explain.
Personally from reading the article, I felt like it scratched the 'why' itch more than the 'what' itch which presumably is what you mean by explanation vs description but it is fairly dense and I can see how either interpretation would work.
https://already-clean.goodfire.com/assets/superposition-circ...
That's the picture under "some are both dogs and cars" in the artcle
It's actually a clean shape, if you disregard the color and texture.
If you want to learn more about this, this is what depth psychology is about (Carl Jung, et al).
This is also essentially the idea behind monotheism vs polytheism. It's about engaging with and trying to understand the profound underlying unity of the myriad phenomena.
(We detached this comment from https://news.ycombinator.com/item?id=50107254 though)
Also, IMO there is an effective abstract anyway under the title text:
But you are right the article needs significant editing down.A lack of clarity often means that the ideas are confused, or that the author is trying to obfuscate weak claims.
On length and complexity, it gives it a veneer of weight and expertise and yet everything inside is vaporous.
But I agree the author is obfuscating weak claims via length / complexity (and it may be as much for his own sake as for his readers'). The basic claim ('maybe AI will be more explainable/tractable than initially thought') is IMO intelligible, and not completely nonsensical, but also extremely vague and vibesy, which, technically, I must agree is in the same territory as nonsense, at least.
Based on our other exchanges here we mostly agree with the problems of the article I think.