Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make.
If you look at it, the diagrams are very similar, but the main differences are that the feedforward is replaced with a MoE (router to multiple feedforwards) and the model has a different attention implementation.
I think the point stands: MoE, a myriad of complex attention approaches, shared layers, you name it. And making it all work together well is a huge trial-and-error pain even for small models, never mind getting to efficient hardware utilization.
The source is the same in the original article too. He is using a different diagram from the same site on the right to justify his point on how much more complicated things have become.
I am _very_ familiar with Claudish, and to some extent, the other AIs' writing styles. This article is human-written and features human writing quirks.
The very first sentence
> Back in 2022 and 2023 there were two big branches of machine learning happening at Meta.
is unmistakably human. That's not how a LLM would phrase this sentence, and if it did, it would have put a comma after 2023.
I am a professional writer and have been for over 30 years. (I do not use any form of LLM ever.) This means I read a lot. This also means that I have 30+ years of experience of readers not understanding what I wrote, or not getting further than the title, or not getting the main message, or inverting it in their heads, or inserting their own message and then complaining when I diverge, and an endless list of Ways People Do Not Get It.
I am also a trained TESOL teacher. Ability to capture gist is a skill we test for and measure, and many, maybe the majority, of native speakers don't have it and don't know.
In recent years I constantly see people going "this is written by AI" and I have yet to see a single of of them able to coherently prove their point. It's all just feelings and hunches.
So I am calling you on this:
How do you know? Show your working. Demonstrate your case.
Claude's writing style is at least as distinctive as any human's personal style. It has a long list of favorite words, verbal tics and common structures. On top of that, LLM writing is often bad in a very particular way: it's weak on actual things to say, but with an overheated style.
Some days, I spend over 4 hours a day reading walls of text written by Claude. If I couldn't recognize Claude's default "voice" by now, something would be wrong. It would be like a Hemingway fan not being able to recognize Hemingway. Except more so, because Claude's writing style is getting worse from version to version, descending into self parody.
On the statistical side, Pangram's model identifies AI-authored text with a 1-in-5,000 false positive rate, measured against hold-out texts from before 2022. My "ear" also agrees closely with Pangram. If I think something sounds AI written, Pangram virtually always comes back with "AI, confidence: high."
You need to start using LLMs a lot and then you will know how we know.
Edit: You know how you can recognise someone just from their gait while they walk towards you? I would struggle to describe that for an individual person but it doesn't mean I can't identify them from that alone.
But AI written pieces do have a certain feeling. A sort of saccatto in the succession of ideas that does not feel natural. They emphasize certain points, and you as a reader, you just wonder why is that. There is the “This thing, not just that thing”. There are also the three successive propositions (mostly in one sentences) to accentuate an idea and “Negation. Strong positive idea in the same direction”.
In general try reading one (vocally) to yourself and it will feel really weird.
https://sebastianraschka.com/llm-architecture-gallery/?compa...
If you look at it, the diagrams are very similar, but the main differences are that the feedforward is replaced with a MoE (router to multiple feedforwards) and the model has a different attention implementation.
I think the point stands: MoE, a myriad of complex attention approaches, shared layers, you name it. And making it all work together well is a huge trial-and-error pain even for small models, never mind getting to efficient hardware utilization.
The page links to the same site you do. No wonder it is similar -- the source is the same!
The very first sentence
> Back in 2022 and 2023 there were two big branches of machine learning happening at Meta.
is unmistakably human. That's not how a LLM would phrase this sentence, and if it did, it would have put a comma after 2023.
I am a professional writer and have been for over 30 years. (I do not use any form of LLM ever.) This means I read a lot. This also means that I have 30+ years of experience of readers not understanding what I wrote, or not getting further than the title, or not getting the main message, or inverting it in their heads, or inserting their own message and then complaining when I diverge, and an endless list of Ways People Do Not Get It.
I am also a trained TESOL teacher. Ability to capture gist is a skill we test for and measure, and many, maybe the majority, of native speakers don't have it and don't know.
In recent years I constantly see people going "this is written by AI" and I have yet to see a single of of them able to coherently prove their point. It's all just feelings and hunches.
So I am calling you on this:
How do you know? Show your working. Demonstrate your case.
Some days, I spend over 4 hours a day reading walls of text written by Claude. If I couldn't recognize Claude's default "voice" by now, something would be wrong. It would be like a Hemingway fan not being able to recognize Hemingway. Except more so, because Claude's writing style is getting worse from version to version, descending into self parody.
On the statistical side, Pangram's model identifies AI-authored text with a 1-in-5,000 false positive rate, measured against hold-out texts from before 2022. My "ear" also agrees closely with Pangram. If I think something sounds AI written, Pangram virtually always comes back with "AI, confidence: high."
Edit: You know how you can recognise someone just from their gait while they walk towards you? I would struggle to describe that for an individual person but it doesn't mean I can't identify them from that alone.
But AI written pieces do have a certain feeling. A sort of saccatto in the succession of ideas that does not feel natural. They emphasize certain points, and you as a reader, you just wonder why is that. There is the “This thing, not just that thing”. There are also the three successive propositions (mostly in one sentences) to accentuate an idea and “Negation. Strong positive idea in the same direction”.
In general try reading one (vocally) to yourself and it will feel really weird.