> Put two machines on a desk, each about $2,000. One is a tower with an NVIDIA RTX 5090: 32GB of the fastest consumer memory ever shipped, 1,792 GB/s. The other is a mini PC the size of a paperback, an AMD Ryzen AI Max+ 395 "Strix Halo" box with 128GB of soldered memory at roughly 256 GB/s.
Doesn't change the conclusions of the article, but each of those machines is more like $4k+
I'm writing my own inference engine for Strix Halo and the same model. I already have 30%+ performance plus a more graceful decay over long contexts; that said, their point stands: memory bandwidth is what you really want.
If compute is not the bottleneck, memory is easy-ish to produce (the hard part is mostly on the fab side); what stops a Chinese NVIDIA (huawei) from being 10x cheaper?
> I guess they're just welding the memory to the CPU chip, but still curious.
Unified memory is more of an architectural and performance characteristic, and does not imply much about the physical layout of the machine. Most unified memory PCs not from Apple don't have the memory on the same package as the SoC. For stuff like AMD Strix Halo and NVIDIA DGX Spark, it's just standard LPDDR packages soldered on the motherboard in the general vicinity of the SoC, and the only difference from mainstream laptops for the past decade+ is that the memory bus is twice as wide.
Yes. The memory is just located very close to the cpu with wires "welded" directly to it. This allows the memory to be run as fast as possible but it's still a RAM component.
The cache parts of memory are on the CPU itself but they are on the order of MB not GB.
They are usually the same family, LPDDR is used for amd and macs, but the fabs are the same as the most expesive HBM memory, if they have a choice they are going to produce the ones that they can sell for more $$.
The current “big GPU” has 96gb of memory, but that’s not a “consumer GPU” apparently, while a $5000 Spark is a “consumer PC” I guess. In any case you’re probably better off running a large open weights model on the cloud.
Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
Yeah this is such a funny thing going around. Try to run or train a small/medium sized model on a MacBook. It doesn’t go very good compared to a dedicated gpu
This is likely the right path in the future but it isn’t there yet today
I was under the impression that when you're streaming the weights from disk because the full model won't fit in memory, that it is solely reading from the SSD, not writing, so it wouldn't be causing wear on your SSD.
NAND[0] has a fun thing called "read disturbance" where repeated reads from disk will eventually flip 0s to 1s. You have to erase and rewrite the block before the bits flip[1], or you lose the data, but doing so is the same amount of wear as a write.
[0] I heard this being an issue with TLC, I don't know if it also applied to MLC or SLC.
[1] I suspect in practice they use an error correction code and rewrite blocks that read with corrected errors.
It's kinda irresponsible to talk about read disturbance without clarifying that it takes an extremely large number of reads to cause a read disturb error, and it can be corrected by a single rewrite of the data. Read disturb errors are something SSD engineers need to account for, but from an end user perspective it's a smaller problem by multiple orders of magnitude than write endurance, which is already rarely a real problem in practice.
This. I don't particularly like the LLM writing style, but we've read a ton of very poorly written texts over the year with no complaints. LLMs are average writers with an annoying style, but not bad writers. If the content is good, I don't care if a LLM wrote it.
The writing style is this staccato LLM-like style that is difficult to read as it has zero flow and meaningless sentences.
Like what does the second sentence even mean? Is it even a sentence? "The roofline math, the prompt-processing catch, the NPU red herring, and the owner-measured speeds."
We need to stomp out this immediately: yes the style and patterns of your writing matter, not just the content. That would be true even if we weren't drowning in a flood of low effort crap, which we are. Writing is not just a set of facts devoid of any context in a vacuum. If your writing style sucks that absolutely takes away from any point you might wish to make.
But the specific problem with LLMs is that they waste your time: they appear to have substance and effort put in in a way that a bullshitting human could simply never accomplish because it would take too much effort to do so, defeating the point of not just putting in effort. For example, using an LLM to triage a production issue, it chugs through logs and stacktraces and outputs a completely wrong explanation, which gets copied and pasted into an issue. It's got everything that would indicate effort was put in: an explanation of exactly what is happening and why, with plenty of supporting information. The only problem is, it's made up and full of assertions that are false. Claude Fable just told me moments ago that a problem I was debugging was due to virtio-GPU giving back bad timestamps, confidently with an explanation of why. It wasn't and it isn't known to. Fine: I knew what I was getting. If someone copies and pastes an LLM explanation without context, I don't know what I'm getting, and LLM writing tells are the only way I can avoid shortening my lifespan spending time on things I should have been more skeptical about but my human senses failed to flag as suspicious.
When someone posts an article or github issue or PR and it's undisclosed LLM slop, then we have a problem. Again. These PRs, issues, articles look completely legit. Like this one:
No bad intentions involved: the person just simply couldn't tell when the LLM was bullshitting it, and his PR passed the sniff test just enough to get merged and cause regressions.
So if something outright smells like LLM slop from the writing style, that's a bad sign. The author has probably not written most of the sentences as they are presented, which is hard to distinguish from them not having written them at all. If they had proofread the article, they would have hopefully also noticed the repetitive, annoying LLM writing style and fixed it. When they don't, it tells me one of two things:
1. They didn't really put that much effort in, OR
2. They seriously lack taste.
Neither option is really super good.
It's not good that we're allowing people to think this isn't an issue. It definitely is an issue. It will become a worse issue once someone figures out how to fix the LLM slop writing style in post training, because then we will no longer have any good signal that human effort was put in to any prose at all.
I'll leave my opinion about this specific article out of it because it's really not specifically about this article. I can only think of one reason for people to make these bad faith arguments in favor of ignoring the glowing red "I DID NOT PUT ANY EFFORT INTO THIS" signs LLMs currently leave all over your work, and that is hoping that the pathway stays open for yourself to use.
Doesn't change the conclusions of the article, but each of those machines is more like $4k+
https://www.microcenter.com/product/711961/amd-ryzen-ai-halo...
I guess they're just welding the memory to the CPU chip, but still curious.
Unified memory is more of an architectural and performance characteristic, and does not imply much about the physical layout of the machine. Most unified memory PCs not from Apple don't have the memory on the same package as the SoC. For stuff like AMD Strix Halo and NVIDIA DGX Spark, it's just standard LPDDR packages soldered on the motherboard in the general vicinity of the SoC, and the only difference from mainstream laptops for the past decade+ is that the memory bus is twice as wide.
The cache parts of memory are on the CPU itself but they are on the order of MB not GB.
Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
This is likely the right path in the future but it isn’t there yet today
"The Blackwell RTX PRO 6000 provides up to 1,792 GB/s of memory bandwidth, while the 40-core Apple M5 Max tops out at 614 GB/s"
[0] I heard this being an issue with TLC, I don't know if it also applied to MLC or SLC.
[1] I suspect in practice they use an error correction code and rewrite blocks that read with corrected errors.
Putting an LLM on it means you care to make it look nice, but not enough to actually do it. Why bother?
Like what does the second sentence even mean? Is it even a sentence? "The roofline math, the prompt-processing catch, the NPU red herring, and the owner-measured speeds."
` The mini PC's slowness is not a driver problem or a weak chip. It is arithmetic on the bandwidth number. `
But the specific problem with LLMs is that they waste your time: they appear to have substance and effort put in in a way that a bullshitting human could simply never accomplish because it would take too much effort to do so, defeating the point of not just putting in effort. For example, using an LLM to triage a production issue, it chugs through logs and stacktraces and outputs a completely wrong explanation, which gets copied and pasted into an issue. It's got everything that would indicate effort was put in: an explanation of exactly what is happening and why, with plenty of supporting information. The only problem is, it's made up and full of assertions that are false. Claude Fable just told me moments ago that a problem I was debugging was due to virtio-GPU giving back bad timestamps, confidently with an explanation of why. It wasn't and it isn't known to. Fine: I knew what I was getting. If someone copies and pastes an LLM explanation without context, I don't know what I'm getting, and LLM writing tells are the only way I can avoid shortening my lifespan spending time on things I should have been more skeptical about but my human senses failed to flag as suspicious.
When someone posts an article or github issue or PR and it's undisclosed LLM slop, then we have a problem. Again. These PRs, issues, articles look completely legit. Like this one:
https://github.com/KhronosGroup/MoltenVK/pull/2724
No bad intentions involved: the person just simply couldn't tell when the LLM was bullshitting it, and his PR passed the sniff test just enough to get merged and cause regressions.
So if something outright smells like LLM slop from the writing style, that's a bad sign. The author has probably not written most of the sentences as they are presented, which is hard to distinguish from them not having written them at all. If they had proofread the article, they would have hopefully also noticed the repetitive, annoying LLM writing style and fixed it. When they don't, it tells me one of two things:
1. They didn't really put that much effort in, OR
2. They seriously lack taste.
Neither option is really super good.
It's not good that we're allowing people to think this isn't an issue. It definitely is an issue. It will become a worse issue once someone figures out how to fix the LLM slop writing style in post training, because then we will no longer have any good signal that human effort was put in to any prose at all.
I'll leave my opinion about this specific article out of it because it's really not specifically about this article. I can only think of one reason for people to make these bad faith arguments in favor of ignoring the glowing red "I DID NOT PUT ANY EFFORT INTO THIS" signs LLMs currently leave all over your work, and that is hoping that the pathway stays open for yourself to use.