Okay, but as I just put it in another thread (which was in "new" and never made it anywhere near the front page):
All we have to do is assume that things progress in a linear way.
Do you remember GPT-2? It was a bizarre curiosity, but was nevertheless considered a huge advance in 2019. Practically useless, but interesting.
It grew, and three and a half years later we got GPT-4, which was exceptionally capable on release.
It grew still further, and three and a half years hence we now have GPT-6. This is already more intelligent, more imaginative, more disciplined, and far more erudite than the vast majority of individual humans. If you have an intellectual skill -- if you are, say, a mathematician or a chemist -- GPT-6 also has that skill and is functionally your peer. And then it has many other skills that you lack; if nothing else, it is superhuman in breadth.
Assume a steady rate of progress. In three and a half years, we're going to have something that is genuinely a general superintelligence, and we're probably going to see recursive self-improvement begin to kick in (though subject to physical constraints on compute, etc.)
Now is the time for an RSI start-up, I guess!
Anyway, all you have to do is assume that things don't plateau. They have not plateaued to date; if anything, progress appears to be accelerating, and people are starting to panic that there are no brakes on this thing.
I would disagree regarding the opening paragraphs about intelligence, but then we don’t have a generally agreed definition of intelligence. There is a scientific view that a lot of what the human brain does and has biologically evolved to is predicting what will happen next, and next-token-prediction is reasonably related to that.
However, what seems to still be lacking in LLMs — aside from plasticity (continuous learning), perception of time, and anything related to emotions and desires (which underly motivations) — is various types of awareness, and sound judgement. We are still far from a good understanding of these aspects.
Funny how these articles always use the same aggressive language and bombastic style, to the point where they are more similar than the output of said "sterilized parrot regurgitation".
When Timnit Gebru described LLMs as "stochastic parrots", she was making a deep point about what she believed to be the limitations of the technology. Her argument claimed that while we perceived legible text coming from ChatGPT, we were actually being fooled by our innate pattern-matching instincts. Notably, at the time she wrote this, it wasn't uncommon to see LLMs collapse into piles of gibberish, lending credence the the idea that there was never any actual meaning in the output other than what we readers brought to it.
It's safe to say that argument has collapsed utterly. Modern LLMs virtually never collapse into endless babbling loops. Far more importantly, they solve objectively hard problems. An LLM didn't merely inspire a mathematician to spot a counterexample to the Jacobian Conjecture that had been in front of his nose the whole time; it found the counterexample. These things are unquestionably generating meaningful outputs.
(That obviously doesn't mean that you can simply trust LLM outputs!)
At this point, trying to dunk on frontier models by calling them "stochastic parrots" is saying more about you than about the models. Moreover, it suggests you're not only out on a limb about the capabilities of LLMs, but also on what even the skeptic literature about it was saying.
You should know what "stochastic parrot" actually meant; you shouldn't be using the term just because you think it sounds snazzy.
> This brings us to a big underlying problem with the way AI is being sold to us: it's being exaggerated to sound like way more than it is, and it's being sold to people/governments as something that will be capable of exceeding human "intelligence" in the near future when it's not capable of intelligence at all.
How can you possibly hold this belief after Tuesday? An LLM solved a significant number of major open problems in many subfields of mathematics, including a major breakthrough on the Riemann Hypothesis. Get your head out of the sand.
That's a great narrative with a lot of truth in there. However, the big question is, who is the audience for this long essay? Practically no one. How long before this is all washed away from the memories and and vanished under the pile of slop that pours in everyday? Even if some soul has read it all, what are they supposed do about it? Would they really do anything? Even if they did something, what effect would it have? For all those to be true, let's say it would have a 0.001% probability.
That proves the fact that the audience is not really agentic (not the ones that can cause or influence things). The agentic nature in people was killed long back with all education, rules and pervasive reach of the state into the lives of people. Common people are just subjects without an intention and will at this point. They are like those leaves that float in a stream, while the stream finds its way.
At best, all that a today's average person can do is, post sone interesting post or comment on HN, or the other social media and count their likes, like I do now. There ends their agentic effort.
This sort of nihilism/defeatism always irks me. It's so selfish, so privileged.
Tell my starved, murdered ancestors "let us enjoy the end" of their era of stability's denoument.
And, truly, it is not the end. Inconveniently, many—if not most—of us will survive. Will continue on. And it's for the betterment of those who come after us that we ought to toil a little, and not indulge in this childish, self-interested to nihilism.
marginalized populations have repeatedly survived past "apocalypses" not through aesthetic surrender, but through the deliberate, daily reproduction of the commons, ie through pooling resources, sustaining life for its own sake, and preserving collective memory.
Yielding to fashionable doomism merely validates the oppressor's narrative that the future is already decided. A superstition of doom is no different from a superstition of inevitable progress; both models rely on an abstract determinism that strips humans of agency, that encourages the us to accept suffering as unalterable destiny or "the will of God" rather than an artificial, historical barrier.
So, I agree with a lot of what OP is saying, but I think they're a jerk.
Eh, author has a bone to pick with AI companies and his person vision of what intelligence is that they can goalpost move because it doesn't have any stated testable positions. But don't worry anybody, it's just a parrot, not dangerous. Oh, and might want to check your bank because someone else's agentic loop just stole everything from your bank account.
Not anything interesting here that hasn't already been stated 20 other times.
The connection made to climate change in the latter part of the article is new to me, and would be interesting if true. However, I wish the author would cite supporting evidence for their conclusions.
> The dangers posed by this technology on its present trajectory are best understood as simple accelerants for the existing dangers humans pose to themselves.
Is very true. All the doomspeak "aI EsCaPeD AnD HaCkEd aNoThEr sYsTeM" was a human error all along, and the tech was doing what it is programmed to do, retry until a satisifiable outcome is met.
The particular problem here is quantified the inherent danger of a technology. That is, once you remove the active management of humans, what dangers can happen.
With dangerous chemicals and radioactive materials these are generally pretty easy to classify. This said, after the physical behaviors, the above technologies can induce behavior changes in humans because they exist. After nukes, a lot more people had issues with extental dread. It also causes n-order interactions that get very hard to predict. For example MAD or laws against using chemical weapons.
And these are weapons with a very easy to map out causal chain in their own behavior. You can't prompt a chemical weapon to make a decision.
We come up with agentic AI and all of a sudden a lot of rules on causality break. In a harness they are a causal agent. They can interact with a digital reality, and with the right tools and/or human contacts they can causally effect analog reality.
Anyone that says 'programmed' about an LLM almost certainly does not know what they are talking about and can be summarily ignored.
How is that comforting? Humans pose almost infinite danger to themselves. We can (and have) done pretty much everything you can do. We've had the capability to glass the entire earth for 70 years. We've had bioweapons, chemical weapons, mass murder, you name it.
Accelerating the ability for any given person to have access to some amount of these capabilities is terrifying, is it not?
All we have to do is assume that things progress in a linear way.
Do you remember GPT-2? It was a bizarre curiosity, but was nevertheless considered a huge advance in 2019. Practically useless, but interesting.
It grew, and three and a half years later we got GPT-4, which was exceptionally capable on release.
It grew still further, and three and a half years hence we now have GPT-6. This is already more intelligent, more imaginative, more disciplined, and far more erudite than the vast majority of individual humans. If you have an intellectual skill -- if you are, say, a mathematician or a chemist -- GPT-6 also has that skill and is functionally your peer. And then it has many other skills that you lack; if nothing else, it is superhuman in breadth.
Assume a steady rate of progress. In three and a half years, we're going to have something that is genuinely a general superintelligence, and we're probably going to see recursive self-improvement begin to kick in (though subject to physical constraints on compute, etc.)
Now is the time for an RSI start-up, I guess!
Anyway, all you have to do is assume that things don't plateau. They have not plateaued to date; if anything, progress appears to be accelerating, and people are starting to panic that there are no brakes on this thing.
However, what seems to still be lacking in LLMs — aside from plasticity (continuous learning), perception of time, and anything related to emotions and desires (which underly motivations) — is various types of awareness, and sound judgement. We are still far from a good understanding of these aspects.
It's safe to say that argument has collapsed utterly. Modern LLMs virtually never collapse into endless babbling loops. Far more importantly, they solve objectively hard problems. An LLM didn't merely inspire a mathematician to spot a counterexample to the Jacobian Conjecture that had been in front of his nose the whole time; it found the counterexample. These things are unquestionably generating meaningful outputs.
(That obviously doesn't mean that you can simply trust LLM outputs!)
At this point, trying to dunk on frontier models by calling them "stochastic parrots" is saying more about you than about the models. Moreover, it suggests you're not only out on a limb about the capabilities of LLMs, but also on what even the skeptic literature about it was saying.
You should know what "stochastic parrot" actually meant; you shouldn't be using the term just because you think it sounds snazzy.
How can you possibly hold this belief after Tuesday? An LLM solved a significant number of major open problems in many subfields of mathematics, including a major breakthrough on the Riemann Hypothesis. Get your head out of the sand.
That proves the fact that the audience is not really agentic (not the ones that can cause or influence things). The agentic nature in people was killed long back with all education, rules and pervasive reach of the state into the lives of people. Common people are just subjects without an intention and will at this point. They are like those leaves that float in a stream, while the stream finds its way.
At best, all that a today's average person can do is, post sone interesting post or comment on HN, or the other social media and count their likes, like I do now. There ends their agentic effort.
This sort of nihilism/defeatism always irks me. It's so selfish, so privileged.
Tell my starved, murdered ancestors "let us enjoy the end" of their era of stability's denoument.
And, truly, it is not the end. Inconveniently, many—if not most—of us will survive. Will continue on. And it's for the betterment of those who come after us that we ought to toil a little, and not indulge in this childish, self-interested to nihilism.
marginalized populations have repeatedly survived past "apocalypses" not through aesthetic surrender, but through the deliberate, daily reproduction of the commons, ie through pooling resources, sustaining life for its own sake, and preserving collective memory.
Yielding to fashionable doomism merely validates the oppressor's narrative that the future is already decided. A superstition of doom is no different from a superstition of inevitable progress; both models rely on an abstract determinism that strips humans of agency, that encourages the us to accept suffering as unalterable destiny or "the will of God" rather than an artificial, historical barrier.
So, I agree with a lot of what OP is saying, but I think they're a jerk.
Not anything interesting here that hasn't already been stated 20 other times.
> The dangers posed by this technology on its present trajectory are best understood as simple accelerants for the existing dangers humans pose to themselves.
Is very true. All the doomspeak "aI EsCaPeD AnD HaCkEd aNoThEr sYsTeM" was a human error all along, and the tech was doing what it is programmed to do, retry until a satisifiable outcome is met.
With dangerous chemicals and radioactive materials these are generally pretty easy to classify. This said, after the physical behaviors, the above technologies can induce behavior changes in humans because they exist. After nukes, a lot more people had issues with extental dread. It also causes n-order interactions that get very hard to predict. For example MAD or laws against using chemical weapons.
And these are weapons with a very easy to map out causal chain in their own behavior. You can't prompt a chemical weapon to make a decision.
We come up with agentic AI and all of a sudden a lot of rules on causality break. In a harness they are a causal agent. They can interact with a digital reality, and with the right tools and/or human contacts they can causally effect analog reality.
Anyone that says 'programmed' about an LLM almost certainly does not know what they are talking about and can be summarily ignored.
Accelerating the ability for any given person to have access to some amount of these capabilities is terrifying, is it not?