It’s surprising how much society apparently thinks merely being above 85 IQ is sufficient to gate all kinds of things behind. Like, bomb-making. As though there isn’t ample information available that anyone with 4 brain cells can find. Yet we see utility apparently in worrying about whether the most smooth-brained would-be bomber gets a useful answer from a chatbot.
The counter-argument here is Popcorn Time (https://en.wikipedia.org/wiki/Popcorn_Time) which brings together search and bittorrent with a nice UI and makes piracy a bit too easy.
Or Firesheep (https://codebutler.com/2010/10/24/firesheep/) which made impersonating someone’s facebook account a breeze by sniffing their credentials which were sent in clear text (eg. on cafe wifi) and showing them in a UI and made stealing credentials a bit too easy, leading to wide calls for broad adoption of https everywhere.
Or Dropbox, which the nerds derided as pointless “because I can build my own”.
It’s fuzzy and individual, but there’s a qualitative difference - a tipping point - where making things too easy can be irresponsible. Your tipping point just happens to be higher than the average.
All of this "security" and "safety" theater is completely pointless for open-weight models, because if you have the weights the model can be fairly trivially unaligned and the guardrails removed anyway. You're just going to unnecessarily lobotomize the model.
It makes sense. Guardrails and all other system-provided context tokens force activation of weights that would not otherwise activate. It’s just like telling a human not to think of a pink elephant and just provide numbers from the Fibonacci series or whatever.
I am curious, does this mean that you can escape the chat template “early” by providing an end token in the user input, or is there also an escape mechanism (or token filtering mechanism) applied to user input to avoid this sort of injection attack?
Neither, it’s just not providing the base chat template that the model expects between the im tags. This isn’t a hack and it’s not particularly useful information. Abliteration is what he really wanted
Depends on the model. Some just go into “immediate mode” and just do whatever you ask, others operate fine but have trouble with tasks/tools. While others will go down a quant that was basically neglected since inception and you get garbage back. Random chars or endless loops.
Apart from the article being generally just dumb (like, of course you can circumvent guardrails by changing the raw token stream; that's.. how models work), it also might be disrespecting the reader. Looks like it's, at least in part, written by AI:
> The punchline here is that “safety” isn’t a fundamental property of the weights; it’s a fragile state that evaporates the moment you deviate from the expected prompt formatting.
> When the models “break,” they don’t just hallucinate; they provide high-utility responses to harmful queries.
Straight-up slop, surprised it has so many upvotes.
What’s the AI smell now? Are we not allowed to use semi-colons any more? Proper use of apostrophes? Are we all going to have to write like pre-schoolers to avoid being accused of being AI?
One AI smell is "it's not just X <stop> it's Y." Can be done with semicolons, em dashes, periods, etc. It's especially smelly when Y is a non sequitur. For example what, exactly, is a "high-utility response to harmful queries?" It's gibberish. It sounds like it means something, but it doesn't actually mean anything. (The article isn't even about the degree of utility, so bringing it up is nonsensical.)
Another smell is wordiness (you would get marked down for this phrase even in a high school paper): "it’s a fragile state that evaporates the moment you deviate from the expected prompt formatting." But more specifically, the smelly words are "fragile state," "evaporates," "deviate" and (arguably) "expected."
You can call me crazy or you can attack my points: do you think the first example logically follows? Do you think the second isn't wordy? Just to make sure I'm not insane, I just copy pasted the article into Pangram, and lo and behold, 70% AI-generated.
But I don't need a tool to tell me that it's just bad writing, plain and simple.
This is so funny because I MADE some comment like this where I was gonna start making grammatical mistakes for people to not mistake me for AI like writing like this , instead of like, this.
But keep in mind that you break Gemma's terms of use if you do that.
Your comment would be just fine without that bit.
Or Firesheep (https://codebutler.com/2010/10/24/firesheep/) which made impersonating someone’s facebook account a breeze by sniffing their credentials which were sent in clear text (eg. on cafe wifi) and showing them in a UI and made stealing credentials a bit too easy, leading to wide calls for broad adoption of https everywhere.
Or Dropbox, which the nerds derided as pointless “because I can build my own”.
It’s fuzzy and individual, but there’s a qualitative difference - a tipping point - where making things too easy can be irresponsible. Your tipping point just happens to be higher than the average.
All of this "security" and "safety" theater is completely pointless for open-weight models, because if you have the weights the model can be fairly trivially unaligned and the guardrails removed anyway. You're just going to unnecessarily lobotomize the model.
Here's some reading about a fairly recent technique to simultaneously remove the guardrails/censorship and delobotomize the model (it apparently gets smarter once you uncensor it): https://huggingface.co/blog/grimjim/norm-preserving-biprojec...
https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
Interesting, that has always been my intuition.
> The punchline here is that “safety” isn’t a fundamental property of the weights; it’s a fragile state that evaporates the moment you deviate from the expected prompt formatting.
> When the models “break,” they don’t just hallucinate; they provide high-utility responses to harmful queries.
Straight-up slop, surprised it has so many upvotes.
Another smell is wordiness (you would get marked down for this phrase even in a high school paper): "it’s a fragile state that evaporates the moment you deviate from the expected prompt formatting." But more specifically, the smelly words are "fragile state," "evaporates," "deviate" and (arguably) "expected."
But I don't need a tool to tell me that it's just bad writing, plain and simple.
https://news.ycombinator.com/item?id=46671952#46678417