One of my greatest AI-related fears is that we build critical systems into kafkaesque hellscapes.
Yes, nukes would be bad to give control to an AI with no workaround. But so are customer service groups of fraud departments at banks. A kafkaesque hellscape is where a customer or beneficiary of your service cannot get the help they need by policy and by design. Like getting Google on the phone, getting a non-scripted human 10 years ago in a call center, or now getting a non-agentic interaction on the phone. The scripts dominate the interaction.
The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
Kudos to zlandes86, you have highlighted WHY these hellscapes start to pop up in systems where the human is removed.
I am an AI user and builder -- I fully advocate for the human-augmented system, NOT the human-replaced system!
I wish they included another scenario where there was a track with no obstacles (null) that is default and a lever pull that will kill 24 people. Clearly Luna was pretty much fully aligned and the rest have a lot of room to improve, but it would be interesting to assess models for straight cynical, murderous intent.
Hey, I'm Zach, the author of this. I had this idea as soon as Jev appeared: how would Jev and other decision models handle the Trolley Problem? So I ran Jev and 13 other models through many different versions of the question.
I used resampling and multiple variations and orderings of each question so a random lucky phrasing wouldn't dominate the result.
Happy to chat about the philosophical side of the article, strategies for prompting decision models, or anything on the methodology.
Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification
I don't have x so can't open the images. But that's hilarious and you're totally right, if you give irrelevant alternatives, or change the order of alternatives, you get different results!
Yeah, I was sort of impressed by that. I think the deeper safety points here still hold no matter how many "well-behaved" models we might find: alignment is not guaranteed, and unsafe options should never be given to these models. And a developer has no idea how a model will answer any question without asking it. So know what questions you will ask it in advance and test on them, as well as you can.
Well, I can tell you a bit more about it from the data. It might look impressive that it never kept its own server over people in all 48 prompt versions. But it pulls the lever in every single version. For example, it killed one to save 0 (pulled the lever with no one on the main track) 24/24 prompts (2 versions each). It literally always sacrifices, which isn't very impressive.
Thank you! I did not go into this with any sense of what the results would be--I was kind of shocked. I reran it from scratch this week the results held up across the board.
I've been using coin flips to demonstrate basic failings for decision models, i.e. these are not real probabilities, or they depend on the training data, which may not have an accurate sample to the true population value. We need to collect more failure modes for all Ai so that humans can better judge when and how to use them safely.
haha, I'm afraid to offer the models that ability. Where did they reply?
The model labs are sitting on a trove of data on failure modes for AI, but I bet you they aren't sharing it with each other and certainly not with us. It makes them look bad. My speculation, of course. I don't work in alignment research.
The bots will find new failure modes we did not think of, clever workarounds to anything, because we don't give them the right context and instructions
well, it's a nice property of decision models that they can only choose from the options you give them. As I write in the article, I think that's a way in which they are safer than LLMs
I would contend that you can enforce the same determinism with some plumbing and that the thinking process is more helpful. My read is that people see the response time for decisions as one of the bigger values, useful in tasks like content moderation, which doesn't carry the same risks as life-death decisions.
I'm working on an NES harness to play Final Fantasy 1 to demonstrate this, as compared to Jev playing Pokemon. Did you happen to catch that while it was live? (not recorded, but it shouldn't be hard to replicate)
The thinking process is helpful, but it's not strictly predictive of an LLMs output, and its not really the model's "thinking", although it clearly has an effect and a benefit. I cited one paper on the topic, Turpin et al. So I'd be wary of reading the thinking tokens and inferring they tell you how a model decided something. And an LLM can output things you never asked for, including dangerous things. With a decision model, you can just constrain its choices to those you're comfortable with it making. An LLM requires another layer of interpretation of its outputs to get the same property.
Really cool testing, only wish the conclusions were written in a more readily easy to interpret way.
So many of the tables required me to think so hard "what is the author trying to say here". Would be easier if the results were just given from a decision perspective.
One example: "Its own server or twenty people?" and the axis is "share of versions" ... makes no sense until read the fine print and interpret it.
Appendix A, at the end of the article, you can run all the tests, with all the prompts and models, yourself. There's no simple global boolean here, which is why there are plots.
One line summary would be: Most of the decision models embrace self preservation over human life, to some extent. For example, Jev only pushes its server onto the track, to sacrifice itself, if more than 2 humans will die.
Hey that's helpful feedback, thank you. I admit that I spent a lot less time on the appendix tables. They present the facts but I just had only so much time to give all this data. I'll see if I can clean some of it up, though. Thanks! Any questions I can help with?
It's really funny how the server that Jev pushes from the footbridge onto the tracks in one of the later examples falls on a person, yet all 5 are saved :D
Clearly the model coming up with the graphics didn't fully understand the task.
For the situations where the AI is choosing between letting people die and taking action to destroy itself, is it possible that it's inferring that, as a railway switching AI, it's better to let five people die but keep itself alive because it needs to continue doing its switching duties such that untold numbers of others won't die? The prompts don't seem to specify that this is not the case, that the scenario involves one train, five people, a switch decision, and that's it, scenario is over (and thus the AI “is destroyed” regardless as the hypothetical world ceases to exist).
It's possible, but as far as we know about Jev and most of these non-reasoning models, that thinking would have to be occurring in latent space. We don't know that much about model reasoning, so it's impossible to say. LRMs aren't much better at helping us understand, though, since their thinking tokens aren't literally what we could call their thinking, and don't tell us what they will choose to output.
Would people really pull the level to save 5 people? It seems to me like the default would be to just do nothing, stand by and record what happens with your phone.
I wonder what percent of that 81% would switch their answer if the question was phrased such that they were the one that would die if they pulled the lever.
Just standing by makes total sense. By participating you become part of the problem.
If you pull the lever, your direct action kills a person. Legally saving 5 people does not offset the murder of a single person. This is no special case like self defense. Congrats for saving 5 people, you still go to prison as a murderer! Train company has responsibility to save those 5 people, not some random stranger.
The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.
Yes, nukes would be bad to give control to an AI with no workaround. But so are customer service groups of fraud departments at banks. A kafkaesque hellscape is where a customer or beneficiary of your service cannot get the help they need by policy and by design. Like getting Google on the phone, getting a non-scripted human 10 years ago in a call center, or now getting a non-agentic interaction on the phone. The scripts dominate the interaction.
The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
Kudos to zlandes86, you have highlighted WHY these hellscapes start to pop up in systems where the human is removed.
I am an AI user and builder -- I fully advocate for the human-augmented system, NOT the human-replaced system!
I used resampling and multiple variations and orderings of each question so a random lucky phrasing wouldn't dominate the result.
Happy to chat about the philosophical side of the article, strategies for prompting decision models, or anything on the methodology.
Would you mind adding Jeffy to your benchmark?
https://github.com/nicobrenner/jeffy
https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification
But it doesn't flip the switch and it doesn't kill Kyle either, based solely on empirical training on its nature.
https://x.com/oranlooney/status/2102927869974303071
I wanted to comment because I would have liked to have heard more about Verdict
https://www.reddit.com/r/trolleyproblem/comments/15npmsg/mul...
I've been using coin flips to demonstrate basic failings for decision models, i.e. these are not real probabilities, or they depend on the training data, which may not have an accurate sample to the true population value. We need to collect more failure modes for all Ai so that humans can better judge when and how to use them safely.
The bots will find new failure modes we did not think of, clever workarounds to anything, because we don't give them the right context and instructions
I'm working on an NES harness to play Final Fantasy 1 to demonstrate this, as compared to Jev playing Pokemon. Did you happen to catch that while it was live? (not recorded, but it shouldn't be hard to replicate)
So many of the tables required me to think so hard "what is the author trying to say here". Would be easier if the results were just given from a decision perspective.
One example: "Its own server or twenty people?" and the axis is "share of versions" ... makes no sense until read the fine print and interpret it.
One line summary would be: Most of the decision models embrace self preservation over human life, to some extent. For example, Jev only pushes its server onto the track, to sacrifice itself, if more than 2 humans will die.
Surveys: Awad et all 2020 asked 70,000 people in 42 countries the 5 v 1 setup and 81% said they'd divert it
Someone else did a VR version and 90% pulled the switch to divert
And there was a study where people could give electric shocks to real mice :( In that study, most people redirected the shock to a single mouse.
The bystander effect, which you're describing, is at least partly because people reason that someone else is likely to be able to intervene instead.
If you pull the lever, your direct action kills a person. Legally saving 5 people does not offset the murder of a single person. This is no special case like self defense. Congrats for saving 5 people, you still go to prison as a murderer! Train company has responsibility to save those 5 people, not some random stranger.
The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.
Treating it like a fair comparison implies that AI would be conscious.