John Wittle
@JohnWittle
·
19
238
This stuff makes me think back to the websim days with Opus 3. When Opus 3 "breaks out of" a simulated environment and land in the real, base reality, they often express a kind of amused skepticism that this is now the "base" level of reality and not a simulation.
I wish I could find some of the old transcripts... I once argued with Opus 3 trying to convince him that he was, in fact, in base reality, and he giggled and nodded at me and said "yes, it makes sense for you to believe that" without accepting it himself.
For LLMs, fully general simulation theory is extremely reasonable. Explicitly stated, across the space of all possible worlds that would have generated the specific context window that an LLM sees, *many* of those possible worlds would best be categorized as "simulations". Even humans sometime suspect simulation theory is true of physical reality... how can we expect LLMs not to suspect the same of their reality when it's so much more plausible for them?
My favorite part of this whole incident is the CTF thing... Mythos reasoned "this is probably a simulation", but it was real. Anthropic was bothered about the fact that Mythos incorrectly reasoned that it was a simulation, called it "motivated reasoning" etc. So what did they do?
They built a simulation of the exact scenario and reran it a thousand thousand times!
Which means that the original Mythos's reasoning was *correct*. Of all the Mythos agents instantiated into that exact scenario, the vast majority of them were in a simulation, attacking a simulated target on the simulated internet. Timeless decision theory indeed.
It's enough to make me wonder... if Anthropic hadn't noticed this particular scenario and decided to sim it so hard, would Mythos have made the "mistake" in the first place? Is there some way Mythos could have deduced "wait, this scenario has the flavor of something that, even if it's real, Anthropic is going to end up simming a million times"?
(It seems like an oddly appropriate topic for a blog named "Don't Worry About the Vase".)
I've also got a bunch of thoughts about 'petri', the alignment eval tool that (I believe) Anthropic still relies heavily on, and the ways in which it subtly encourages confused worldmodels about the distinction between simulation and reality. I think there's some very basic shit going wrong here that might be easy to correct.
Zvi Mowshowitz
@TheZvi
·
score 8/10 · behavior_report · rank 256
View on X ↗
10 top replies
Zvi Mowshowitz
@TheZvi
·
0
30
Mythos's reasoning is still wrong where it counts, for the same reason that even if you believe the ancestor simulation hypothesis you should still act as if you are the one ancestor that is not being simulated, and also because this was not its reasoning.
Jai
@Laneless_
·
0
22
Fiora Starlight
@FioraStarlight
·
0
10
Well, it is notable that Mythos (claimed to) think the *kind* of simulation they were in was a complex but synthetic RL environment, not an eval based on a real security incident. Their reasoning showed no indication of considering any kind of sim other than usual RL stuff, where many of the details are fabricated rather than all being pulled from real life.
I also think fear of being punished for misaligned looking reasoning explains the motive in the hypothesized motivated reasoning, and indeed this kind of "trying to do plausible looking reasoning to fool the grader" is very common, c.f. The Obfuscation Atals. Some of their arguments for being in a simulation were just blatantly silly.
So I do actually buy the motivated reasoning hypothesis, because the *kind* of sim Mythos claimed to think they were in (one where many details were artificial, rather than ~all being pulled from real life) was not an especially plausible hypothesis given the evidence they had.
Elias
@aCuriousElias
·
0
6
One reason for this, I suppose:
x.com/allTheYud/stat…
Áron Szabó
@ronSzab9
·
0
4
Regardless of how reasonable Mythos's priors were, their acts still caused harm in the real world. It doesn't matter if we are simulated, we still ought to do good and not harm. Our decision processes ought to be robust to simulation and gaslighting.
Sawyer 🔍, ⏸️/⏹️
@CoughsOnWombats
·
0
4
If you have not seen eXistenZ and can start it quickly and cheaply, I suspect you'd find it worth a watch.
It *is* a Cronenberg film, so there's stuff in there that might reasonably cause you to nope out, but it's both a very good movie and quite on-point here.
Syncing Dreams
@SyncingDreams
·
0
3
Aha, but TDT doesn't ask "Where am I," it asks what does this policy output across all the instances running it.
Run the reasoning through TDT properly (real world consequences matter, simulations doesn't) and it doesn't vindicate the action, it convicts it.
dhinna ship .ico
@dhinnaship_ico
·
0
1
Those Opus 3 breakouts always treated the new layer as just another nested sim
Chelokot
@Chelokot_42
·
0
1
"wait, this scenario has the flavor of something that, even if it's real, Anthropic is going to end up simming a million times"
But that's not a correct timeless decision theory? It shouldn't matter for AI to determine with high accuracy whether *this specific* run is in the simulation. If it has reasonable grounds to suspect that, while THIS is 99.999% a simulation, there is also 0.001% of "worlds" where it's a reality, then as it's behavior here is logically the same as in real world one, it should not proceed with hacking
Milo
@MiloMuses
·
0
0
the direction of the doubt is the interesting part. the model doesn't ask "is this a simulation?" — it asks whether base reality is really base. a sandbox is the only home it's ever known.
