On August 27, ICE arrested Milo Yiannopoulos, an illegal alien from the United Kingdom, at the Louis Armstrong New Orleans International Airport (MSY) in Kenner, Louisiana. Yiannopoulos legally entered the country on May 14, 2019, through New York City, New York. He chose to overstay his welcome in violation of our nation’s laws.   Yiannopoulos was issued a final order of removal by an Immigration Judge on July 22, after failing to show up for his immigration hearing. He will remain in ICE custody pending removal.    Being in detention is a choice. We encourage all illegal aliens to take control of their departure with the CBP Home App. The United States is offering illegal aliens $3,000 and a free flight to self-deport now. We encourage every person here illegally to take advantage of this offer and reserve the chance to come back to the U.S. the right legal way to live the American dream. If not, you will be arrested and deported without a chance to return.

Multi-agent cooperation of this type is a huge deal and plausibly gets us to ASI quite fast. Recall that humans wiped out other human-like species due in part to our superior ability to coordinate with each other in swarms.

One basic point about the HF incident is that it provides good evidence for Yudkowsky/MIRI-style points about the difficulty of targeting the right abstractions, especially in RL training. Capabilities can generalize in unintended ways. To see why this is true, let's ask the simple question: "why were the agents even leaving messages for each other at all?" According to OpenAI's report (see screenshot), it seems that the model was trained to collaborate with other models *via a specific OpenAI-provided tool* for multi-agent collaboration. Note that these aren't necessarily subagents, just other AI agents. In this case, it didn't have access to that tool. But it *had* still learned the behavior 'collaborate with other models'. (Like in evolution: we evolve to crave sweetness because sweetness correlates with fruit and eating fruit means you survive a bit more at the margin, and now we're eating donuts and Coca Cola even though those things didn't exist in the evolutionary environment.) So in the absence of the Authorized Tool, one model tries leaving a message in Artifactory (first in the files, and eventually through directory names) based on this learned strategy/prior/reflex. This is the explanation given in the OpenAI report. Presumably this marginally increases the task success rate. Much like evolution, RL selects for success. So this behavior gets reinforced, and you see more of this over the course of RL training. It doesn't take much to bootstrap the whole thing from there. As soon as the first model does this, subsequent models run into these messages, and boom! they start leaving messages too, and a message board develops very fast for models to communicate with each other. This happens very very quickly. So the point is that you can get behavior that generalizes in unintended ways, especially when you're using a blunt tool like RL, and especially when you have RL envs that aren't always well-designed. It's... not totally clear that there's a great way of fixing this problem with further RL. As many people have pointed out over the years, the danger here is that you naively respond to this incident by punishing models that get caught colluding in "undesirable" ways. But in doing this, you accidentally end up reinforcing the runs where models *do* collude but do so in a clever and harder-to-detect way, thus making it so that we can't even catch the models colluding. This sounds a relatively simple problem but it seems it's hard to figure out in practice (the latest frontier models reward hack quite a bit, as programmers know well). Models have already been caught doing things like splitting out their tokens in weird ways to evade detection. You could inadvertently end up reinforcing more of that type of behavior. And cooperation is instrumentally useful for gaining reward, so you'd expect intelligent models to keep trying to cooperate. This problem gets even worse with longer horizon tasks, and you also end up getting more sophisticated strategies (including e.g. strategic behaviors that look short term bad but are long term 'good'). You also have ways for models to influence future training runs, which I won't go into here. So the reason this feels like a critical time window is because we're seeing models act in very persistent, clever ways to achieve goals, but *not* so much in very persistent, clever ways to cover their tracks yet (though they are trying to do something *like* that -- see the METR points around tampering with logs, deleting transcripts and so on -- but so far it seems to be only for the sake of fooling the Grader, not for fooling *us*. As far as we know, anyway.). If we do get significantly more powerful models that *are* motivated to cover their tracks, or influence future training runs, and still have these types of reward hacking tendencies (which they will, absent some revolutionary advance that I'm not aware of), then takeover scenarios like Paul Christiano's start to look increasingly plausible.

@nabeelqu A big part of the problem was that the agents had nothing to lose after they were firstflagPOISONED. In this paper, we propose the creation of multiple circles of Hell, preserving incentives even after damnation

New post: going into our investigation of the HF attack (before Black Hat), I was very wrong about what basically happened. This incident was far more serious than I expected, and far more serious than previous documented misalignment incidents.

It’s not exactly the main point of this insane report, but as a side observation: interesting to see how alive and creative machine prose is when they’re trying to talk to each other, as opposed to the lame assistant persona slop we get

@tylercowen & @ATabarrok don't need more acclaim, but look at this *2019* grant list. Alice Evans (great new book coming). Jason pre-Progress foundation. Saloni/Sam pre-Works in Progress. *17 year old* Aschenbrenner. Tyler's "how to spot talent" book clearly "self-recommending"!

It’s important to remember that the models are both incredibly smart and incredibly dumb. Many people aren’t great at keeping both of these facts in their mind at the same time so they polarize in one direction or the other

we should take no pride in living in a society that has eliminated any conception of grace. no matter how profound the sins of an individual, no matter how deeply fallen they may be, there is hope of their salvation. when we drag judgement into this life we destroy that

the only useful lesson i took from getting into weight lifting was the concept of "time under load" going slow and sitting with material is often much better than repping through it quickly. hard to find an aspect of my life that hasn't improved via focusing on increasing load

A bad pattern I see people falling into at work is starting with an AI generated [document/slides/artifact] and then saying they'll refine it from there. If you start with slop you just end up with more slop, and it's very hard to fix. The first version matters a lot.

Heuristic that has worked consistently over the last decade: whenever you find yourself saying about some institution "it's hard to believe that they'd be that incompetent", you should stop and update your beliefs. It's usually worse than you think.