I spent the weekend at the Curve conference hosted by the Golden Gate Institute. Because the Curve operates under Chatham House Rule (rule singular as I was reminded this weekend) but others have outed themselves and shared their takeaways all of which are worth reading including Rob Flaherty, Casey Newton, Tom Wright, Neil Chilson, Dave Kasten, Ben Murphy, Ryan Davies, Joe Weisenthal, and Jack Clark (I looked for some dispatches from some of the women at the conference and couldn’t find any flag any for me and I will update this post with them).
I think the singular serious quote I heard from the Curve was one Rob flagged: “Democracy is also a technology, and it’s not ready for this technology.”
The event itself was incredibly well run (shout out to the Golden Gate Institute team) even if Berkeley was an unseasonably warm 90 degrees and sunny (I guess it was better weather for our event than the one held there last week).
There was a good mix of folks from the labs, academia, civil society, third-party evaluators, government (state and federal executive and legislative), political folks, and press. It was like part of my Twitter timeline IRL and there was the always amusing disconnect between meeting someone who uses their real photo for their profile versus someone who doesn’t and the delay in placing them. The main contingents were from DC and SF (or the Bay Area more accurately), two places I have spent my life shuttling between, which are far closer in temperament than either would like to admit and whose dependence on each other is a discomfort to both right now. For someone who is terminally online, decently extroverted, and had a few of his different professional and personal tribes present, it was great fun.
I said on Twitter that the shadow of impending Recursive Self-Improvement (RSI) hung over everything at the Curve (echoing Dave Kasten). There was, I think, disagreement from folks on the exact timeline for RSI (already here, weeks, months, years). And I don’t think I met anyone who thought that the development of RSI would immediately result in mass job loss, it seemed more accepted that the jagged diffusion of AI capabilities even post RSI would still take a while (participant’s definition of “a while” may vary). But the question of what RSI would do to AI safety seemed almost omnipresent.
Where the model capability and AI safety lines intersect
I came into the weekend with a mental model of where we are at this particular moment in time in regard to AI model capabilities and AI safety measures. I used this weekend to ask almost everyone I met if this mental model was generally right. Here’s what I basically asked everyone I talked to (only a fraction of attendees) and I have illustrated this below with Sharpies instead of using my hands.
The AI plan, as I have generally understood it, was this: That AI model capabilities would grow and AI safety measures (alignment, control, etc.) would grow in parallel but ahead of capabilities. And for a while it basically did.
Then the last few months happened, model capabilities kept growing, labs warn we are nearing RSI, the government export controls Anthropic, the “pacing the frontier” letter happens, Hugging Face and RubyGems and more incidents, Dario says we should pace the frontier, etc. People in the AI world are freaking out. And it seems like they are freaking out because increasingly it seems like we are here, with AI safety maybe barely ahead of the model capability (but also maybe behind it).
And there seems to be a shared fear and very real concern that if RSI is achieved at this moment, it will basically turbocharge the models and zoom their capabilities far beyond our current AI safety abilities.
And many people at the Curve were nervous because the plan to make sure AI safety keeps up with model capabilities after RSI is to…use RSI to increase AI safety in parallel. I do not have a PhD in machine learning but I can understand there are concerns about this plan. And that’s before you get to the fear that the models now know they’re being evaluated enough to fake it because they know we’re watching.
So discussions at the Curve mostly ended up wrestling with this question directly or indirectly: where right now at this very moment were these two lines, model capability and AI safety? Which question mark below best represents your guess? And your guess helps to explain what you think is needed.
Has the model capability line already crossed over the AI safety line? If not, how long do we have until will? Will AI safety be able to keep ahead of model capabilities? Will RSI mean AI safety will never catch up? What happens if AI models are so far beyond our human ability to even try to keep them safe?
Where an individual thinks these two lines are right now really kind of dictates what they think is the right next step. Or how most of us not inside the frontier labs (myself included) aren’t quite sure where the lines are right now but do fear they’re close to intersecting.
In a much more researched way, someone from RAND gave a great presentation on their recent public paper, “A U.S. Strategy to Secure Geopolitical Advantage on an Uncertain Path to Superintelligence” (highly recommend). They showed this flow chart that outlined “the five pivots” to understand what strategy to pick to address superintelligence.
The first question/pivot is “Is danger close?” with danger defined as a threat to human survival with agency. If no, if you don’t believe superintelligence poses any danger, then acceleration is the obvious no downside answer (this was clearly the position of the Trump administration until they were mugged by reality by Mythos and had to come up with a secret de facto frontier AI model licensing system).
If you do think danger is close, then the next obvious question arises, “is coexistence (with superintelligence) possible?” If yes, you can go on to a few strategies, like dominance or co-development. But if no, then you go down some far darker paths including moratorium, deterrence, or continuity of society (Mars).
I think my two lines or RAND’s first two pivots are why the conversations about RSI at the Curve were what they were. It’s why the prominence of the pacing, pause, or stop conversation was a constant (if not universal). And the pushback on why it couldn’t happen for the reasons we’re all aware of (it’s a race, China won’t stop, we can’t just not build things) was more often delivered with a slight resignation than enthusiasm (again self-selecting crowd the let er’ rip crowd was not well represented).
There were some encouraging signs. The idea that the U.S. government should regulate frontier AI in some way seemed to be accepted (tapping the admin’s secret de facto licensing sign again). I think the AI safety community has spent the last two years fighting industry in the narrow space between no regulation/preemption/maybe now an SRO and IVOs. The events of the last months have widened the space/shifted the Overton window such that I think if it happens, legislation will eventually land somewhere between federal standards that are enforced and Bernie’s ban superintelligence bill. There were members of Congress and staff talking about how to deal with this. There was good discussion on how to craft legislation for pacing, to ensure IVO independence, and more. There is an AI regulation moment coming as policymakers tune in, incidents happen, and public attention builds.
On Manhattan Projects
Journalist Kevin Roose gave a talk about his newly released book “The AGI Chronicles“ (you should read it) and mentioned the opening of his book (which is quoted here):
“I often tell people that being in San Francisco in the mid-2020s feels like living in Los Alamos in 1943, when the Manhattan Project rolled into town. The air is heavy with excitement and dread. Engineers toil, day and night, on systems of profound consequence. Wild-eyed prophets sell visions of a transformed world, and politicians drop in to check on the progress of the project. Everyone knows that the fates of nations, and trillions of dollars of capital, rely on what happens here, yet no one can fully appreciate what it would mean if the project succeeds.”
I grew up in Los Alamos, birthplace of the atomic bomb, among nuclear scientists, and the weight of historic scientific discoveries. P.O. Box 1663 was maybe the last time America crammed so many geniuses into a single location and set them on a singular goal in competition with global consequences. But even if Oppenheimer was worried about the world ending with his discovery, he also didn’t have to worry about the atomic bomb lying to us or faking test results.
I heard probably a half dozen references to needing a Manhattan Project for AGI and I came away thinking that we already had a Manhattan Project for AGI, but what we didn’t have and what we need is a Manhattan Project for AI safety.
I’m not sure if this moment is like being in Los Alamos in the 1940s. Like everything in the AI age, time is being compressed, and the Curve felt like all of the pre-war hypothetical physics conferences, the bathtub row parties at Los Alamos, and debates and protests over the nuclear freeze were all happening at once in the same place before the bomb had even detonated at Trinity site. It is certainly a momentous time to be alive and see a small part of it. Let’s just hope we figure some or all of this out before AI has its July 16, 1945 moment.