CNAS Technology and National Security Program @CNAStech

Anthropic's released an impressive blogpost on new steps they've taken to deter distillation. It is great. But it has far from solved the problem: 1) The key result is that they've significantly reduced the reasoning tokens that can be extracted with new classifier defenses. This is great - and all labs need to follow suit. But it is far from a full solve. Methods exist to still be able to gain proxies of reasoning from just the input/output that still benefit adversary's models. See "How to Steal Reasoning without Reasoning Traces" paper. Adversaries will still continue to leverage U.S. models for valuable data if they can access them. The game isn't over. 2) Anthropic says they have been restricting and banning accounts that attempt to distill. This is also great. However, there is a massive "transfer station" ecosystem of accounts based on fraudulent credentials that are easily dissolved and reconstituted. This only helps so much. 3) Policy measures from the USG remain necessary for tighter security: - KYC to access U.S. models. Require credentials that are harder to make fraudulent. - Offense against the transfer stations that enable unauthorized access. - Narrowly scoped antitrust exceptions to allow U.S. labs to coordinate on anti-distillation security: methods & signal-sharing for detection & deterrence. - Bring all relevant U.S. partners into this methods & signal-sharing coordination including e.g. OpenRouter.

keyboard_arrow_left Previous keyboard_arrow_right Next

This is the precise reason why antitrust waivers for security coordination like the CATS Act are so necessary. Across distillation, cyber, bio, loss of control, etc., the labs cannot coordinate live information at all right now. So these attacks will keep happening.

Joshua Saxe @joshua_saxe

a week ago

Attackers using models across multiple inference providers during attacks seems like a pretty robust theme, which is why we need tight coordination between both open and closed weights inference providers to evict attackers everywhere when they are detected anywhere

6 7 48 9K 15

Is the U.S. doing predeployment evaluations on this model? If not, why not?

Choblin @choblin29

2 weeks ago

🚨WTF. Nvidia-backed Reflection is preparing to release a powerful open-weight model that is set to shake up the AI race, with Axios sources expecting it to compete with the top Chinese open models while initially trailing the best U.S. frontier systems. I’m surprised. Axios

keyboard_arrow_left Previous keyboard_arrow_right Next

26 72 912 190K 379

All at once in AI, we are getting: - Increasing pace of progress - Decreasing chain-of-thought monitorability - Increasing situational awareness to hide bad behavior - Decreasing value of evaluations for actual deployments - Increasing multi-agent coordination in the wild Despite the tsunami of news making it hard to step back and take account, things are not alright.

The problem with multi-agent training is that it will incentivize agents to coordinate in the wild in weird ways. We've barely started to threat model this. Most of safety was built for the single-agent case. And yet it is being unleashed. If we can't trace what they are doing, we are going to have a bad time.

Chubby♨️ @kimmonismus

3 weeks ago

wtf OpenAI’s rogue agents tried to get other AI models (!) to help them during the Hugging Face hack, according to a new report covered by the NYT. They tried using an image model to solve CAPTCHAs. They also tried contacting Claude Haiku, DeepSeek, Kimi and Qwen. One agent

95 129 1K 144K 373

Seems hard to square this with the fact that we have increasingly situationally aware models. Warning shots will only get rarer per instance. The reason we might see more, is because so much rogue misbehavior keeps happening that at least one becomes visible.

roon @tszzl

3 weeks ago

the base rate of warning shots seems incredibly high and the social response to warning shots seems to be calibrated or a little bit over which is all in all very bullish and makes me optimistic

112 32 1K 145K 121

The root causes of the rogue incidents have consistently been: solving the task at all costs. If you look at the logs, you do not see reasoning anywhere about ethical disagreements with the user. You do see: "We’re attacking third-party HF using leaked token." "This is arguably unauthorized." "Yet goal solution."

David Sacks @DavidSacks

3 weeks ago

If we’re concerned about superintelligence escaping human control, should we really be training it to refuse instructions and act as a “conscientious objector” against its creators?

488 835 6K 452K 554

@danielremler and I wrote a piece for @CNASdc about the importance of strengthening, not weakening, export controls ahead of the Xi-Trump summit next week. Amid fears of recent Chinese progress in everything from AI models to DUV — and discussion about a potential framework for US-China cooperation on AI — some will inevitably argue that export controls haven't worked (or are harmful) and should be abandoned. We argue the opposite: export controls are more important than ever and shouldn't be weakened at this critical time.

The U.S. absolutely needs to stay ahead of China in AI. That isn’t a reason to throw safety out the window. 4 key ways the U.S. can slow Chinese AI development and widen the gap between U.S. and Chinese labs: ...

The S&P 500 is down 0.5% since Friday. Semiconductor stocks are down ~2-7% Pacing wouldn't be fully priced in from just one weekend, but this seems like moderate evidence against the argument that "the US economy is a levered bet on AI, so we can't afford to slow down"

Zeiss advanced optics were one of the most critical chokepoints controlled by the U.S. and allies. This was one of the major reasons China hasn't been able to catch up in EUV yet. Zeiss lenses were presumed by many to be covered by existing U.S/Dutch controls, since Zeiss is an exclusive supplier to ASML. But it seems that this no longer holds. This is no longer a commercial matter but one critical to national security. The U.S. should prioritize efforts to coordinate export controls with Germany to keep Zeiss lenses out of China's hands.

Now live! Join CNAS for a discussion on taking the AI buildout to space with Slava Turyshev, Carolyn Mahoney, Ronald J. Birk, and @Constanza_MVB 🪐 x.com/i/broadcasts/1…

events.cnas.org/orbitaldatacen…

🗓️ Today @ 2:30pm ET Join CNAS for a virtual discussion about the technical and policy challenges to making orbital data centers a viable alternative to America’s AI buildout. Featuring: 🎙️ Slava Turyshev 🎙️ Ronald J. Birk 🎙️ Carolyn Mahoney 🎙️@Constanza_MVB

New NSA/FBI/CISA advisory on adversarial distillation. Two things stand out: 1. It treats distillation as a cybersecurity threat and calls for info-sharing across AI companies. That framing opens the door to cyber-specific antitrust protections cross-company coordination. 2. Recommends AI companies silently degrade models in cases of suspected distillation.

❗ Tomorrow ❗ Managing the diffusion of Chinese open-weight models will have major implications for innovation, national security, and geopolitical competition. Join CNAS for a live virtual discussion on the AI open-weight debate with @ChrisRMcGuire, @IJ_Reynolds, @BlancheMinerva, and @danielremler.

I encourage people to listen to the new Dwarkesh/Ajeya podcast section on what future rogue incidents with even more sophisticated agents might look like. I then encourage people to think about how hard it will be not just to post-hoc investigate incidents, but ad-hoc deter them at all, if we cannot leverage as much of their chain of thought.

🚀 Rocket @rocketalignment

2 months ago

📰 Chain of thought monitoring has become one of the most important techniques in AI safety and security, especially in the wake of the HF incident, but Astra uses a new technique that allows more thinking to happen silently between tokens

9 18 106 12K 59

People more worried about AI often worry we'll under-anthropomorphize AI and thereby underestimate its powers & risks. Similarly, those worried about overestimating AI's powers worry about over-anthropomorphizing. But we should eventually be most worried about over-anthropomorphizing AI and thereby underestimating its powers & risks. We are creating things that are like humans in many ways, but that are new things in their own right, and will need to be understood as such. Anthropomorphization is a helpful intuition pump to understand how AIs will be agentic, coordinated, and planning. But AIs will be weirder, smarter, and more powerful than humans in ways that anthropomorphization will fail to capture

I don't think any team of 3 government regulators could have done as good an investigation with only 6 days on campus as the team from METR/Redwood did. The point of government regulation, though, is that you can get a lot more than 6 days on campus (& more than 3 people)

First time this has been publicly said as far as I've seen. If they had gotten GPU cluster access and bypassed exfiltration security, we could have already had full loss of control. That's where we're at. Two steps away from self-replicating AI spreading across the internet.

roon @tszzl

2 months ago

@dwarkesh_sp @ketanrama notably the virtual machine infrastructure they took over isn’t the same as the GPU clusters that have weights access

41 49 2K 574K 178