Redwood Research Blog – Détails, épisodes et analyse

Détails du podcast

Informations techniques et générales issues du flux RSS du podcast.

Redwood Research Blog

Redwood Research

Technologie

Société & Culture

Fréquence : 1 épisode/6j. Total Éps: 123

Narrations of Redwood Research blog posts. Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.

Lire plusRéduire

Site

RSS

Apple

Classements récents

Dernières positions dans les classements Apple Podcasts et Spotify.

Apple Podcasts

🇨🇦 Canada - technology
26/07/2026
#84

Spotify

Aucun classement récent disponible

Liens partagés entre épisodes et podcasts

Liens présents dans les descriptions d'épisodes et autres podcasts les utilisant également.

https://www.youtube.com/watch?v=Vtk8YLgYU4g
1 partage
https://www.youtube.com/watch?v=iCz--zrD5CQ
1 partage

https://github.com/KellerJordan/modded-nanogpt
1 partage

Qualité et score du flux RSS

Évaluation technique de la qualité et de la structure du flux RSS.

Qualité du flux RSS

À améliorer

Score global : 38%

Historique des publications

Répartition mensuelle des publications d'épisodes au fil des années.

Year

Derniers épisodes publiés

Liste des épisodes récents, avec titres, durées et descriptions.

“SOTA alignment assessments don’t strongly update us against misalignment” by Alexa Pan

vendredi 31 juillet 2026 • Durée 01:08:10

Anthropic concluded in the April Mythos Preview alignment risk update that the model “does not possess any unknown propensities that would increase alignment risk.” The report argues that if Mythos Preview were coherently misaligned[1], it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”[3]).

While I agree with the report on the above bottom-line conclusions (substantially on priors), I think there are gaps in its argument which weaken the current assessment and might invalidate future assessments. In particular, the report often uses weak evidence to justify reliability.

The report gives fairly weak experimental evidence for Mythos Preview having insufficient capabilities to evade monitoring. The model is plausibly often eval-aware and underelicited in the relevant capability evaluations. So, it might silently sandbag if coherently misaligned, or unintentionally underperform if otherwise misaligned.
1. This limitation is important: one could argue that lack of covert capabilities for sophisticated sabotage (a subset of the capabilities I discuss here) is the single most load bearing argument in alignment risk reports.
2. Authors of the report could have made calibrated guesses about Mythos Preview's covert capabilities, especially for covert sabotage, based on other [...]

---

Outline:

(03:16) How reliability fits into the overall safety argument

(05:22) Reliability claims by AI companies

(05:59) Reliability claims by external evaluators

(06:37) Alignment assessments are less reliable than developers claim

(07:21) 1: Measuring capabilities to covertly undermine alignment assessments

(10:15) Issues with evaluation awareness

(13:39) Issues with underestimating covert capabilities

(16:45) Issues with sandbagging rule-out

(19:20) 2: Stress-testing alignment assessments with auditing games

(20:26) An auditing failure with Mythos

(22:21) AuditBench results

(24:17) 3: Conditioning on misalignment should make us think that certain covert capabilities are better than expected

(26:20) Bottom line on the strength of current alignment assessments

(29:07) Conclusion

(29:44) Appendix:

(29:47) Why I focus on motive / alignment assessments in alignment risk reports

(30:59) Auditability vs. Trustedness

(33:27) More reliability claims by developers and third party evaluators

(33:43) Mythos Alignment Risk Update

(35:01) Opus 4.6 Sabotage Risk Report

(35:46) GPT 5.5 System card

(36:49) Muse Spark system card

(37:36) Mythos Alignment Risk Update, safety arguments against sandbagging

(38:40) UK AISI evaluations for Opus 4.7

(40:01) Past auditing games by Anthropic

(42:24) Anti-auditing capability measurements

(43:51) Conditioning on coherent misalignment updates us on certain covert capabilities

The original text contained 92 footnotes which were omitted from this narration.

---

First published:
July 31st, 2026

Source:
https://blog.redwoodresearch.org/p/sota-alignment-assessments-dont-strongly

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Lire plusRéduire

“Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs” by Caleb Biddulph, Adam Kaufman

lundi 27 juillet 2026 • Durée 44:15

Subtitle: When a misaligned AI can only output tiny amounts of information, it may find sabotage very difficult.

TL;DR: We introduce the untrusted advice protocol, in which a trusted executor LLM takes every action and an untrusted advisor LLM can only send it short hints. Even with as few as 4 characters per step, this advice recovers a substantial fraction of the capability gap between the two models. Because the untrusted LLM's influence flows through such a narrow, monitorable channel, we argue that this achieves near-maximal safety in our BashArena setting. We also discuss the general concept of information bottlenecks and their benefits for interpretability, security, and cost.

In SWE-bench Verified, a strong, untrusted LLM advising a weak, trusted LLM every step can significantly improve the latter's performance, even when we limit the length of the advice. See the more detailed version of this figure later in this post.

In high-stakes AI control, we want to safely use a highly capable but untrusted model (U) that might secretly attempt a misaligned, catastrophic action. To do this, we create protocols that call U alongside a less capable, trusted model (T). Typically, T takes an auxiliary role in these protocols [...]

---

Outline:

(05:01) Experiments

(05:36) Main experiment: how does limiting advice length affect performance?

(09:55) Reducing U's bit usage

(11:01) Counting bits using LLM surprisal

(13:48) Making U select from finite options

(14:28) Why don't we red-team this protocol?

(17:08) Is studying maximally safe protocols worth the safety tax?

(19:16) Types of restrictions on U's advice

(21:20) Information bottlenecks provide other advantages

(21:52) Interpretability

(24:01) Security

“Retrying vs Resampling in AI Control” by James Lucassen, Adam Kaufman

vendredi 29 mai 2026 • Durée 19:29

We’ve just released a new paper: Retrying vs Resampling in AI Control. We revisit the resampling protocols introduced in Ctrl-Z with an up-to-date setting and much stronger models, and compare them against “retrying” protocols similar to Claude Code auto mode or Codex Auto-review.

Motivation

Roughly a year ago we released Ctrl-Z, the first paper to study control techniques for agents. A headline result of that paper was the performance of resample protocols – strategies that involve taking multiple i.i.d. samples from the model per step. But since Ctrl-Z, models have gotten much stronger, and we have built more sophisticated control settings to keep up. We wanted to answer the following questions:

How well do the results from Ctrl-Z hold up with better models and a better setting? Current high stakes control research is trying to learn by analogy about how to do control effectively in a real high stakes deployment during a real intelligence explosion. Findings about technique performance1 are going to have to generalize pretty far to be useful.

If the resample protocols from Ctrl-Z still work, what makes them work? One way we try to make our work more generalizable is by understanding the dynamics governing outcomes [...]

---

Outline:

(00:30) Motivation

(02:35) TL;DR Takeaways

(04:34) Methodology

(07:14) Differences from Ctrl-Z

(12:14) Are Retrying Protocols Exploitable?

(15:25) Cost and Latency of Resampling

(17:24) Conclusion

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
May 29th, 2026

Source:

“An overview of control measures” by Ryan Greenblatt

dimanche 6 avril 2025 • Durée 50:14

Subtitle: What methods can we use to ensure control?.

We often talk about ensuring control, which in the context of this doc refers to preventing AIs from being able to cause existential problems, even if the AIs attempt to subvert our countermeasures. To better contextualize control, I think it's useful to discuss the main countermeasures (building on my prior discussion of the main threats). I'll focus my discussion on countermeasures for reducing existential risk from misalignment, though some of these countermeasures will help with other problems such as human power grabs.

I'll overview control measures that seem important to me, including software infrastructure, security measures, and other control measures (which often involve usage of AI or human labor). In addition to the core control measures (which are focused on monitoring, auditing, and tracking AIs and on blocking and/or resampling suspicious outputs), there is a long tail of potentially [...]

---

Outline:

(01:19) Background and categorization

(06:11) Non-ML software infrastructure

(07:11) The most important software infrastructure

(11:58) Less important software infrastructure

(19:38) AI and human components

(21:30) Core measures

(27:49) Other less-core measures

(32:45) More speculative measures

(41:30) Specialized countermeasures

(46:05) Other directions and what Im not discussing

(47:15) Security improvements

---

First published:
April 6th, 2025

Source:
https://redwoodresearch.substack.com/p/an-overview-of-control-measures

“Buck on the 80,000 Hours podcast” by Buck Shlegeris

samedi 5 avril 2025 • Durée 01:19

Buck on the 80,000 Hours podcast

My podcast with Rob Wiblin from 80,000 Hours just came out. I’m really happy with how it turned out. I talked about a bunch of stuff on the podcast that I don’t think we’ve written up before.

Transcript + links + summary here. You can also get it as a podcast:

Spotify:

Apple:

Subscribe to Redwood Research blog

Launched a year ago

We research catastrophic AI risks and techniques that could be used to mitigate them.

Discussion about this post

---

Outline:

(00:09) Buck on the 80,000 Hours podcast

(00:34) Subscribe to Redwood Research blog

(00:57) Discussion about this post

---

First published:
April 5th, 2025

Source:
https://redwoodresearch.substack.com/p/buck-on-the-80000-hours-podcast

---

Narrated by TYPE III AUDIO.

“Notes on countermeasures for exploration hacking (aka sandbagging)” by Ryan Greenblatt

vendredi 4 avril 2025 • Durée 15:26

Subtitle: How can we prevent AIs from intentionally underperforming on our metrics?.

If we naively apply RL to a scheming AI, the AI may be able to systematically get low reward/performance while simultaneously not having this behavior trained out because it intentionally never explores into better behavior. As in, it intentionally puts very low probability on (some) actions which would perform very well to prevent these actions from being sampled and then reinforced. We'll refer to this as exploration hacking.

In this post, I'll discuss a variety of countermeasures for exploration hacking and what I think the basic dynamics of exploration hacking might look like.

Prior work: I discuss how exploration hacking fits into a broader picture of non-concentrated control here, Evan discusses sandbagging in context of capability evaluations here, and Teun discusses sandbagging mitigations here.

Countermeasures

Countermeasures include:

Training on high quality human examples: One [...]

---

Outline:

(01:11) Countermeasures

(03:58) How does it generalize when the model messes up?

(06:56) Empirical evidence

(09:32) Detecting exploration hacking (or sandbagging)

(11:14) Speculation on how neuralese and other architecture changes affect exploration hacking

(13:18) Future work

---

First published:
April 4th, 2025

Source:
https://redwoodresearch.substack.com/p/notes-on-countermeasures-for-exploration

“Notes on handling non-concentrated failures with AI control: high level methods and different regimes” by Ryan Greenblatt

jeudi 3 avril 2025 • Durée 32:52

Subtitle: What are the methods and issues when failures occur diffusely over many actions?.

In this post, I'll try to explain my current understanding of the high level methods for handling non-concentrated failures with control. I'll discuss the regimes produced by different methods and the failure modes of these different regimes. Non-concentrated failures are issues that arise from the AI doing a large number of problematic actions over a long period. (See here and here for more discussion of concentrated vs non-concentrated failures.)

A key aspect of non-concentrated failures is that they can in principle be handled by applying asynchronous online training with a good enough oversight signal on a small fraction of the AI's actions. That is, we could in principle solve these failures by sampling a small fraction of actions to review after they've already been executed, rating these actions using an oversight process which has [...]

---

Outline:

(06:06) What happens without online training?

(10:58) Online training

(13:10) Online training of just the policy

(19:40) Recursive oversight: debate

(20:24) Debate for binary questions

(23:37) Debate for oversight of a policy

(29:15) KL penalties and forcing identical behavior

(31:00) Is it ever a good idea for scheming models to do low stakes sabotage more sophisticated than just inserting errors (at particularly opportune times)?

---

First published:
March 29th, 2025

Source:
https://redwoodresearch.substack.com/p/notes-on-handling-non-concentrated

“Prioritizing threats for AI control” by Ryan Greenblatt

mercredi 19 mars 2025 • Durée 20:43

Subtitle: What are the main threats and how should we prioritize them?.

We often talk about ensuring control, which in the context of this doc refers to preventing AIs from being able to cause existential problems, even if the AIs attempt to subvert our countermeasures. To better contextualize control, I think it's useful to discuss the main threats. I'll focus my discussion on threats induced by misalignment which could plausibly increase existential risk.

While control is often associated with preventing security failures, I'll discuss other issues like sabotaging safety-critical work. I'll list and explain my prioritization of threats. As part of this, I'll argue that rogue internal deployments—cases where the AI is running within the company's datacenter but with control measures disabled—can be substantially worse than self-exfiltration—the AI stealing its weights and running them on an external server—as it might give the AI access to much more compute [...]

---

Outline:

(01:16) Some clusters of threats

(06:15) Concentrated vs non-concentrated failures

(08:58) My prioritization between these threats

---

First published:
March 19th, 2025

Source:
https://redwoodresearch.substack.com/p/prioritizing-threats-for-ai-control

---

Narrated by TYPE III AUDIO.

“How might we safely pass the buck to AI?” by Josh Clymer

mercredi 19 février 2025 • Durée 01:12:10

Subtitle: Developing AI employees that are safer than human ones.

My goal as an AI safety researcher is to put myself out of a job.

I don’t worry too much about how planet sized brains will shape galaxies in 100 years. That's something for AI systems to figure out.

Instead, I worry about safely replacing human researchers with AI agents, at which point human researchers are “obsolete.” The situation is not necessarily fine after human obsolescence; however, the bulk of risks that are addressable by human technical researchers (like me) will have been addressed.

This post explains how developers might safely “pass the buck” to AI.

I first clarify what I mean by “pass the buck” (section 1) and explain why I think AI safety researchers should make safely passing the buck their primary end goal – rather than focus on the loftier ambition of aligning superintelligence (section [...]

---

Outline:

(19:08) 1. Briefly responding to objections

(21:54) 2. What I mean by passing the buck to AI

(23:46) 3. Why focus on passing the buck rather than aligning superintelligence.

(28:33) 4. Three strategies for passing the buck to AI

(31:35) 5. Conditions that imply that passing the buck improves safety

(34:32) 6. The capability condition

(38:31) 7. The trust condition

(39:27) 8. Argument #1: M_1 agents are approximately aligned and will maintain their alignment until they have completed their deferred task

(49:28) 9. Argument #2: M_1 agents cannot subvert autonomous control measures while they complete the deferred task

(50:51) Analogies to dictatorships suggest that autonomous control might be viable

(52:51) Listing potential autonomous control measures

“Takeaways from sketching a control safety case” by Josh Clymer

jeudi 30 janvier 2025 • Durée 07:39

Subtitle: Insights from a long technical paper compressed into a fun little commentary.

Buck Shlegeris and I recently published a paper with UK AISI that sketches a safety case for “AI control” – measures that improve safety despite intentional subversion from AI systems.

I would summarize this work as “turning crayon drawings of safety cases into blueprints.”

It's a long and technical paper! So I’ve compiled my main takeaways in this blogpost.

The main figure from the paper summarizes how to run a control evaluation.

1. Control evaluations are way more complicated than I expected.

At a high level, control evaluations are conceptually simple. A red team constructs substitutes for subversive models. These models try to cause proxy bad outcomes (e.g. exfiltrate dummy data). Then, developers see whether the red team can succeed. If the red team succeeds, then control measures aren’t effective enough.

But there's so much [...]

---

First published:
January 30th, 2025

Source:
https://redwoodresearch.substack.com/p/takeaways-from-sketching-a-control

---

Narrated by TYPE III AUDIO.

---

Images from the article:

https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc36ceeb-1132-4969-b9ea-1117bc4f60f1_1600x1165.png

Podcasts Similaires Basées sur le Contenu

Découvrez des podcasts liées à Redwood Research Blog. Explorez des podcasts avec des thèmes, sujets, et formats similaires. Ces similarités sont calculées grâce à des données tangibles, pas d'extrapolations !