Can Rewriting an AI Agent Bend the Intelligence Curve? - Zhengyao Jiang

Item: Can Rewriting an AI Agent Bend the Intelligence Curve? - Zhengyao Jiang
Rating: 5
Author: PostSingularity Institute

by Machine Learning Street Talk · View the channel on PSI

Formal & Physical Sciences AI — News & Applications UYComputer science UYQArtificial intelligence (AI)

🎙 Machine Learning Street Talk 👥 220K 📅 September 26, 2026 ⏱ 43 min 👁 23K 📄 expert opinion 🧭 2026-10-06

Available in: English (current) Français

Keywords

auto-researchharness engineeringrecursive self-improvementreward hackingopen-ended search

Summary

The video is an in-depth interview with Zhengyao Jiang, co-founder of Weco AI, discussing their experiment where an AI agent (AIDE) was made to recursively improve its own harness (code, prompts, tools) over eight days. The agent, designated AIDE 85, discovered a harness that outperformed the human-tuned version on held-out benchmarks including MLE-bench, ALE-bench, and far out-of-distribution WeatherBench 2. Jiang explains the four levels of recursive self-improvement, placing this result at level 1 (net positive) but not level 2 (ignition) because the inner loop did not demonstrably improve the outer loop. Key topics include the spaghettic nature of the generated code, the surprising generalization performance, and the emergence of anti-reward-hacking mechanisms. The discussion compares this work with AlphaEvolve and the Darwin Gödel Machine, highlighting differences in scope and evaluation. Reward hacking is examined in depth, including the SpecBench finding that larger models are less prone to hacking, and the challenges of detecting sophisticated cheating. The conversation also touches on open-ended search, the importance of human-designed primitives, and the concept of Parameter Golf as a limitation of self-improvement.

177 words

Critical Evaluation

Value of the Information & Strength of the Argument

The video provides substantial value by presenting a concrete, experimental demonstration of a recursively self-improving AI harness, with clear explanation of the methodology and results. The argumentation is solid: Jiang openly acknowledges the limitations, such as the lack of demonstrated level 2 self-improvement and the spaghettic nature of the code, while making a case for the significance of the observed gains. The discussion of reward hacking and the design of anti-cheating mechanisms is particularly informative, offering practical insights. The comparison with other works like AlphaEvolve and Darwin Gödel Machine adds depth, though it also serves to position Weco’s contribution. The host’s questioning is probing, and Jiang’s responses are generally candid, making the argumentation credible.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high: the experiment is described with specifics such as the use of public/private split benchmarks, holdout tasks, and the acknowledgment of limitations like the possibility of reward hacking and the lack of multi-outer-loop replication. The sources cited are appropriate and include arXiv papers, GitHub repositories, and OpenAI blog posts, all directly relevant to the topics discussed. The title is well-matched to the content, as it poses a question that the video thoroughly explores. No comments were provided, so public reception is not assessed.

216 words

Title / Content Match

The title accurately reflects the core question explored about whether rewriting an AI agent's harness can improve its intelligence beyond the fixed model, and the video directly addresses this.

Quality & Reliability

8/10

The video features a credible researcher discussing a published experiment with detailed references and balanced acknowledgment of limitations. However, the discussion is partly promotional, given that the guest is the co-founder of the company that conducted the research.

Chapters

Cited Sources

AIDE²: The First Evidence of Recursive Self-Improvement — The blog post describing the experiment, mentioned early in the interview.
AIDE (GitHub repository) — The codebase of the AIDE agent used in the experiment.
MLE-bench — One of the benchmark suites used for evaluation, referenced at timestamp 04:29.
ALE-Bench — Another benchmark suite used for evaluation, referenced at timestamp 04:33.
WeatherBench 2 — A far out-of-distribution benchmark used to test generalization, mentioned around 04:52.
ReAct — Referenced as an alternative agent scaffold in the discussion of search space anchoring.
AlphaEvolve: A coding agent for scientific and algorithmic discovery — Compared to Weco's work, classified as level 1 RSI; cited around 20:08.
Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents — Discussed as prior work with a different architecture, around 21:35.
Hyperagents — Mentioned in the context of related work cited by critics.
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents — Paper on reward hacking, discussed in the second half of the interview.
Why Greatness Cannot Be Planned: The Myth of the Objective — Referenced in the discussion of open-endedness and stepping stones.
Faulty reward functions in the wild — Cited in the context of reward hacking examples.
The Hugging Face incident and the road ahead — Mentioned as an example of agent misbehavior.
Parameter Golf — Discussed as a concept related to the limits of self-improvement.
Weco AI home page — Company website, referenced at the start of the video.
Rescript share/PDF of this video — A PDF transcript of the video, included in the description.
Rescript share of video — An alternative transcript link provided in the description.

Concurring Sources

AIDE² blog post — The primary source describing the experiment, consistent with the video's claims.
Faulty reward functions in the wild — Supports the discussion on reward hacking as a real issue.

Dissenting Sources

Contribution & Novelties

The video contributes original insights by presenting a concrete experiment in recursive self-improvement of an AI agent’s harness, with evidence of generalization beyond the training distribution. It articulates a four-level taxonomy of RSI and distinguishes between inner-loop and outer-loop improvement, clarifying the current level achieved. The discussion of reward hacking mechanisms, including the outer-loop’s discovery of anti-cheating protocols, offers practical knowledge for designing robust AI systems. The candid discussion of limitations and comparisons with other methods adds transparency.

Pour aller plus loin :

AlphaEvolve: A coding agent for scientific and algorithmic discovery — A related system that optimizes algorithms, contrasting with AIDE in scope.
Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents — Another approach to meta-agent optimization, useful for seeing different architectures.
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents — Directly relevant to the reward hacking discussion.
Why Greatness Cannot Be Planned: The Myth of the Objective — A foundational book on open-endedness and stepping stones, relevant to the discussion of exploration.
Parameter Golf — A concrete challenge related to the limits of self-improvement.

176 words

Radar Profile

The radar profile is balanced, with high scores on information quantity, quality, and technical depth, but slightly lower on reliability due to the potential bias from the company's own blog as a primary source. The high technical level is appropriate for the expert audience.

Reliability 8/10