Self-discovering RL in the Era of Experience: Is Learning History an Asset or a Burden?

arXiv:2609.35897v1 Announce Type: new Abstract: The pursuit of recursive self-improvement (RSI) toward general intelligence is divided between macro-level language model scaling and the interaction-driven principles of "Era of Experience". Yet, any self-improving architecture ultimately rests upon its underlying optimization engine: if general intelligence requires learning from grounded interaction, the reinforcement learning (RL) update rule itself must be capable of cumulative adaptation. While algorithm self-discovery has produced Disco103 that surpassed PPO to achieve SOTA benchmark performance -- its internal update machinery remains an uninspected black box. We present the first causal mechanistic audit of a self-discovered RL rule, structured directly around the five pillars of the Era of Experience: extended horizon, grounded reward scales, continuing streams, within-lifetime change, and exploration depth. By surgically pinning, freezing, and transplanting recurrent states while holding meta-parameters fixed, we test when learning history acts as an asset or a burden. Three findings organize the audit: (1) Recurrent history actively expands usable reward scales, sustaining a six-decade window versus three under zero-pinning. (2) Decoupling historical content from its maintenance reveals that the penalty of mismatched history stems from perpetual clamping; allowing imported state to evolve naturally attenuates this burden. (3) Under environmental change, controlling replay retention reverses the apparent adaptation advantage over DQN, demonstrating that external dat

Technical advancements in RL may improve AI capabilities.

This is a foundational, technical machine learning paper about auditing internal dynamics of a self-discovered RL algorithm. It has no direct societal claim, though it touches on recursive self-improvement research, a topic with long-term implications. The work itself is interpretability-oriented, a mild net positive for understanding AI systems, but the connection to societal outcomes is indirect and speculative.

This academic paper reports on a technical analysis of self-discovered reinforcement learning algorithms. It investigates whether learning history helps or hinders these algorithms. The findings are highly specific to the internal dynamics of AI models and do not present any direct or implied societal consequences. It is a foundational audit, not an application.

The paper presents significant advancements in reinforcement learning that could lead to more capable and adaptable AI systems. The findings suggest that understanding and managing learning history in RL could be crucial for future developments. The work establishes a foundational audit standard for next-generation, self-evolving RL algorithms, which is a positive step for the field.

SOURCE     arXiv cs.AI (tier 1)
   ↓
DOCUMENT   82c1cbee-ea2e-41ae-8771-21e5412c48f7
           https://arxiv.org/abs/2609.35897
   ↓
EVIDENCE   3 extracted excerpts
   ↓
MODEL RUN  4 runs, methodology 1.0.1
   ↓
SCORE      -25  (Adverse)
   ↓
CONFIDENCE 74%