Seq@K: Evaluating Language Models Under Sequential Feedback

arXiv:2610.37859v1 Announce Type: new Abstract: Language-model agents differ fundamentally from prior intelligent systems in their ability to learn from natural language in context. This capability enables agents to iteratively refine their answers in response to feedback, a prevalent mode of operation in modern agentic workflows. However, the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts. We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure. Unlike multi-turn agentic benchmarks that examine long-horizon planning, Seq@K isolates and formalizes the core capability of refinement from feedback. We evaluate Seq@K on a suite of 7 benchmarks and 16 frontier models, and show that Seq@K reveals a materially different model-selection landscape: under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K. We further show that analyzing model trajectories under Seq@K helps characterize iterative refinement behavior, such as the rate at which models address issues raised in feedback, or how rigidly they hold onto their own answers. Finally, we study how the nature of feedback affects models' iterative refinement, showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems

No claim about AI's effect on society.

The paper proposes a new evaluation metric (Seq@K) which improves the measurement of iterative feedback learning in language models, a technical advancement. However, it does not make direct claims about broader societal implications, so its impact on human flourishing is unclear and likely indirect.

This is a methodological contribution to model evaluation, not a claim about societal impact. It improves benchmarking rigor for agentic systems but carries no direct implication for human flourishing beyond better tooling for AI development.

The paper introduces a new metric, Seq@K, to evaluate language models based on their ability to learn from sequential feedback. This metric aims to provide a more accurate assessment for models operating in iterative feedback loops. The implications for humanity are genuinely neutral as it is a technical contribution to AI evaluation, not a direct application or societal outcome.

SOURCE     arXiv cs.AI (tier 1)
   ↓
DOCUMENT   f371e8b4-d7ca-4038-a97d-0045cb34867c
           https://arxiv.org/abs/2610.37859
   ↓
EVIDENCE   1 extracted excerpt
   ↓
MODEL RUN  4 runs, methodology 1.0.1
   ↓
SCORE      -50  (Adverse)
   ↓
CONFIDENCE 60%