StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator Agents

arXiv:2610.10942v1 Announce Type: new Abstract: Reinforcement learning environments are now a primary lever for improving large language model (LLM) capabilities in post-training, yet most agentic benchmarks remain static: the world moves only when the agent acts, the reward is a terminal verdict, and the pass bar is set arbitrarily. We introduce StoreBench, a live-commerce environment in which an agent runs a mid-size online apparel store on a production-grade commerce backend, testing long-horizon planning and economic judgment under uncertainty. Customers order around the clock, suppliers reprice and fail, and market shocks arrive with partial or no warning. The agent acts through the same 29 merchant tools a human operator would use, under a windowed operation budget that makes simulated time a function of actions taken, so model latency cannot influence simulated time. Pass thresholds are calibrated against scripted anchor policies, the reward is hardened against a catalogue of reward hacks, and every episode replays identically given a sequence of actions. We evaluate seven frontier LLMs on 11 scenarios of 30 to 45 days and a full simulated year, over three world seeds at matched reasoning effort. No model matches the scripted smart-triage policy on average: the best, DeepSeek-V4-Pro, passes 49% of task-seed cells against the heuristic's 97%. Human experts working through the same tools and budgets outscore every model (mean composite 0.708 vs. 0.700). Over a full simulated year under the Claude Code harness, most models show dramatic performance improvement. In a GRP

No clear implication for human flourishing

This paper describes a benchmark for training and evaluating AI agents to autonomously manage an online store. While framed as a technical improvement, the core implication is the development of AI agents capable of replacing human operators in complex, uncertain economic roles, which could lead to job displacement. The current performance indicates human experts still outperform models, but rapid improvement is noted with further training.

The development of StoreBench represents a significant advancement in training and evaluating autonomous agents. The environment's complexity and realism suggest potential for substantial improvements in AI capabilities. However, the fact that no model matches human performance indicates ongoing challenges.

SOURCE     arXiv cs.AI (tier 1)
   ↓
DOCUMENT   607964ea-f2cc-4b25-bc7e-c42e067a5bb6
           https://arxiv.org/abs/2610.10942
   ↓
EVIDENCE   3 extracted excerpts
   ↓
MODEL RUN  4 runs, methodology 1.0.1
   ↓
SCORE      -7  (Mixed or uncertain)
   ↓
CONFIDENCE 72%