SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

arXiv:2611.05512v1 Announce Type: new Abstract: AI Agents are becoming digital coworkers that can browse the web, write code, run experiments, and assist humans over extended horizons. Yet existing benchmarks fall short of real knowledge work: many rely on synthetic data or isolated tasks, missing the complex, multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions. Even simple workplace questions may require locating relevant channels and direct messages, recovering context from long discussions, inspecting repository state, reconciling stale or conflicting evidence, and synthesizing across sources. We introduce SlackBench, a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state. Grounded in privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs. It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations. These tasks are not reducible to static corpus question answering: agents must determine which conversations matter, traverse threads, inspect repository state, and synthesize evidence across communication and code artifacts. Across 33 frontier

Improves AI agents' ability to collaborate with humans, but also increases reliance on AI in workplaces.

SlackBench benchmarks AI agents' abilities in complex collaborative coding projects, reflecting realistic workplace demands. This could improve AI tools for developers, enhancing productivity. However, current agent performance is modest, indicating limitations and risks such as overreliance on imperfect AI coworkers. The benchmark supports safer, more effective progress but with moderate societal impact.

This is a benchmarking paper measuring agent performance on collaborative workflows, not a claim about societal deployment or impact. It reveals current AI limitations (most agents below 66%) more than capability advances, making implications modest either way.

The development of benchmarks for AI agents operating in collaborative, code-centric environments suggests a future where AI will perform complex knowledge work. While this specific paper introduces a benchmark, the underlying implication is that agents are being developed to autonomously manage and contribute to software projects, potentially displacing human roles in engineering and research teams. The current performance indicates room for improvement, but the benchmark's existence validates the direction of this research.

The paper describes a benchmark for AI agents in collaborative work. It shows progress in AI's ability to handle complex, real-world tasks. The best system scores around 80%, indicating significant but not complete mastery.

SOURCE     arXiv cs.AI (tier 1)
   ↓
DOCUMENT   97e64b53-a811-4376-9aaa-ae49676332ba
           https://arxiv.org/abs/2611.05512
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  5 runs, methodology 1.0.1
   ↓
SCORE      -5  (Mixed or uncertain)
   ↓
CONFIDENCE 63%