Agentic Benchmarks Multiply
A slew of new benchmarks and metrics are sharpening how we evaluate LLM agents. KaliBench catalogues 8,504 query‑command pairs across 1,642 tools, while UserProxyBench shows a 15.2‑point reward swing when swapping user simulators. TokenCast adds live token‑consumption forecasts, and KV‑streams promise 2.6–5× faster context compaction. Together they map a richer landscape for measuring agent reliability and efficiency.
- KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable RewardsFri 2 Oct · arXiv cs.CL
- UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and TrainingWed 30 Sept · arXiv cs.AI
- TokenCast: Forecasting Token Consumption During LLM Agent ExecutionTue 29 Sept · arXiv cs.LG
- KV-streams for Efficient Compaction in Agentic Reinforcement LearningTue 29 Sept · arXiv cs.LG