Paper 2606.05563
SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 0 stars
01 In brief
Summary
SoCRATES is a benchmark for evaluating proactive LLM mediators in realistic, multi-domain conflict scenarios.
It uses an agentic pipeline to curate scenarios from real conflicts across eight domains, probes mediators along five socio-cognitive axes (strategic posture, party composition, history length, emotional reactivity, cultural identity), and scores trajectories with a topic-localized evaluator that aligns with human experts (Pearson r=0.82).
Benchmarking eight LLM mediators, the strongest closes only about a third of the unmediated consensus gap, with performance varying sharply by axis.
Key findings: proprietary models lead, scale alone does not determine success, and effective mediation requires adapting intervention timing to socio-cognitive demands.
The evaluator more than doubles the alignment of per-turn baselines, and the benchmark reveals uneven mediator profiles across abilities.
02 From the paper
Abstract
Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants' shifting emotions, intentions, and context. Existing testbeds rely on a few expert-authored domains, vary mainly strategic posture, and score every turn against every topic, introducing off-topic noise. We introduce SoCRATES, a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It constructs scenarios from real conflicts through an agentic pipeline across eight domains, probes five socio-cognitive adaptation axes (strategic posture, party composition, history length, emotional reactivity, and cultural identity), and scores each topic only on the turns that advance it via a topic-localized evaluator. The evaluator reaches 0.82 alignment with human experts, more than doubling a per-turn baseline. Benchmarking eight frontier LLMs, we find that even the strongest mediator closes only about a third of the unmediated consensus gap under diverse and realistic testbeds, with performance varying sharply by socio-cognitive axis, highlighting that progress lies in social adaptation to diverse conditions.