研究提出锚点集与校准函数，将 LLM 对话评判分数对齐到统一量表
Calibrating LLM Judges for Human and AI Conversations
依据 arXiv 预印本摘要（未经同行评审），作者在 CANDOR 上评估当前先进 LLM 作为对话成功度的逐点与成对评判者：逐点评分与人类评分呈中等相关，成对比较受长转写文本和位置偏差影响，导致不同模型的评判分数不可比。
arXiv:2609.29431v2 Announce Type: replace Abstract: Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias. Since this leaves judge scores incomparable across models, we propose a small anchor set and a calibration function that calibrates any judge onto a shared, interpretable scale. We further release the Voice Arena Goal Dataset (VA), 200 task-oriented human-AI and human-agent conversations with pairwise annotations, revealing a substantial gap between current judges and human-level discrimination. Using VA, we test whether CANDOR-fitted calibration transfers to human-AI conversations, finding it brings judges onto a shared scale despite never observing VA during fitting.
中文翻译还没有生成，先显示原文。
来源：arXiv · Human-Computer Interaction · arxiv.org