Statistics and Data Science Seminar: From LLM Evaluation to Ranking: Low-Rank Inference for Reliable LLM Leaderboards, by Wei Sun
Statistics and Data Science Seminar
October 7, 2026
4:30 PM - 5:30 PM
Location
636 SEO
Calendar
Download iCal FileWei Sun, Purdue University: From LLM Evaluation to Ranking: Low-Rank Inference for Reliable LLM Leaderboards
Modern LLM evaluation increasingly relies on sparse and heterogeneous pairwise human preferences, making reliable estimation and uncertainty quantification challenging. In this talk, I will present a low-rank statistical framework that shares information across related tasks, models, and evaluation settings while preserving task-specific heterogeneity. I will first formulate LLM evaluation as a structured tensor-completion problem under pairwise comparisons and develop semiparametrically efficient inference for model abilities, score gaps, and other smooth functionals. A key technical ingredient is a score-whitening approach that handles the anisotropic Fisher information induced by non-uniform comparisons. I will then turn from score estimation to task-specific ranking, where ranks and top-K membership are nonsmooth functions of many dependent score gaps. We develop entrywise-accurate low-rank estimation, efficient debiased inference, and multiplier-bootstrap procedures that provide simultaneous rank intervals and uncertainty-aware top-K certificates. Together, these results provide statistical foundations for moving from sparse human feedback to reliable, task-specific LLM evaluation and ranking.
Date posted
Oct 5, 2026
Date updated
Oct 5, 2026