Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information

This paper develops new aggregation methods for combining outputs from multiple large language models (LLMs). The authors introduce two algorithms—Optimal Weight (OW) and Inverse Surprising Popularity (ISP)—that leverage both first-order and second-order information to improve upon standard majority voting approaches. Theoretical analysis shows that these methods mitigate limitations of majority voting under mild assumptions, while empirical evaluations on synthetic datasets, LLM benchmarks such as UltraFeedback and MMLU, and real-world healthcare applications demonstrate consistently improved collective decision-making performance. The work contributes a training-free framework for robust multi-agent LLM aggregation.