An Instrument to Evaluate Governance Proposals: AI Policy Analysis at Scale

This paper introduces a policy analysis framework designed to support systematic, transparent assessment of artificial intelligence (AI) governance proposals in a rapidly evolving and contested regulatory landscape. AI policy debates often collapse into binary positions that obscure underlying tradeoffs and normative assumptions. The framework structures policy analysis around multiple policy attributes, allowing users to surface priorities and tensions without prescribing outcomes. The research adopts a mixed-methods approach that integrates qualitative insights from subject matter experts with computational text analysis to inform the design of policy attribute indices and rubrics. The resulting approach quantifies the relative emphasis of different policy objectives and presents them through comparative visualizations that support interpretability and cross-policy comparison. The paper also examines the use of commercial large language models for rubric-based policy analysis, benchmarking their outputs against a domain-trained rubric-calibrated model with explicitly defined analytical assumptions. Rather than assessing policy effectiveness or desirability, the framework focuses on relevance and alignment across attributes. By making analytical assumptions explicit, including attribute selection, rubric construction, and weighting schemes, the framework enables users to evaluate whether its embedded priorities align with the users’ own normative commitments. The approach is jurisdiction-agnostic and intended to support policymakers, analysts, and researchers navigating complex AI governance environments.

The framework makes three contributions: (1) it operationalizes multidimensional policy assessment through empirically grounded rubrics that surface tradeoffs rather than resolving them; (2) it develops a transparent hybrid methodology combining feedback from subject-matter experts with computational validation; and (3) it demonstrates how domain-trained rubric-calibrated models can be used as a benchmark for comparing different general-purpose large language models. These methodological advances enable more systematic, reproducible policy comparison while maintaining transparency about embedded normative choices.