IHEval: Evaluating Language Models on Following the Instruction Hierarchy
IHEval evaluates language models on instruction hierarchy compliance, finding they frequently disregard system-level priority instructions.
Published 20254 citationsPaper ↗
67%
OverallHighly rated
?
OverallHighly ratedVote to see the scoreThe exact score shows once you've voted, so every vote is your own call. The first half of each home page shelf shows its scores.
Readers
–
Only vote on papers you've read. Sign in with GitHub to vote.
AI panel2/20reviewers recommend it
lenient 2/5
medium 0/10
strict 0/5
AI panel?Vote to see what the 20 AI reviewers said
IHEval delivers a well-timed, focused benchmark for instruction-hierarchy failures that the field urgently needs, though its mechanism-agnostic framing, missing baselines, and unexamined scaling dynamics leave its full diagnostic value still unproven.
Abstract
Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, Yichuan Li, Qingyu Yin, Bing Yin, Meng Jiang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.