WildIFEval: Instruction Following in the Wild

Abstract

Recent LLMs have shown remarkable success in following user instructions, yet handling instructions with multiple constraints remains a significant challenge.In this work, we introduce WILDIFEVAL -a large-scale dataset of 7K real user instructions for singleturn constrained text generation, exhibiting diverse, multi-constraint conditions.Unlike prior datasets, our collection spans a broad lexical and topical spectrum of constraints, extracted from natural user instructions.We categorize these constraints into eight high-level classes to capture their distribution and co-occurrence dynamics in real-world scenarios.Leveraging WILDIFEVAL, we conduct extensive experiments to benchmark the instruction-following capabilities of leading LLMs.WILDIFEVAL clearly differentiates between small and large models, and demonstrates that all models have room for improvement on such tasks.Our analysis reveals that as constraint count grows, models' overall success drops sharply while perconstraint success remains stable, indicating a capacity bottleneck in juggling multiple constraints, and that models struggle more with rigid form-based constraints than with softer content-based ones.We release our dataset to promote further research on instructionfollowing under complex, realistic conditions. 1 * This work was conducted during a summer internship at IBM Research.1 WILDIFEVAL is available at https://huggingface. co/datasets/gililior/wild-if-eval.The code for replication, along with model predictions and evaluation scores, is at https://github.com/gililior/wild-if-eval-code.