Robust MT Evaluation with Sentence-level Multilingual Augmentation

Abstract

Automatic translations with critical errors may lead to misinterpretations and pose several risks for the user.As such, it is important that Machine Translation Evaluation systems are robust to these errors in order to increase the reliability and safety of the translation process.Here we introduce SMAUG, a novel Sentence-level Multilingual AUGmentation approach for generating translations with critical errors and apply this approach to create a test set to evaluate the robustness of Machine Translation metrics to these errors.We show that current State-ofthe-Art methods are improving their capability to distinguish translations with and without critical errors and to penalize the first accordingly.We also show that metrics tend to struggle with errors related to named entities and numbers and that there is a high variance in the robustness of current methods to translations with critical errors.