A framework on which automatic evaluation metrics over reference outputs can be robustly combined for the purpose of machine translation, summarization or language generation system development. This tool allows to combine metrics operating at different linguistic levels (lexical, shallow syntactic, syntactic, and shallow semantic). The framework provides measures to quantify the power of metric combinations and the appropriateness of the testbed.

Perl

It requires multiple references for each evaluation test case and the evaluation results from each individual metric and each reference.

Metric scores can be combined into a single measure of quality, without depending on scales properties or weighting criteria.

This work has been developed in the context of a PhD. thesis.

• Enrique Amigó, Julio Gonzalo, Anselmo Peñas and Felisa Verdejo, QARLA: a framework for the evaluation of text summarization systems. Proceedings of the 43rd Meeting of the Association for Computational Linguistics (ACL 2005). 2005.
• Enrique Amigó, Julio Gonzalo, Anselmo Peñas and Felisa Verdejo. Evaluating DUC 2004 Tasks with the QARLA Framework. Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization. 2005.
• Enrique Amigó, Julio Gonzalo, Anselmo Peñas and Felisa Verdejo, Evaluación de resúmenes automáticos mediante QARLA. Procesamiento del Lenguaje Natural, Nº 35 pp. 59-66, 2005.
• Jesús Giménez and Enrique Amigó. IQMT: A Framework for Automatic Machine Translation Evaluation. Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC’06). Genoa, Italy, 22-28 May. 2006. Departament Research Technical Report (LSI-07-29-R).