AutoPan is a tool that helps in the evaluation of Automatic Summaries. In DUC 2001 to 2004, the manual evaluation was based on comparison with a single human-written model and a lot of the information of evaluated summaries (both human and automatic) was marked as “related to the topic, but not directly expressed in the model summary”. The pyramid method (proposed by [Nenkova and Passoneau, 04]) addresses the problem by using multiple human summaries to create a gold-standard and by exploiting the frequency of information in the human summaries in order to assign importance to different facts. However, the method of pyramids for evaluation requires a human annotator to match fragments of text in the system summaries to the SCUs in the pyramids. We have tried to automate this part of the process.

The text in the SCU label and all its contributors is stemmed and stop words are removed, obtaining a set of stem vectors for each SCU. The system summary text is also stemmed and freed from stop words.

A search for non-overlaping windows of text which can match SCUs is carried. A window and an SCU can match if a fraction higher than a threshold (experimentally set to 0.90) of the stems in the label or some of the contributors of the SCU are present in the window, without regarding order. Each match is scored taking into account the score of the SCU as well as the number of matching stems. The solution which globally maximizes the sum of scores of all matches is found using dynamic programming techniques.

The constituent annotations automatically produced are scored using the same metrics as for manual annotations, and it is found that there is statistical evidence supporting the hypothesis that the scores obtained by automatic annotations are correlated to the ones obtained by manual ones for the same system and summary.

Takes a pyramid file and a summary and produces the peer annotation file that afterwards can be evaluated using the software provided by DUC.

Perl 5.6.0 or greater.

XML::Parser Perl Module; Expat Library.

AutoPan was developed for Maria Fuentes’ Phd thesis within the framework of the CHIL projects.

• Maria Fuentes, Edgar Gonzàlez, Daniel Ferrés, Horacio Rodríguez. QASUM-TALP at DUC 2005 Automatically Evaluated with the Pyramid based Metric AutoPan DUC 2005 Evaluation Campaign, 2005.
• Maria Fuentes, Enrique Alfonseca, Horacio Rodríguez “Support Vector Machines for Query-focused Summarization trained and evaluated on Pyramid data”. In Proceedings of the ACL 2007, Prague, Czech Republic, June 2007.