On the practice of error analysis for machine translation evaluation
2012 (English)In: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12), European Language Resources Association , 2012, 1786-1790 p.Conference paper (Refereed)
Error analysis is a means to assess machine translation output in qualitative terms, which can be used as a basis for the generation of error profiles for different systems. As for other subjective approaches to evaluation it runs the risk of low inter-annotator agreement, but very often in papers applying error analysis to MT, this aspect is not even discussed. In this paper, we report results from a comparative evaluation of two systems where agreement initially was low, and discuss the different ways we used to improve it. We compared the effects of using more or less fine-grained taxonomies, and the possibility to restrict analysis to short sentences only. We report results on inter-annotator agreement before and after measures were taken, on error categories that are most likely to be confused, and on the possibility to establish error profiles also in the absence of a high inter-annotator agreement.
Place, publisher, year, edition, pages
European Language Resources Association , 2012. 1786-1790 p.
Mahine translation, statistical machine translation, error analysis, inter-annotator agreement
National CategoryLanguage Technology (Computational Linguistics) General Language Studies and Linguistics Computer Science
IdentifiersURN: urn:nbn:se:liu:diva-80353ISI: 000323927701143ISBN: 978-2-9517408-7-7OAI: oai:DiVA.org:liu-80353DiVA: diva2:546565
The Eight International Conference on Language Resources and Evaluation (LREC'12)