ELEXR: Automatic Evaluation of MachineTranslation Using Lexical Relationships
2013 (English)In: Advances in Artificial Intelligence and Its Applications: 12th Mexican International Conference on Artificial Intelligence, MICAI 2013, Mexico City, Mexico, November 24-30, 2013, Proceedings, Part I / [ed] Félix Castro, Alexander Gelbukh, Miguel González, Springer Berlin/Heidelberg, 2013, 394-405 p.Conference paper (Refereed)
This paper proposes ELEXR, a novel metric to evaluate machine translation (MT). In our proposed method, we extract lexical co-occurrence relationships of a given reference translation (Ref) and its corresponding hypothesis sentence using hyperspace analogue to language space matrix. Then, for each term appearing in these two sentences, we convert the co-occurrence information into a conditional probability distribution. Finally, by comparing the conditional probability distributions of the words held in common by Ref and the candidate sentence (Cand) using Kullback-Leibler divergence, we can score the hypothesis. ELEXR can evaluate MT by using only one Ref assigned to each Cand without incorporating any semantic annotated resources like WordNet. Our experiments on eight language pairs of WMT 2011 submissions show that ELEXR outperforms baselines, TER and BLEU, on average at system-level correlation with human judgments. It achieves average Spearman’s rho correlation of about 0.78, Kendall’s tau correlation of about 0.66 and Pearson’s correlation of about 0.84, corresponding to improvements of about 0.04, 0.07 and 0.06 respectively over BLEU, the best baseline.
Place, publisher, year, edition, pages
Springer Berlin/Heidelberg, 2013. 394-405 p.
Lecture Notes in Computer Science, ISSN 0302-9743 (print), 1611-3349 (online) ; 8265
Machine Translation, Natural Language Processing, Automatic Evaluation, Evaluation Metrics, BLEU, TER, Lexical Relationships, Machine Translation Evaluation
IdentifiersURN: urn:nbn:se:liu:diva-107633DOI: 10.1007/978-3-642-45114-0_32ISBN: 978-3-642-45113-3 (print)ISBN: 978-3-642-45114-0 (online)OAI: oai:DiVA.org:liu-107633DiVA: diva2:726135
12th Mexican International Conference on Artificial Intelligence, MICAI 2013, Mexico City, Mexico, November 24-30, 2013