Assessing Human-Parity in Machine Translation on the Segment Level
Citation:
Yvette Graham, Christian Federmann, Maria Eskevich, Barry Haddow, Assessing Human-Parity in Machine Translation on the Segment Level, Findings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP - Findings), Virtual, 16/11/20, Association for Computational Linguistics, 2020, 4199 - 4207Download Item:
Abstract:
Recent machine translation shared tasks have shown top-performing systems to tie or in some cases even outperform human translation. Such conclusions about system and human performance are, however, based on estimates aggregated from scores collected over large test sets of translations and unfortunately leave some remaining questions unanswered. For instance, simply because a system significantly outperforms the human translator on average may not necessarily mean that it has done so for every translation in the test set. Firstly, are there remaining source segments present in evaluation test sets that cause significant challenges for top-performing systems and can such challenging segments go unnoticed due to the opacity of current human evaluation procedures? To provide insight into these issues we carefully inspect the outputs of top-performing systems in the most recent WMT-19 news translation shared task for all language pairs in which a system either tied or outperformed human translation. Our analysis provides a new method of identifying the remaining segments for which either machine or human perform poorly. For example, in our close inspection of WMT-19 English to German and German to English we discover the segments that disjointly proved a challenge for human and machine. For English to Russian, there were no segments included in our sample of translations that caused a significant challenge for the human translator, while we again identify the set of segments that caused issues for the top-performing system.
Sponsor
Grant Number
SFI stipend
13/RC/2106
Author's Homepage:
http://people.tcd.ie/ygrahamDescription:
PUBLISHEDVirtual
Author: Graham, Yvette
Sponsor:
SFI stipendOther Titles:
Findings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP - Findings)Publisher:
Association for Computational LinguisticsType of material:
Conference PaperCollections
Availability:
Full text availableSubject (TCD):
International Integration , ARTIFICIAL INTELLIGENCE , Machine Translation , Natural Language ProcessingDOI:
http://dx.doi.org/10.18653/v1/2020.findings-emnlp.375Metadata
Show full item recordLicences: