Limitations of MT Quality Estimation Supervised Systems: The Tails Prediction Problem
Citation:
Moreau, Erwan and Vogel, Carl, Limitations of MT Quality Estimation Supervised Systems: The Tails Prediction Problem, Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Association for Computational Linguistics, 2014, 2205-2216Download Item:
Abstract:
In this paper we address the question of the reliability of the predictions made by MT Quality Estimation (QE) systems. In particular, we show that standard supervised QE systems, usually trained to minimize MAE, make serious mistakes at predicting the quality of the sentences in the tails of the quality range. We describe the problem and propose several experiments to clarify their causes and effects. We use the WMT12 and WMT13 QE Shared Task datasets to prove that our claims hold in general and are not specific to a dataset or a system.
Sponsor
Grant Number
Science Foundation Ireland (SFI)
12/CE/I2267
Author's Homepage:
http://people.tcd.ie/vogelDescription:
PUBLISHEDDublin
Author: VOGEL, CARL
Sponsor:
Science Foundation Ireland (SFI)Other Titles:
Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical PapersPublisher:
Association for Computational LinguisticsType of material:
Conference PaperCollections
Availability:
Full text availableSubject (TCD):
Digital Humanities , Intelligent Content & Communications , Computational linguisticsMetadata
Show full item recordLicences: