Affiliations: [a]
Department of Informatics, Constantine the Philosopher University in Nitra, Tr. A. Hlinku, Nitra, Slovakia
| [b]
Department of Translation Studies, Constantine the Philosopher University in Nitra, Stefanikova, Nitra, Slovakia
| [c]
Institute of System Engineering and Informatics, University of Pardubice, Studentska, Pardubice, Czech Republic
Correspondence:
[*]
Corresponding author. Lubomir Benko, Institute of System Engineering and Informatics, University of Pardubice, Studentska 95, 532 10 Pardubice, Czech Republic. E-mail: [email protected].
Abstract: The study describes an experiment with different estimations of reliability. Reliability reflects the technical quality of the measurement procedure such as an automatic evaluation of Machine Translation (MT). Reliability is an indicator of accuracy, the reliability of measuring, in our case, measuring the accuracy and error rate of MT output based on automatic metrics (precision, recall, f-measure, Bleu-n, WER, PER, and CDER). The experiment showed metrics (Bleu-4 and WER) that reduce the overall reliability of the automatic evaluation of accuracy and error rate using entropy. Based on the results we can say, that the use of entropy for the estimation of reliability brings more accurate results than conventional estimations of reliability (Cronbach’s alpha and correlation). MT evaluation, based on n-grams or edit distance, using entropy could offer a new view on lexicon-based metrics in comparison to commonly used ones.
