Corpus-Based Quality Assessment of Error-Tagged Learner Translations

About the authors:

Maria A. Kunilovskaya, Cand. Sci. (Philol.), Associate Professor, Department of English Philology and Translation, Institute of Philology and Journalism, University of Tyumen; mkunilovskaya@gmail.com

Tatyana A. Ilyushchenya, Cand. Sci. (Philol.), Associate Professor, Department of English Philology and Translation, Institute of Philology and Journalism, University of Tyumen; tatyana1223@mail.ru

Marina A. Kovyazina, Cand. Sci. (Philol.), Associate Professor, Department of English Philology and Translation, Institute of Philology and Journalism, University of Tyumen; makovyazina@mail.ru

Abstract:

This paper sets out to develop a method of learner translation quality assessment, based on error statistics. It is used to rank student translations in descending order by their relative quality. It means that each translation is assessed in comparison with other translations of the same text. The resulting rating can be converted into absolute values such as grades or number of points earned if necessary. The authors use expert assessment conducted within a holistic approach as a quality standard. In this research the rank order of multiple translations to the same source is effected as a statistical approximation to the consensus rating produced for the same texts from independent ratings by three experts. This paper has a brief description of the error typology and its technical implementation, as well as of the corpus project behind this research. The agreement statistics used in the experiment show that the best results are achieved, if the error categories, discussed in the paper, are accounted for in the multiple sort of translations in the following order: 1) critical, 2) content and 3) total. The authors suggest that it makes sense to manually adjust the ranking to the number of annotated “good translational solutions”. Finally, the authors outline the current and perspective uses of the learner translator corpus and its error-annotated part in translator education.

References:

1. Garbovsky N. K. 2012. “Russkiy perevodnoy diskurs: mif ili real’nost’” [The Russian Translation Discourse: Myth or Reality]. Papers of 3rd International scientific-practical conference “Russkiy yazyk i kul’tura v zerkale perevoda”, April 25-29, pp. 130-136.
2. Knyazheva E. A. 2015. “O nekotorykh vozmozhnostyakh ispol’zovaniya metodov sistemnogo analiza v tselyakh otsenki kachestva perevoda” [Possible Uses of System Analysis Methods in Translation Quality Assessment]. The Herald of Voronezh State University. Linguistics and intercultural communication series, no. 3, pp.113-119.
3. Korpus studencheskikh perevodov (RusLTC) [The Corpus of Student Translations (RusLTC)]. www.rus-ltc.org/about_ru 
4. Kostikova O. I. 2013. “Didakticheskaya funktsiya otsenki perevoda” [The Didactic Function of Translation Quality Assessment]. Russkiy yazyk v sovremennom mire: traditsii i innovatsii v prepodavanii russkogo yazyka kak inostrannogo i v perevode. Papers of IV International scientific-practical conference, April 26 — May 1, pp. 422-429.
5. Leontyeva K. I. 2012. “Diskursivnaya ontologiya perevoda: k obosnovaniyu statusa” [Discourse Ontology of Translation: Proving Its Status]. The Herald of Tver State University. Philology Series, vol. 2, no 10, pp. 78-85.
6. Sdobnikov V. V. 2015. Otsenka kachestva perevoda (kommunikativno-funktsional’nyy podkhod) [Translation Quality Assessment (Communicative and Functional Approach)]. Moscow: FLINTA: Nauka. 
7. Bernardini S. 2008. “Corpora for Translator Education and Translation Practice”. Topics in Language Resources for Translation and Localization, no 79, pp. 39-55.
8. Diploma in Translation Handbook for Candidates. 2011. IoL Educational Trust.
9. Doyle M. 2003. “Translation Pedagogy and Assessment”. In: The ATA Chronicle, November/December 2003, pp. 21-29.
10. Kunilovskaya M. 2015. “How Far do We Agree on the Quality of Translation?” English Studies at NBU, no 1, pp. 18-31.
11. Smith K. “Don’t Visit ‘Miss Peregrine’s Home for Peculiar Children’”. http://nypost.com/2016/09/29/dont-visit-miss-peregrines-home-for-peculiar-children/
12. Stenetorp P., Pyysalo S., Topić G., Ohta T., Ananiadou S. and Tsujii J. 2012. “BRAT: A Web-Based Tool for NLP-Assisted Text Annotation”. Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp.102-107.
13. Waddington C. 2001. “Different Methods of Evaluating Student Translations: The Question of Validity”. Meta, no 46 (2), pp. 312-325.
14. Yepes G.R. 2011. “Parallel Corpora in Translator Education”. Redit: Revista electrónica de didáctica de la traducción y la interpretación, no 7, pp. 65-80.