Can Viewers Identify Raw Machine Translation in Subtitles, and What It Means for Reception?

Abstract

This research compares the audience’s reception of fansubs and raw machine-translated subtitles, aiming to investigate the possibility of integrating raw machine translation into subtitling Chinese period dramas into English. The data-collecting instruments include think-aloud protocols, screen recordings, and interviews. Eighteen native English-speaking participants were invited to explore their screen-based activities triggered by each mode of subtitles, their ability to differentiate between the two subtitle modes and their emotional involvement. The results show that raw machine-translated subtitles did not trigger more screen-based activities than fansubs did during the viewing process because the viewer’s interaction with the video was not limited to subtitle-related issues. No participants could differentiate raw machine-translated subtitles from fansubs, but seven participants identified the presence of machine translation. Consequently, the seven participants had lower degrees of self-reported involvement compared to those who did not notice the presence of machine translation. Still, the awareness of machine translation did not significantly affect the participant’s willingness to continue watching.

Lay summary

With the rise in popularity of foreign TV shows and the need for quick translations, machine translation is being used more often in the subtitling industry. In this study, we look at two types of subtitles: fansubs (made by fans, not professionals) and raw machine-translated subtitles (generated by a machine translation system, like Google Translate, without human editing). We try to see whether English speakers will read, understand, and interact differently with these two types of subtitles. Our goal is to find out whether we can use machine translation alone to subtitle Chinese period dramas into English. To do this, we asked eighteen native English speakers to watch a video clip from a Chinese period drama with both types of subtitles. The participants did not know there were two types of subtitles, and they could pause, rewind, or fast forward as they liked. However, machine-translated subtitles did not make participants pause or rewind more frequently than fansubs did. Interestingly, none of the participants realised there were two types of subtitles, nor could they tell the machine-translated subtitles from the fansubs. However, seven participants did notice the machine-translated subtitles, but they could not identify the fansubs. The seven participants noticed typographical, grammatical, or lexical errors in the machine-translated subtitles and associated them with machine translation. This shows that viewers tend to doubt the accuracy of machine-translated subtitles and do not think that machine-translated subtitles are easy to read or understand. The seven participants who noticed the machine-translated subtitles reported feeling less engaged compared to those who did not notice them, but knowing about the machine translation did not affect their willingness to keep watching. Thus, this study suggests that machine translation systems can generate subtitles that are understandable and acceptable. However, comprehensive implementation of machine translation in subtitling is still not possible because viewers generally perceive machine translation negatively and are sceptical about its accuracy.