Can you tell the difference? A study of human vs machine-translated subtitles

Abstract

While machine translation offers the potential for improved efficiency and cost savings, there are concerns about its accuracy and reliability compared to human translation. This study aims to investigate the potential of machine translation systems by analysing viewers’ ability to distinguish between subtitles generated by ChatGPT and those created by human translators in the English to Spanish language pair. The study involved 119 Translation and Interpreting degree students who watched eight subtitled clips containing puns, cultural references, humour, and irony: five of these were generated by ChatGPT and the remaining three were created by a human translator. Results indicate that participants were unable to accurately distinguish between ChatGPT-generated and human-generated subtitles, although lower quality subtitles were associated with non-human translation. Factors such as experience with ChatGPT and exposure to subtitled content were not significant predictors of the ability to identify ChatGPT-generated subtitles. However, year of study was found to be a significant predictor, suggesting that translation expertise is a crucial factor for non-human subtitle detection. Overall, these results have important implications for the use of machine translation in subtitle generation and the quality of subtitled content.