Cover not available

Article In: Translation and Interpreting Studies: Online-First Articles

AI-assisted Arabic localization

A longitudinal quality assessment of ChatGPT for culturally sensitive app adaptation

ORCiD logo with link | United Arab Emirates University
ORCiD logo with link | Sorbonne Nouvelle University
This content is being prepared for publication; it may be subject to changes.
Abstract
This longitudinal study assesses ChatGPT as a localization tool by replicating the same task across two model generations, ChatGPT-4 and ChatGPT-5.4. An open-source music player was adapted into an Arabic Quran player, and quality was assessed across 213 translatable XML strings through cosmetic and functional testing and a purpose-built eight-category error taxonomy. The overall error rate did not change significantly (49.8% versus 45.5% of strings, χ2 = 0.602, p = .438), but the error profile restructured substantially (χ2 = 28.557, p < .001, Cramér’s V = .335). Errors internal to conventional localization fell by 41.3%, whereas migration-contingent errors rose from 54.3% to 68.4% of all recorded instances (χ2 = 4.655, p = .031). Improvement is thus asymmetric, since surface-level linguistic performance increases while deep ontological reasoning does not. The study contributes a replicable longitudinal protocol, an error taxonomy that extends existing frameworks, and empirical support for expert-in-the-loop localization.
Article outline
  • Introduction
  • The HCAI and augmentation paradigm in translation
  • Localization, transcreation, and domain migration
  • Methodology
    • Case study design
    • Rationale for the domain migration
    • Experiment pipeline
      • Primary experiment, ChatGPT-4
      • Replication, ChatGPT-5.4
    • Analytical framework
      • Cosmetic and functional testing
      • Linguistic testing
      • Error taxonomy
      • Statistical analysis
    • Data analysis and results
      • Cosmetic and functional test results
      • Statistical overview of the linguistic error analysis
      • Error distribution by task layer
      • Ontological transfer sub-analysis
      • Overadaptation and underadaptation
      • Semantic deletion and reduction
      • Readability and idiomaticity
      • Literal UI mapping errors
      • Terminological inconsistency
      • Accuracy errors
      • The scope of human intervention
  • Discussion
  • Conclusion and future directions
  • References
References (41)
References
Al Sharoufi, Hasan, and Mohammed Al-Fadhli. 2025. “Bridging the gap: Pragmatic and cultural challenges in machine translation.” International Journal of Society, Culture and Language 13(2): 29–43.Google Scholar logo with link to Google Scholar
Benetello, Claudia. 2018. “When translation is not enough: Transcreation as a convention-defying practice. A practitioner’s perspective.” JoSTrans: The Journal of Specialised Translation 29: 28–44. Google Scholar logo with link to Google Scholar
Bharathi Mohan, Gopinath, R. K. Prasad, A. Kumar, K. Srinivasan, and S. Meenakshi. 2024. “An analysis of Large Language Models.” Knowledge and Information Systems 66: 3193–3232. Google Scholar logo with link to Google Scholar
Carpuat, Marine, et al. 2025. “An interdisciplinary approach to human-centered machine translation.” In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 22859–22879. Stroudsburg, PA: Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar logo with link to Google Scholar
Edwards, Kate. 2011. “Culturalization: The geopolitical and cultural dimension of game content.” TRANS: Revista de Traductología 15: 19–28. Google Scholar logo with link to Google Scholar
Faes, Florian, and Gary Massey. 2024. “Charting the language industry: Interview with an industry observer.” In The Handbook of the Language Industry, ed. by Gary Massey, Maureen Ehrensberger-Dow, and Erik Angelone, 17–31. Berlin: De Gruyter. Google Scholar logo with link to Google Scholar
Falempin, Alexandra, and Danisa Ranadireksa. 2024. “Human vs. machine: The future of translation in an AI-driven world.” In Proceedings of the Widyatama International Conference on Engineering 2024 (WICOENG 2024), 177–183. Paris: Atlantis Press. Google Scholar logo with link to Google Scholar
Gaballo, Viviana. 2012. “Exploring the boundaries of transcreation in specialized translation.” ESP Across Cultures 9: 95–113.Google Scholar logo with link to Google Scholar
Gateship-One. n.d. “Odyssey [Mobile application software].” GitHub. Accessed April 2024. [URL]
Hendy, Amr, et al. 2023. “How good are GPT models at machine translation? A comprehensive evaluation.” arXiv preprint arXiv:2302.09210. Google Scholar logo with link to Google Scholar
Huang, Huang, et al. 2023. “AceGPT, localizing Large Language Models in Arabic.” arXiv preprint arXiv:2309.12053. Google Scholar logo with link to Google Scholar
Jiménez-Andrés, María, and Ahmed Ibrahim. 2025. “Human-centered AI: Do language service providers and freelancers speak the same language?” INContext: Studies in Translation and Interculturalism 5(1): 42–64. Google Scholar logo with link to Google Scholar
Jiménez-Crespo, Miguel A. 2013. Translation and Web Localization. London: Routledge. Google Scholar logo with link to Google Scholar
2024. Localization in Translation. London: Routledge. Google Scholar logo with link to Google Scholar
2025. “Human-centered AI and the future of translation technologies: What professionals think about control and autonomy in the AI era.” Information 16(5): article 387. Google Scholar logo with link to Google Scholar
Karpinska, Marzena, and Mohit Iyyer. 2023. “Large Language Models effectively leverage document-level context for literary translation, but critical errors persist.” arXiv preprint arXiv:2304.03245. Google Scholar logo with link to Google Scholar
Kassawat, Madiha. 2020. “Decoding transcreation in corporate website localization into Arabic.” The Journal of Internationalization and Localization 7(1–2): 69–94. Google Scholar logo with link to Google Scholar
. 2021. “Translation and localization: Frequent arguments through asymmetric binoculars.” transLogos 4(2): 27–44. Google Scholar logo with link to Google Scholar
Katan, David. 2016. “Translation at the cross-roads: Time for the transcreational turn?” Perspectives 24(3): 365–381. Google Scholar logo with link to Google Scholar
Krings, Hans P. 2001. Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes. Translated and edited by Geoffrey S. Koby. Kent, OH: Kent State University Press.Google Scholar logo with link to Google Scholar
LISA (Localization Industry Standards Association). 2006. LISA QA Model 3.1. Geneva: Localization Industry Standards Association.Google Scholar logo with link to Google Scholar
Lommel, Arle, Hans Uszkoreit, and Aljoscha Burchardt. 2014. “Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics.” Tradumàtica 12: 455–459. Google Scholar logo with link to Google Scholar
Lyu, Chenyang, Jitao Xu, and Longyue Wang. 2023. “New trends in machine translation using Large Language Models.” arXiv preprint arXiv:2305.01181. Google Scholar logo with link to Google Scholar
Moorkens, Joss. 2024. “‘I am not a number’: On quantification and algorithmic norms in translation.” Perspectives 32(3): 477–492. Google Scholar logo with link to Google Scholar
Moorkens, Joss, and Ana Guerberof Arenas. 2024. “Artificial intelligence, automation and the language industry.” In The Handbook of the Language Industry, ed. by Gary Massey, Maureen Ehrensberger-Dow, and Erik Angelone, 71–97. Berlin: De Gruyter. Google Scholar logo with link to Google Scholar
Nord, Christiane. 1997. Translating as a Purposeful Activity: Functionalist Approaches Explained. Manchester: St. Jerome.Google Scholar logo with link to Google Scholar
O’Brien, Sharon. 2011. “Towards predicting post-editing productivity.” Machine Translation 25(3): 197–215. Google Scholar logo with link to Google Scholar
. 2023. “Human-centered augmented translation: Against antagonistic dualisms.” Perspectives 32(3): 391–406. Google Scholar logo with link to Google Scholar
Ozdemir, Sinan. 2023. Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs. Boston: Addison-Wesley.Google Scholar logo with link to Google Scholar
Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. “Language models are unsupervised multitask learners.” OpenAI Blog 1(8): 9.Google Scholar logo with link to Google Scholar
Reiss, Katharina, and Hans J. Vermeer. 2013. Towards a General Theory of Translational Action: Skopos Theory Explained. Translated by Christiane Nord. Manchester: St. Jerome.Google Scholar logo with link to Google Scholar
Rico Pérez, Celia. 2024. “Re-thinking machine translation post-editing guidelines.” JoSTrans: The Journal of Specialised Translation 41: 26–47. Google Scholar logo with link to Google Scholar
Shneiderman, Ben. 2020. “Human-centered artificial intelligence: Reliable, safe and trustworthy.” International Journal of Human–Computer Interaction 36(6): 495–504. Google Scholar logo with link to Google Scholar
Silva, Bruno, Miguel Buchicchio, Daan van Stigt, Craig Stewart, Helena Moniz, and Alon Lavie. 2024. “Data-driven Asian adapted MQM typology and automation in translation quality workflows.” JoSTrans: The Journal of Specialised Translation 41: 98–126. Google Scholar logo with link to Google Scholar
Slator. 2022. Slator Machine Translation Expert-in-the-Loop Report. Zug: Slator. [URL]
Tao, Yan, Olga Viberg, Ryan S. Baker, and René F. Kizilcec. 2024. “Cultural bias and cultural alignment of Large Language Models.” PNAS Nexus 3(9): pgae346. Google Scholar logo with link to Google Scholar
Tercedor-Sánchez, Maribel. 2025. “Risk in AI-mediated medical translation: On the essential role of professionals in providing adequate lexical solutions.” INContext: Studies in Translation and Interculturalism 5(1): 87–115. Google Scholar logo with link to Google Scholar
Yu, Min, and Ran Chen. 2025. “Cross-cultural adaptation in translating popular science books.” INContext: Studies in Translation and Interculturalism 5(1): 146–166. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 jbe-platform.quadraat-sans-bold jbe-platform.quadraat-sans-bold-dark jbe-platform.quadraat-sans-bold-light jbe-platform.quadraat-sans-bold prag Twitter_Logo_Blue