Evaluating AI-Assisted Translation Quality: A Comparison of ChatGPT and Google Translate in the English–Bahasa Indonesia Context
DOI:
https://doi.org/10.24256/foster-jelt.v7i4.472Keywords:
AI-Assisted Translation, ChatGPT, Google Translate, English–Indonesian TranslationAbstract
This study investigates the translation quality produced by ChatGPT and Google Translate in translating texts from English to Bahasa Indonesia. It compares the performance of both AI-assisted translation tools across five aspects of Translation Quality Assessment (TQA), consisting of Accuracy, Equivalence, Translation Skill, Text Function, and Grammar and Style. The comparison was conducted across three text types, including informative, persuasive, and instructional texts. A questionnaire of nineteen items using a Likert-scale was used to evaluate the translation results, assessed by three raters. A quantitative analysis was employed using the paired-samples t-test and Cohen’s d effect size. In addition, a qualitative examination by the raters was also conducted to identify translation issues generated by the translation tools. The findings reveal that ChatGPT obtained higher scores than Google Translate across all evaluated aspects. Further statistical analysis indicates that the differences are significant and meaningful in practical contexts. ChatGPT showed better performance in producing natural, contextual, and communicative translation results. However, it was also found that ChatGPT generates additional information not explicitly present in the source text. Meanwhile, Google Translate showed the tendency to produce literal translation that sounds awkward and unnatural translation results. It highlights that revision and post-editing efforts are required to improve the quality of translation results by both AI-assisted translation tools. This study concludes that both ChatGPT and Google Translate are more suitable as translation aids rather than complete substitutes for professional human translators.
References
Abdelhalim, S. M., Alsahil, A. A., & Alsuhaibani, Z. A. (2025). Artificial intelligence tools and literary translation?: a comparative investigation of ChatGPT and Google Translate from novice and advanced EFL student translators ’ perspectives. Cogent Arts & Humanities, 12(1), 1–20. https://doi.org/10.1080/23311983.2025.2508031
Adawiyah, A. R., & Wardana, L. A. (2023). Comparing Post-Editing Translations by Google NMT and Yandex NMT. Universitas Mataram. TEKNOSASTIK, 21(1), 23–34.
Al-jarf, R. (2025). Google Translate Then and Now: Translations From Five Languages into English and Arabic (2012–2025). Journal of Computer Science and Technology Studies, 7(12), 413–427. https://doi.org/10.32996/jcsts.2025.7.12.50
Alkaissi, H., & Mcfarlane, S. I. (2023). Artificial Hallucinations in ChatGPT?: Implications in Scientific Writing. Cureus, 15(2), 2–5. https://doi.org/10.7759/cureus.35179
Amilia, I. K., & Yuwono, D. E. (2020). A study of the translation of Google Translate: An error analysis of the translation of eliza riley ’ s return to paradise. Lingua Jurnal Ilmiah, 16(2), 1–21. https://doi.org/10.35962/lingua.v16i2.50
Amini, M., Ravindran, L., & Lee, K. (2024). Implications of Using AI in Translation Studies?: Trends , Challenges , and Future Direction. Asian Journal of Research in Education and Social Sciences, 6(1), 740–754. https://doi.org/10.55057/ajress.2024.6.1.67
DeVellis, R. F., & Thorpe, C. T. (2022). Scale Development: Theory and Applications (5th ed.). SAGE Publications.
Farahsani, Y., Puspita, I., Patria, R., & Jaya, H. (2021). Google Translate Accuracy in Translating Specialized Language From English to Bahasa Indonesia?: A Case Study on Mechanical Engineering Terminology. Advances in Social Science, Education and Humanities Research, 518 (ICoSIHESS 2020), 427–435. https://doi.org/10.2991/assehr.k.210120.156
Franzon, J. (2021). The liberal mores of pop song translation: Slicing the source text relation six ways. In J. Franzon, K. A. Greenal, S. Kvam, & A. Parianou (Eds.), Song Translation: Lyrics in Contexts (pp. 83–121).
Govender, S. C., & Jaffer, T. (2021). Designing Instructional Text What Does Instructional Text Do? In EdTech Books (pp. 347–362).
Guerreiro, N. M., Alves, D. M., Waldendorf, J., Haddow, B., Birch, A., Colombo, P., & Martins, A. (2023). Hallucinations in Large Multilingual Translation Models. Transactions of the Association for Computational Linguistics, 11, 1500–1517. https://doi.org/10.1162/tacl_a_00615
Han, L., Smeaton, A., & Jones, G. (2021). Translation quality assessment: A brief survey on manual and automatic methods. Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age, 15–33.
Herdawan, D. (2020). An Analysis on Indonesian-English Abstract Translation by Google Translate. English Education: Jurnal Tadris Bahasa Inggris, 13(2), 40–53. https://doi.org/doi.org/10.24042/ee-jtbi.v13i2.7701
House, J. (2015). Translation Quality Assessment: Past and Present (1st ed.). Routledge.
Ismailia, T. (2023). The Analysis of Machine Translation Performance on Translating Informative Text from English into Indonesian. Ebony - Journal of English Language Teaching, Linguistics, and Literature, 3(2), 129–138. https://doi.org/10.37304/ebony.v3i2.9809
Khramchenko, D. S. (2023). The power of synergy in discourse?: Exploring persuasive language in English mass media. Indonesian Journal of Applied Linguistics, 13(2), 368–379. https://doi.org/10.17509/ijal.v13i2.63068
Kiss, I. (2018). The Persuasive Discourse Function in the Language of Tourism. Argumentum, 14, 150–162. https://argumentum.unideb.hu/2017-anyagok/kissi.pdf
Lewandowska-Tomaszczyk, B., & Paw?owski, G. (2025). Testing ChatGPT on Terminology Generation, Definitions: Translation, and Ontology Creation in German, English and Polish. Research in Language (RiL), 23(1), 320–340. https://doi.org/10.18778/1731-7533.23.20
Lomaka, A. O. (2017). Literal Approach to Translation: A Classification and Literature Review. XLinguae, 10(1), 234–244. https://doi.org/10.18355/XL.2017.10.04.19
Lyu, C., Xu, J., & Wang, L. (2023). New Trends in Machine Translation using Large Language Models?: Case Examples with ChatGPT. https://doi.org/10.48550/arXiv.2305.01181
Mohamed, Y. A., Khanan, A., Bashir, M., & Elsadig, M. A. (2024). The Impact of Artificial Intelligence on Language Translation?: A Review. IEEE Access, 12(February), 25553–25579. https://doi.org/10.1109/ACCESS.2024.3366802
Napitupulu, S. (2017). Analyzing Indonesian-English abstracts translation in view of translation errors by Google Translate. International Journal of English Language and Linguistics Research Vol.5, 5(2), 15–23.
Nickl, M. (2018). Instructional Texts: Learn How to Get Things Done. In J. Humbley, G. Budin, & C. Lauran (Eds.), Languages for Special Purposes: An International Handbook (pp. 322–342).
Rane, N. L., Tawde, A., Choudhary, S. P., & Rane, J. (2023). Contribution and performance of ChatGPT and other Large Language Models ( LLM ) for scientific and research advancements?: a double-edged sword. International Research Journal of Modernization in Engineering Technology and Science, 5(10), 875–899. https://doi.org/10.56726/IRJMETS45312
Ruoqi, S., & Gochuico, M. T. (2023). Translators and Their Use of ChatGPT. Academia Lasalliana Jurnal of Education and Humanities, 5(1), 49–59. https://doi.org/10.55902/PLRU9197 Translators
Sebo, P., & de Lucia, S. (2024). Performance of machine translators in translating French medical research abstracts to English?: A comparative study of DeepL, Google Translate, and CUBBITT. PLoS ONE, 19(2), 1–13. https://doi.org/10.1371/journal.pone.0297183
Shahmerdanova, R. (2025). Artificial Intelligence in Translation?: Challenges and Opportunities. Acta Globalis Humanitatis et Linguarum, 2(1), 62–70. https://doi.org/10.69760/aghel.02500108
Shormani, M. Q., & Al-sohbani, Y. A. (2025). Artificial intelligence contribution to translation industry?: looking back and forward. Discover Artificial Intelligence, 5, 1–29. https://doi.org/10.1007/s44163-025-00487-3
Siu, S. C. (2024). Revolutionising Translation with AI: Unravelling Neural Machine Translation and Generative Pre-trained Large Language Models. In Y. Peng, H. Huang, & D. Li (Eds.), New Advances in Translation Technology (pp. 29–54). Springer. https://doi.org/10.1007/978-981-97-2958-6_3
Sofyan, R., & Tarigan, B. (2019). Developing a Holistic Model of Translation Quality Assessment. Proceedings of the Eleventh Conference on Applied Linguistics (CONAPLIN 2018), 254(Adv. Soc. Sci. Educ. Humanit. Res.), 266–271. https://doi.org/10.2991/conaplin-18.2019.267
Stasimioti, M., Sosoni, V., Mouratidis, D., & Kermanidis, K. (2020). Machine Translation Quality?: A comparative evaluation of SMT , NMT and tailored-NMT outputs. Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, 441–450.
Sutrisno, A. (2020). The Accuracy and Shortcomings of Google Translate Translating English Sentences to Indonesian. Education Quarterly Review, 3(4), 555–568. https://doi.org/10.31014/aior.1993.03.04.161
Taira, B. R., Kreger, V., Orue, A., & Diamond, L. C. (2021). A Pragmatic Assessment of Google Translate for Emergency Department Instructions. J Gen Intern Med, 36(11), 3361–3365. https://doi.org/10.1007/s11606-021-06666-z
Vries, E. De, Schoonvelde, M., & Schumacher, G. (2018). No Longer Lost in Translation?: Evidence that Google Translate Works for Comparative Bag-of-Words Text Applications. Political Analysis, 26(4), 417–430. https://doi.org/10.1017/pan.2018.26
Woodrum, C. (2024). ChatGPT and Language Translation. In H. Degen & S. Ntoa (Eds.), Artificial Intelligence in HCI. Springer. https://doi.org/10.1007/978-3-031-60615-1_10
Zhao, S. (2025). Applications and Challenges of ChatGPT in Translation: A Multidimensional Exploration of Theory and Practice. Sino-US English Teaching, 22(2), 35–41. https://doi.org/10.17265/1539-8072/2025.02.01
Zheng, W. (2017). Translation Strategies for Texts of Science and Technology. International Conference on Humanities Science, Management and Education Technology (HSMET 2017), 96(HSMET), 32–36. https://doi.org/10.2991/hsmet-17.2017.6
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Deri Herdawan; Dhesi Wulan Sari, Iis Sujarwati
This work is licensed under a Creative Commons Attribution 4.0 International License.
In submitting the manuscript to the FOSTER: Journal of English Language Teaching, the authors certify that:
- They are authorized by their co-authors to enter into these arrangements.
- The work described has not been formally published before, except in the form of an abstract or as part of a published lecture, review, thesis, or overlay journal.
- That it is not under consideration for publication elsewhere,
- The publication has been approved by the author(s) and by responsible authorities – tacitly or explicitly – of the institutes where the work has been carried out.
- They secure the right to reproduce any material that has already been published or copyrighted elsewhere.
- They agree to the following license and copyright agreement.
License and Copyright Agreement
Authors who publish with FOSTER: Journal of English Language Teaching agree to the following terms:
- Authors retain copyright and grant the FOSTER: Journal of English Language Teaching right of first publication with the work simultaneously licensed under Creative Commons Attribution License (CC BY 4.0) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors can enter into separate, additional contractual arrangements for the non-exclusive distribution of the FOSTER: Journal of English Language Teaching published version of the work (e.g., post it to an institutional repository or edit it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) before and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
.jpg)
1.jpg)
.jpg)
