LLMs Are Great for Many Things. Translation is Not One of Them.

Guest blog by Jiri Stejskal, CETRA, CEO — We recently conducted a study comparing translation output generated by Large Language Models (LLMs) using RWS’s  Trados Copilot – an AI Assistant connected to OpenAI’s ChatGPT – to that of machine translation engine ModernMT. Our study included translating a variety of topics including business, technical, and marketing materials from English into Spanish, German, and Brazilian Portuguese.

We then compared the results and found out that the LLM solution is 6 to 34 times slower and 64 to 225 times more expensive compared to ModernMT (depending on which ChatGPT version is used). While the cost and speed of either solution is negligible compared to a translation done by a human, a more significant problem is the quality of the translation. In addition to the already well-documented “hallucination” problem, we also found that LLMs do not respect established terminology, provide different outputs for the same source, and introduce unintended conversion of dates, measurements, and other numerical units. Unfortunately, many of these issues are hard to detect during the review process as the LLM-generated translation reads fluently, which in turn demands a thorough review by human specialists and thus adds to the overall cost.

There are instances where LLMs outperform commonly used neural machine translation engines (NMT) such as ModernMT, and can help with translation-related tasks that  NMT cannot perform at all. Here are some examples:

• Transcreation and content where accuracy of translation is not important.
• Terminology extraction and management when identifying term candidates from large volumes of content.
• Detection of non-translatable content with assistance for creating rules and filters to automate content preparation.
• Changing the register of translated content, e.g. from formal to informal language.
• Adapting content to different locales, e.g. localizing US English content for the UK market.

Many of our findings were confirmed by CSA Research in the “Discussion Session on the State of the Art for LLMs in the Language Sector”, a webinar that took place on April 24, 2025. The webinar was part of the CSA Research GenAI Program of which CETRA is a member.

About the Author: Jiri Stejskal is the President and CEO of CETRA. He has more than 25 years of experience as a translator and founded CETRA in 1997. He earned a PhD in Slavic Languages and Literatures at the University of Pennsylvania and an Executive MBA at Temple University.