Agentic AI tools can translate more than just words — they can also incorporate video and audio sources to further refine and add context to their actions and results.

Generative AI (genAI) agents are changing language translation, with actions, emotions and diagrams now driving communication across languages.

Translation agents from a few companies go beyond text and use voice, image, and video to communicate across languages in real time and with high accuracy. “You need to be able to deal with multimodality. It’s not just plain text. There’s a lot more relevant data that you have to ingest,” said Matt Hardy, president of linguistic AI at RWS.

Computer translation isn’t new. But with genAI agents talking and automating actions in productivity and customer service, the stakes are higher. There’s little room for mistakes, especially as agents replace humans to get work done.

Translation agents from RWS, DeepL and Grammarly analyze information sources, verify intent, establish context, and deliver translations that go beyond words. “The beauty of agents is that they are so pliable. They’re supposed to be modular,” Hardy said.

The ability for AI to translate audio, video and text helps the genAI tools capture context and nuance — actions in a video, or specific tones in audio, for example — and pack that into translation.

“Languages are not about just a written representation of pure thoughts,” said Stefan Mesken, chief scientist at DeepL. “There is context, both cultural, also situational. To get this right, you have to be very close to your customers, which includes speaking their language.”

Translations are now trending toward a broader understanding of the intent of the actual user. Like expression, physical context plays a role in getting the nuances right.

“The other way around to address this is to create a multilingual agent… [that] can freely switch between any language that might be beneficial for this line of work, which much more closely resembles how humans interact,” Mesken said.

The UN estimates there are 8,324 languages worldwide. For high-resource languages like English, the translation tools are generally good, but translation quality sharply drops off in other languages.

“Another way to address this, of course, is to bring language AI into the agentic space, essentially as an accessibility layer, similar to how we do this with humans,” Mesken said.

Historically, companies hired multilingual support teams who spoke multiple languages, outsourced it, or built separate language-specific teams. Translation AI agents are changing that, said Ailian Gan, director of product management at Grammarly.

Multilingual communication improves the customer experience “while also making it easier and more cost-efficient for businesses to expand internationally,” Gan said.

DeepL recently released a “DeepL agent” — a general-purpose AI agent with built-in translation capabilities based on a translation API that supports 36 languages. The agent can establish multimodal context.

“For media, images and diagrams specifically, since so much of this world is encoded in feature and media rich documents, this is of course one of the areas where we focus as well,” Mesken said.

The company also has writing and real-time voice translations. It supports 36 languages and is focusing on quality, not quantity, as accuracy is important.

ComputerWorld

Article link