Researchers from Tencent’s Weixin AI team and Beijing Jiaotong University have released PolyWorkBench, a benchmark of 67 multilingual enterprise workflows, with results showing that even frontier AI agents such as Claude Opus 4.8 and GPT-5.5 lose significant accuracy when workplace tasks cross languages.
The benchmark is the first standardized test of LLM agents on long-horizon tasks, defined as workflows requiring planning, tool use, and structured output across multiple steps, where the workflow itself mixes languages.
Earlier agent benchmarks such as SWE-bench, WebArena, and OSWorld assume a monolingual setting; earlier multilingual benchmarks such as XGLUE and M-MMLU test isolated language tasks. PolyWorkBench combines both, the team said.
The benchmark and what it tests
PolyWorkBench spans 67 manually-curated tasks across five enterprise domains: commerce, knowledge work, legal analysis, localization, and manufacturing. The benchmark covers 10 languages: English, Mandarin, Japanese, Korean, Vietnamese, Russian, French, Spanish, German, and Arabic.
Tasks are designed so that language switching is part of the workflow rather than an isolated translation step. An instruction might arrive in German, a source document might be in Japanese, and a required output might specify French. Across the 67 tasks the average workflow is 8.5 steps long, and 88% of tasks require the agent to operate in three or more languages within a single execution.
Localization is one of the five domains tested, accounting for 11 of the 67 tasks. Sample localization tasks include a Korean chatbot persona adaptation (LOC-09), a Vietnamese marketing campaign (LOC-02), Chinese video subtitles (LOC-06), and a multi-locale software release (LOC-10).
Grading combines three layers: structural checks that confirm outputs conform to schemas, executable verification through Pytest tests that confirm the agent’s outputs run and produce the correct type of result, and an LLM-as-judge layer that assesses semantic and linguistic consistency across the workflow. All tasks are hand-authored by domain-familiar annotators in the source language, with machine translation never used to produce source files, the paper said.
Even Claude Opus 4.8 shows per-language degradation
The best result on the benchmark, from Anthropic’s Claude Opus 4.8 running on the ClaudeCode harness, was a Pass@1 of 0.921. Only three of the 18 leaderboard entries tested exceeded 0.79 Pass@1.
The top-line number masks per-language variance within a single model. Claude Opus 4.8 scored 0.963 on Vietnamese and French tasks but only 0.822 on Russian tasks, a 14-point spread. Mid-tier open-weight models show much larger gaps: Alibaba’s Qwen3.6-35B-A3B running on the Hermes harness scored 0.951 on Mandarin but 0.352 on German.
“The gap between the best and worst per-language mean of a fixed model can exceed 30 Grade points, larger than the overall Pass@1 differences between adjacent leaderboard rows,” the authors wrote.
The frontier model’s reported score also varied by the harness it ran in. Claude Opus 4.8 scored 0.921 with ClaudeCode but 0.712 with the OpenClaw harness, a 21-point swing that the authors say makes raw leaderboard numbers misleading without disclosure of the harness. “Reporting a model’s benchmark score without disclosing the harness is therefore not meaningful,” the paper said.
Two distinct failure modes surfaced in the benchmark, the authors found. The first is source-language misunderstanding, where the agent retrieves or reasons over the wrong content from the start. The second is cross-step consistency loss, where an early misinterpretation compounds through the workflow and corrupts every downstream step. Both patterns can pass deterministic checks but fail the semantic LLM-as-judge layer, the paper reported.
Localization as a separate axis
Localization is “the only domain where a non-Claude model obtains the second-best score at every tier,” the paper reported. On the localization subset of tasks, DeepSeek-v4-Flash with the Codex harness scored 0.855 and Alibaba’s Qwen-Agent-World with OpenClaw scored 0.822, while Claude Opus 4.7 with ClaudeCode led the localization domain specifically at 0.836.
The result reflects an underlying pattern the paper documents: “localization tasks reward strong multilingual base models more than they reward long-horizon planning.” A model with strong multilingual pre-training can outperform a model with stronger agentic scaffolding on localization tasks specifically, the paper found, even when the agent-scaffolding model wins overall.
Per-language failures are most pronounced in European languages for mid-tier open-weight models. Alibaba’s Qwen3.6-27B with the OpenClaw harness scored 0.237 on Spanish tasks and 0.133 on German tasks. Spanish and German weaknesses are particularly relevant for European-focused language services companies operating in regulated industries, where localization accuracy in Romance and Germanic languages is often a baseline requirement.
No single model consistently performed best across all enterprise workflows tested, the paper found. Some systems showed much larger performance gaps between languages than others; the spread within a single model can exceed the spread between adjacent leaderboard rows.
Implications for the language industry
PolyWorkBench is the first standardized benchmark to evaluate AI agents on multilingual long-horizon workplace workflows, with localization tested explicitly as a separate domain. The benchmark, including task definitions, Pytest suites, reference solutions, and LLM-as-judge prompts, is released publicly with the paper.
The findings matter for language services companies deploying AI agents in localization, translation, and content adaptation work. Even the top model on the benchmark loses up to 14 points depending on the language being processed, and mid-tier open-weight models can lose more than 50 points between languages on the same workflow.
A linguistic misunderstanding at step 2 changes what the agent passes to step 3, which changes step 4’s input, and so on; by the end of an 8 to 12-step task, a single early error can propagate into every downstream output.
The benchmark is also the first to document that an agent’s reported score can swing by 8 to 21 points depending on the harness it runs in, the authors said, meaning a model marketed as multilingual may not be so in any given deployment configuration. The researchers plan to expand the benchmark with additional languages, domains, and enterprise workflows over time.
The paper was authored by Hongliang Li and Yijin Liu, Zhiwei Zhang, Zihe Liu, Xinyue Lou, Jinan Xu, Fandong Meng, and Kaiyu Huang at Beijing Jiaotong University and Weixin AI, with Li and Zhang working on the project as Tencent interns. Kaiyu Huang is the corresponding author.
