In 2024, a research team set out to learn how well AI chatbots handle Hindi, Tamil, Bengali and seven other Indian languages. They did not hand the grading to another machine. They had native speakers, hired through a data company called Karya that the paper describes as ethical, read the answers and score them. The study, called Pariksha and posted to the arXiv research archive in June 2024 by a team that included Microsoft researchers, counted about 90,000 human evaluations across 30 models.

That kind of work explains why so many AI listings ask for a second language.

Why machines cannot grade themselves

A 2023 paper by Rishav Hada and colleagues, also on arXiv, tested whether one AI model could stand in for human reviewers when judging another model's answers in many languages. It found that the automated judges tended to give higher scores than people did. The authors concluded that those judges need to be checked against native speakers, especially for languages with fewer digital resources and for languages written in non-Latin scripts. Evaluation by native speakers, they wrote, remains the gold standard, but it is hard to scale because of cost, turnaround time and the difficulty of finding speakers. As a result, how well AI performs is unknown for most of the world's languages.

Translation is not the same as native writing

A second study, posted in June 2024 by Pinzhen Chen and colleagues at the University of Edinburgh and Tsinghua University, compared models tuned on text originally written in a language with models tuned on text translated from English. Across eight base models and eight tests, the researchers found a gap between the two. The gap was larger on tests built by native speakers and on open-ended writing tasks. Their advice was that non-English evaluation should include native-language and generative tasks. The Pariksha authors made a related point: many multilingual benchmarks are translations of English originals, which can lose linguistic and cultural context.

What one vendor says it does

Welo Data, a company that sells data services to AI developers, published an account on May 12, 2026, of a four-week pilot for an unnamed AI lab. The project covered 20 locales and more than 36,000 annotations across text and speech tasks. According to the company, a native-speaking quality controller in each locale audited about half of the completed tasks. In one workflow, native speakers recorded spoken answers of 90 to 120 seconds so reviewers could judge delivery, tone and completeness. The company also said that translating and adapting English-language guidelines improved clarity and performance in non-alphabetic languages. This is the vendor's own description of its work, so readers should weigh it accordingly.

What it means for the listings

Many of this week's bilingual listings pair a second language with English at an upper-intermediate level, shown on the listing as B2 or higher, along with attention to detail and written communication. A few involve audio, such as the British English voice role and the Hindi role that lists voice recording. Most of the bilingual roles advertise $30 to $65 an hour. Those are advertised ranges, not promises, and nothing in the research says which languages will be needed next. Check each posting for its current requirements.

I may be paid if you apply through the links on this page. I am not an employee or representative of the company that posts these roles.

Browse current expert roles

Sources: Pariksha study, arXiv, June 2024; Hada and colleagues, arXiv, 2023; Chen and colleagues, University of Edinburgh and Tsinghua University, arXiv, June 2024; Welo Data case study, May 12, 2026.