Apart from English, if you talk to AI in Hindi or any other language, then using AI chatbot can prove costly for you. Companies like OpenAI, Anthropic and Google often tout their new AI models as tools that work the same for everyone, no matter where they live or what language they speak. But new data from researchers shows that users who use AI in languages like Hindi, Arabic and Chinese may actually have to pay more than those who talk to AI in English.
What is the reason for this?
Depending on how the AI model processes the language, the same prompt (instruction) in Hindi can generate many more tokens, i.e. the units that the AI system uses to read and understand the text, than in English. In simple words, AI spends more tokens on saying the same thing in Hindi than in English. Therefore, for those who do not speak English, AI It becomes more expensive to use.
Language tax, what is the matter?
Researchers and developers often call this a language tax or consider it a hidden cost, which is due to the way AI models process different languages. A few weeks ago, OpenAI researcher Aran Komatsuzaki described an experiment examining how OpenAI and Anthropic’s tokenizers process text in different languages. Using AI researcher Rich Sutton’s famous article The Bitter Lesson as a benchmark, Komatsuzaki translated the text into multiple languages and looked at how many tokens different AI systems produced.
The results showed that there is a huge difference between English and other languages. According to the analysis, on OpenAI’s tokenizer, Hindi text required 1.37 times more tokens than English. At the same time, this figure increased to 3.24 times on Anthropic’s Claude Tokenizer. On Claude, 2.86 times more tokens were required for the Arabic language, while 1.71 times more tokens were required for the Chinese language.
Read this also- ‘Dangerous files’ coming on WhatsApp, system can be hacked as soon as clicked
Leave a Reply