Arabic AI in Production: Tokenization, Dialects, and Search That Finds the Right Document

An Arabic AI demo usually works. A vendor loads twenty clean policy pages and asks five polite questions in Modern Standard Arabic. The answers look right, and the budget gets approved.

Then real traffic arrives. Customers write in Saudi or Emirati dialect. Staff paste half-English emails. The policy library turns out to be scanned PDFs with Arabic text and Western digits mixed on every line. Retrieval misses obvious documents, answers get slower, and the token bill runs higher than the estimate.

None of this is mysterious. Arabic breaks English-first pipelines in four predictable places. Each one can be tested before launch and fixed with known tools. The demo simply never touched the data that matters.

1. Tokenization: why Arabic can cost more and run slower

Language models do not read words. They read tokens, which are fragments of words. A model's tokenizer decides how text gets cut up, and it was built from whatever text the model maker prioritized.

When a tokenizer saw relatively little Arabic during construction, it cuts Arabic words into more pieces. That has three business consequences:

• ▸Cost. Hosted models bill per token. More tokens per word means a higher bill for the same document.
• ▸Speed. Models write answers one token at a time. If Arabic needs more tokens per sentence, the user waits longer for the same answer.
• ▸Capacity. Every model has a fixed context window, the amount of text it can consider at once. More tokens per page means fewer pages of evidence fit in.

How large is the penalty? It depends on two things: which tokenizer, and which text. Newer tokenizers from the major providers handle Arabic noticeably better than older ones. OpenAI's o200k encoding, used by its recent model families, was built partly to improve non-English efficiency. Open-weight families with large multilingual vocabularies, such as Qwen, and Arabic-first models such as Jais, also cut Arabic more efficiently than older English-centric tokenizers.

We will not quote you a single ratio. Published figures vary by tokenizer version and by text type. Contracts, chat messages and call transcripts all tokenize differently. Quoting one number would mislead you.

You can measure it on your own data in an afternoon, and you should, before you choose a model:

1. ▸Pull 200 representative documents or messages from your real traffic. Include dialect, not just formal text.
2. ▸Count words, then count tokens with each candidate tokenizer. OpenAI publishes its tokenizer as the open-source tiktoken library. Open-weight models ship their tokenizers alongside the weights.
3. ▸Divide tokens by words for each tokenizer. That gives you tokens per word on your own text.

Then put your own numbers into this:

Monthly token cost = Arabic words per month × tokens per word × price per token

Run it once per candidate model. The gap between rows is your real tokenization penalty, in your currency.

Latency works the same way. Answer time ≈ answer length in words × tokens per word ÷ generation speed in tokens per second. For a voice agent, that difference is the pause the caller hears.

Tokenization is one input, not the whole decision. A cheaper tokenizer attached to a weaker model can still lose money through wrong answers. Measure tokens, then measure accuracy on the same test set.

2. Dialect: the customer's words are not in your documents

Your policies are written in Modern Standard Arabic (MSA), the formal register of contracts and government publications. Your customers do not write that way. They write in Gulf, Egyptian or Levantine dialect, often switching to English mid-sentence. Some write Arabizi: Arabic in Latin letters, with digits standing in for sounds, like "3" for ع and "7" for ح.

A dialect question and the MSA paragraph that answers it can share almost no words. Keyword search finds nothing. The system then tells a paying customer it has no answer, and the ticket goes to a human anyway.

The fix sits in the embedding model. That is the component that turns text into numbers, so that passages with similar meaning end up close together. An English-first embedding model places a dialect query and its MSA answer far apart. A strong multilingual embedding model places them close.

Production-grade choices include the multilingual E5 family, BGE-M3, and Cohere's multilingual embedding models. Add a reranker, a second model that re-scores the top candidates more carefully, and dialect-to-MSA matching improves again.

Do not take any vendor's word for which one wins on your data. Build a test set that pairs real dialect questions with the MSA passages that answer them. Then compare recall: how often the right passage appears in the top five results. You can build the paired dialect test set the same way as a 100-question RAG test set, drawn from real tickets rather than written by engineers.

Fine-tuning the language model is rarely the answer here. The failure happens at retrieval, before the model ever writes a word.

3. Keyword search: clitics, not chunking

It is a common misconception that chunking cuts Arabic words in half. Most RAG frameworks use recursive splitters. These split on paragraph breaks, then line breaks, then spaces. They rarely cut a word in the middle. Only a raw character splitter with no separators does that, and you should not be using one for any language.

The real Arabic problem is in keyword search. Production retrieval usually runs hybrid search: meaning-based vector search plus classic keyword search (BM25) for exact terms like names, product codes and article numbers. Keyword search matches exact words. Arabic attaches small words directly to the word they modify. These attached pieces are called clitics.

Take the word for "contract", عقد. In real text it appears as العقد (the contract), والعقد (and the contract), بالعقد (by the contract), and عقدنا (our contract). Without Arabic-aware processing, keyword search treats these as four unrelated words. A search for العقد misses every paragraph that says والعقد.

Spelling variation makes it worse. The same word can appear with different alef forms (أ، إ، آ، ا). Final ة and ه get swapped, and so do ى and ي. Diacritics appear in some documents and not others. Writers stretch words with the tatweel character (ـ). Numbers appear as Arabic-Indic digits (٢٠٢٦) or Western digits (2026).

The fix has two parts:

• ▸Normalization. Unify alef forms, strip diacritics and tatweel, and standardize digits. Do this before indexing and before every query, with the same rules. Elastic documents a character filter that converts Arabic-Indic digits to Western digits during tokenization.
• ▸Light stemming or segmentation. Strip common clitics so all four forms of عقد match. Elasticsearch and OpenSearch ship a built-in Arabic analyzer that does both normalization and light stemming. For heavier analysis, Farasa from QCRI and CAMeL Tools from NYU Abu Dhabi segment Arabic words into their parts.

If you use Qdrant, recent releases added language-aware stemming to its full-text index. Confirm Arabic is supported in the version you run before relying on it. Otherwise, normalize and stem in your own pipeline before indexing. That approach works with any vector database. The companion build sheet carries the full normalization rule list and PDF extractor ranking, with index-side and query-side columns for each rule.

4. Document extraction: scrambled before the AI sees it

Many Gulf document libraries are PDFs that mix Arabic text with English terms and Western numerals. PDFs store text in drawing order, not reading order. Extract carelessly and you get Arabic lines reversed, numbers moved to the wrong side, or tables flattened into noise.

Some PDFs also store Arabic as presentation forms. These are special Unicode characters for the joined shapes of letters, not the letters themselves. The text looks correct on screen but matches nothing in search. Unicode normalization (NFKC) converts them back to standard letters.

Scanned documents need OCR with Arabic support, then a manual check on a sample. Downstream metrics are unreliable if the text going in is flawed. Microsoft documents Arabic support in the Read, Layout and Custom Form models of Azure Document Intelligence.

The Arabic failure-mode checklist

Run this before you call an Arabic system production-ready. Each row is a test you can run in under a day.

Failed Arabic pilots tend to fail at the top three rows. Those are also the cheapest to fix.

Three architectures, one sensible default

You have three broad options for handling Arabic.

Translate everything to English, run an English pipeline, then translate back. It is quick to prototype in a workflow tool. It adds two extra steps, and their cost, to every interaction. It also loses dialect nuance and mistranslates legal and regulatory terms. That is exactly where errors are expensive. We do not recommend it for production.

Use a hosted frontier model directly in Arabic. The leading Claude, GPT and Gemini families handle MSA well and dialect reasonably. Infrastructure overhead is zero. Your tokenization cost depends on the provider's tokenizer, so measure it.

Run an Arabic-capable open-weight model on your own servers, such as Qwen or Jais, with multilingual embeddings. You pay fixed infrastructure costs instead of per-token fees. You keep data inside your environment, which matters under Gulf data residency rules. MBZUAI, Cerebras and Inception describe Jais 2 as the largest open Arabic-centric model trained from scratch, at 70B parameters.

The right default for most Gulf businesses is the second option at pilot stage. Move to the third once volume or data residency justifies the infrastructure. Either way, the retrieval fixes in the checklist above apply. A better language model cannot answer from a document that search never found.

Where Arabic AI pays first in the Gulf

The highest-value Gulf use cases share one trait: high volume in Arabic, with English mixed in.

• ▸Call and reception automation for clinics, real estate and hospitality. Speech recognition quality on Gulf dialect decides success here. See our guide on a voice AI pipeline <500ms end-to-end before choosing one for your voice AI build.
• ▸Document processing for Arabic contracts, permits and regulatory filings, where extraction and normalization do most of the work.
• ▸Internal knowledge systems over Arabic policy manuals and procedures. This is where every row of the checklist applies at once.

Our team works natively in Arabic and English. The dialect test sets and the review of extracted text are part of the build, not an afterthought.

Frequently Asked Questions

Can we put a translation API in front of our existing English AI system? For a prototype, yes. For production, it is the wrong choice. Every interaction pays for two extra translation steps and waits for them. Dialect and regulatory terminology mistranslate, so the system retrieves the wrong documents and answers confidently from them.

How do we measure what the Arabic tokenization penalty costs us? Measure it rather than trusting a published ratio. Take 200 real documents, count tokens per word with each candidate model's tokenizer, and apply the cost formula above. The result is specific to your text and your prices, which is the only number worth budgeting against.

Do we need to fine-tune a model to understand Gulf dialect? Usually not. Most dialect failures happen at retrieval, before the language model writes anything. A multilingual embedding model, a reranker, and proper Arabic normalization fix most of them. Fine-tuning makes sense only after a dialect test set shows the language model itself is the weak point.

This week, run rows 1 to 3 of the checklist on your own data. Extract 20 real PDFs and compare them to the page. Take 50 real queries from your logs, search for each in its alef and ta marbuta variants, and query the base forms to see which prefixed forms go missing. Record every miss. That list tells you whether your problem is extraction, normalization or stemming, before anyone argues about which model to buy.