Multimodal RAG for Tables, Charts and Scans: Three Architectures and When to Route by Page

The question is about third-quarter margin by region. The number sits in a stacked bar chart on page 47 of the board pack. The ingestion pipeline turned that chart into one string: "Figure 4: Q3 Revenue." The model never saw the bars. It answered anyway, and the answer was a guess.

This is not a prompt problem. It is an architecture problem. A text-only pipeline reads the text layer of a page and throws away the layout that gave the text its meaning. If your documents contain tables, charts, forms, schematics or scanned pages, that loss is built in.

Three ingestion architectures handle this. Each has a different cost and a different way of failing. Choosing the right one per page matters more than choosing the best model.

Why text-only extraction keeps failing

A standard pipeline does four things. It detects layout, extracts the text layer or runs OCR, splits the text into chunks, and embeds each chunk. The failures are predictable:

• ▸Multi-column tables are read across columns, so rows mix. In a clinical protocol this is a pediatric dose read as part of the adult column.
• ▸Merged cells and nested headers collapse into ambiguous strings.
• ▸Charts and diagrams are skipped, or replaced by their caption.
• ▸Scanned pages inherit OCR character errors, which cascade into bad chunks.
• ▸Forms lose the pairing between a label and its value.

Teams respond by adding parsers. A table extractor here, a figure-captioning model there, a regular expression to repair one vendor's template. Each new template is a new rule. Each rule is a new place the pipeline returns the wrong answer without raising an error.

The useful question is not "which parser is best". It is "does this page need its text extracted, or does it need to be retrieved as a page?"

Three architectures compared

There is no single multimodal RAG. There are three patterns, and production systems usually combine them.

1. Page-image retrieval (ColPali style)

Render each page as an image. Pass it through a vision embedding model that produces many small vectors per page, one for each patch of the image. A patch is a small square of the page, so a chart's bars and a table's grid each get their own vectors. Store all of them. At query time, embed the question and compare it against every patch vector of every page. This is called late interaction: the match happens between the query and each patch, not between the query and one summary vector. The best-matching page images then go to a vision-capable model, which answers from the picture itself.

The ColPali paper (arXiv 2407.01449, July 2024) introduced this approach and showed it could skip text extraction entirely.

• ▸Strength: no extraction step to break. Charts, diagrams, handwritten forms, Arabic script and messy scans all retrieve on what the page looks like.
• ▸Cost: storage. A single-vector text index stores one vector per chunk. A page-image index stores one vector per patch. As an illustrative example, a page split into 1,000 patches of 128 dimensions stores 128,000 numbers, against 1,024 for one 1,024-dimension chunk vector. That is about 125 times more storage per page before compression. Pooling neighboring patches and binary quantization bring it down. Qdrant's multivector documentation caps sub-vectors times dimension at 1,048,576 per point and recommends pooling token vectors before storing them.
• ▸Weakness: citations are page-level, not sentence-level. You cannot highlight the line the answer came from. Scoring every patch adds latency, more so without a GPU.
• ▸Best for: visual documents whose layout you cannot predict. Bills of lading, old scanned archives, mixed engineering manuals, slide decks.

2. Layout-aware parsing (Docling, Unstructured, Marker)

Use a model that understands document structure to extract text with the layout kept. Tables stay tables, headings stay headings, reading order survives, figures are tagged. Chunk by unit: a section, a table, a caption. Embed those chunks with an ordinary text embedding model.

• ▸Strength: cheap at query time, single-vector retrieval you already run, sentence-level citations, any vector database.
• ▸Cost: the parser. Layout detection and table recovery need GPU time per page, and quality varies by document type. Docling describes its method in a technical report. Unstructured publishes a benchmark page scoring parsers on text fidelity, invented tokens and table cell accuracy over 1,000 enterprise pages. It is vendor-published, so read the methodology before trusting the ranking. Marker is the open-source option for PDFs with equations and figures.
• ▸Weakness: charts still become captions. Hand-drawn or non-standard forms degrade. You are trusting a parser to be right.
• ▸Best for: well-structured documents with a real text layer. Regulatory filings, policy manuals, contracts, published reports.

3. Vision-model captioning of figures

Keep the text pipeline for prose. For every detected figure, chart or diagram, call a vision-language model and ask for a structured description. An illustrative caption: "line chart; x-axis quarters Q1 to Q4 2025; y-axis revenue in USD millions; three regions; EMEA falls from 42 to 31 between Q2 and Q3." Store that description as an extra chunk linked to the figure's page.

• ▸Strength: it composes with any existing stack. Charts become searchable without changing retrieval.
• ▸Cost: one vision-model call per figure at ingestion. Budget it as a one-time cost per document, not a recurring one.
• ▸Weakness: the caption is only as good as the model and the prompt. A chart compressed into a paragraph loses small values.
• ▸Best for: text-heavy corpora with a long tail of important figures. Research reports, product documentation, pitch decks.

Which to pick

Page routing: the pattern that beats a single pipeline

Real corpora are not one document type. A procurement archive holds typed contracts, scanned faxes, spreadsheet exports saved as PDF, and vendor datasheets full of diagrams. Push all of it through one pipeline and one subset is handled badly.

The pattern that works is a router at ingestion time, applied to each page:

1. ▸Run a cheap classifier on the page. The classes are text-dominant, table-heavy, figure-heavy, and scanned with a thin or missing text layer.
2. ▸Route by class. Text-dominant pages go through layout-aware parsing. Figure-heavy pages go to captioning or page-image embedding. Scanned and visual pages go to page-image retrieval.
3. ▸Store the page type with the chunk, so retrieval can weight or filter by it. The fields page_type, page number and bbox belong in the chunk metadata schema, next to dates, access groups and status.

This is unglamorous infrastructure work. It is also the single most effective change for a mixed corpus. A system that picks the right tool per page beats a single clever embedding model.

What a working ingestion pipeline looks like

A production multimodal pipeline, stripped to its essentials:

PDF → page splitter → page classifier
                        │
                        ├── text pages  → layout-aware parse → text chunks → text embeddings
                        ├── figure pages → VLM caption + text pass → paired chunks → text embeddings
                        └── visual pages → page render → vision embeddings (multi-vector)

                        all chunks → vector store (with page_type metadata, document_id, page_number, bbox)
                        raw page images → object store (S3, R2), referenced by URL

A bbox, short for bounding box, is the rectangle on the page a chunk came from: four coordinates. It is what lets the interface draw a highlight on the source page.

Two rules that save pain:

• ▸Never store image bytes in the vector database payload. Store the URL to the object store. A lean index keeps search fast.
• ▸Always store the page number, and the bbox when you have it. A citation without a page reference is not a citation. Users and auditors need to click through to the source.

Cost: use your own numbers

Multimodal ingestion costs more per page than text-only. How much more depends on the architecture and on how heavy your pages are. Rather than quote someone else's multiplier, build the figure from your own corpus.

Let P be pages ingested per month, V the fraction of pages routed to vision embedding, and F the figures per text page sent to captioning. Monthly ingestion cost is then approximately:

• ▸Text embedding: P × tokens_per_page × text_embed_price
• ▸Vision embedding: P × V × tokens_per_page_image × vision_embed_price
• ▸Figure captioning: P × (1 − V) × F × (vlm_input_tokens + vlm_output_tokens) × vlm_price
• ▸Storage: proportional to vector count times dimension. Multi-vector pages multiply the count by the patches kept per page after pooling.

Take the prices from your provider's pricing page on the day you build the budget. Prices move. The pattern does not: ingestion is paid once per document, retrieval is paid per query. If query volume is high and the corpus is bounded, pay for better ingestion once.

The comparison worth making is not "text is cheaper than vision". It is "what does a wrong answer cost us?" If the answer feeds a procurement decision, a clinical workflow or a filing, returning the caption instead of the chart is not an API line item.

Running it at scale

Images are heavier than text. Page-image embedding at scale is bound by GPU time, and a sequential script will not finish before users notice. Two things matter:

• ▸Parallelize by page, not by document. A distributed data engine such as Ray Data, Daft or Spark fans the pages out across GPUs. Watch for backpressure. That is what happens when a slow stage, such as one vision-model call per figure, cannot keep up with the stage feeding it. The queue between them grows until something fails. Size the slow stage, or cap how far the fast stage can run ahead.
• ▸Separate ingestion from serving. Ingestion is a batch job with retries and checkpoints. Serving is a low-latency path. They share nothing except the vector store and the object store.

Retrieval with multi-vector page embeddings is slower than single-vector text retrieval. Answering from a page image is slower than answering from a string. How much slower depends on your hardware, your pooling choice and your model. Measure both on your own stack before you promise a response time.

Where this fits with fine-tuning

Multimodal RAG and fine-tuning solve different problems. Retrieval gets the right page in front of the model. Fine-tuning changes how the model reads that page. For most document work, retrieval is the bottleneck. A well-routed pipeline with a general vision model beats a fine-tuned text-only model working on broken extractions. The RAG vs fine-tuning framework covers when that flips.

Frequently asked questions

Do we need to replace our vector database? Usually not. Qdrant, pgvector and most mature stores accept whatever embedding dimension you use. Multi-vector retrieval in the ColPali style needs a store that supports it natively, or a reranker in your application. Qdrant supports multivectors directly; pgvector needs more plumbing. If you stay single-vector, your existing database is fine.

Can this run fully on-premise? Yes. Open-weight vision embedding models from the ColPali family run on your own GPUs. So do vision-capable models from the Qwen and Llama families, under vLLM or SGLang. Plan for more GPU memory than text-only serving, and for image preprocessing throughput, not just tokens per second. The on-prem LLM speed guide covers the serving levers.

What breaks first when we scale from 1,000 to 100,000 documents? Ingestion time and index size. A pipeline that handles 1,000 PDFs on a laptop in an afternoon can take weeks on 100,000 if it runs single-threaded. The multi-vector index can also outgrow memory. Decide on a distributed ingestion framework, and on quantization and pooling, before indexing the full corpus. Reindexing 100,000 documents is a project nobody wants to run twice.

This week, take 50 pages from your real corpus and classify them by hand into text, table, figure and scan. Count each class. The shares tell you which pipeline each slice of your corpus needs, and whether you need a router at all. Then write ten questions whose answers sit in those tables and charts. Add them to a test set that includes questions whose answers sit in tables and charts. If your current system fails them, you have found the page-47 problem before a user did.