AI Product NewsEnterprise Retrieval Brief

NeoMME 260M Puts Retrieval First: A Single-Transformer Multimodal Encoder That Doubles Page Throughput

NeoMME’s 260M-parameter encoder processes text tokens and raw 32×32 image patches in one bidirectional Transformer, returning dense and late-interaction embeddings in a single pass. The result: competitive ViDoRe retrieval quality and about 2× page-encoding throughput versus popular baselines—making visual document search and RAG cheaper, simpler, and easier to scale.

NexusAI EditorialSep 4, 20262.4K views9 min read

NeoMME 260M Puts Retrieval First: A Single-Transformer Multimodal Encoder That Doubles Page Throughput

AI Brief

NeoMME reframes enterprise retrieval: instead of bolting a vision tower onto a language model, it runs text and images through a single long-context encoder and produces both dense and late-interaction embeddings at once. The 260M model sits on the ViDoRe Pareto frontier for quality vs. size and encodes high‑res pages at roughly twice the throughput of common baselines. Late‑interaction normally explodes storage, but hierarchical token pooling plus asymmetric quantization collapse footprint to kilobytes per page with minimal quality loss. For builders, this means faster indexing, simpler serving, and practical visual RAG that preserves layout, charts, and tables without brittle OCR chains. Use dense embeddings for fast recall and late‑interaction for precise reranking—without running two models.

NeoMME’s core idea is architectural restraint in service of retrieval. A single bidirectional Transformer ingests multilingual text tokens alongside raw 32×32 image patches, dropping the separate vision tower and causal decoder common in VLM-style stacks. Trained from scratch with a masked denoising objective, the 260M variant returns dense and late‑interaction embeddings in one pass. In practice, that means fewer moving parts at inference, tighter latency budgets, and simpler scaling. On matched 2048×2048 inputs, the model encodes about 51 pages per second on an L40S—around twice the throughput of widely used visual document retrievers—while keeping retrieval quality competitive for real‑world search and RAG pipelines.

Why does this matter for enterprise search? Most document pipelines still depend on OCR to extract text chunks, then try to reconstruct a page’s layout semantics downstream. NeoMME flips that: it retrieves on page images directly, preserving tables, plots, and typography cues that OCR tends to flatten. The dual‑head design offers workflow flexibility: use dense vectors for fast recall via ANN, then apply late‑interaction to capture token‑to‑region matches without loading a second model. With a 16k context window, dynamic image resolution, and modern encoder features, NeoMME concentrates parameter budget where retrieval benefits most: long context, high‑resolution documents, and multilingual queries.

Late‑interaction accuracy historically came with a steep storage bill. High‑res pages can yield thousands of vectors per document and megabytes of per‑page index footprint. NeoMME mitigates this by combining hierarchical token pooling (to reduce the number of vectors) and asymmetric quantization (to compress document vectors more aggressively than queries). With measured configurations, footprint falls from roughly megabytes per page to tens of kilobytes—down to about 6 kB per page in a more aggressive setting—while retaining the vast majority of baseline retrieval quality. The takeaway: you can afford late‑interaction where it matters, even at scale, without breaking storage budgets.

For builders, the deployment calculus changes. Indexing speeds near 51 pages per second per GPU radically shorten time‑to‑search for large archives and reduce ongoing corpus refresh costs. One forward pass yields both dense and late‑interaction embeddings, so you can standardize your pipeline and defer trade‑offs to the index layer. Fine‑tune with Sentence Transformers for domain‑specific dense or late‑interaction heads, or keep the generalist checkpoint and rely on compression to balance cost and relevance. In visual RAG, retrieve page images first, then pass top‑k pages to a VLM only for answer generation—shifting most of the heavy lifting to a lean, retrieval‑native encoder.

Key Takeaways

One pass, two embeddings

NeoMME returns dense and late‑interaction vectors together, letting you run fast ANN recall and precise reranking without serving two different models.

Throughput translates to cost savings

Around 2× faster page encoding means shorter indexing windows, lower GPU hours, and more frequent corpus refreshes for regulated or time‑sensitive repositories.

Late-interaction is now storage-friendly

Hierarchical pooling plus asymmetric quantization compress multi‑vector pages to kilobytes while preserving most of the retrieval quality you care about.

What Changed: A Single-Tower Multimodal Encoder

NeoMME replaces the dual‑tower or VLM‑plus‑projector pattern with one bidirectional Transformer for both text and images. Inputs are multilingual BPE tokens and non‑overlapping 32×32 patches. The stack uses sliding‑window attention with periodic global layers, grouped‑query attention, QK‑norm, 2D rotary embeddings, and long 16,384‑token context. Because images and text share the same path, pretraining, fine‑tuning, and serving are simpler and more parallelizable. For teams trying to ship retrieval features rather than generation, this concentrates capacity on what matters—fast, accurate embeddings across modalities—without carrying decoder overhead.

Throughput and Quality: Practical Gains for Indexing

At a matched 2048×2048 input size on a single L40S, the 260M retriever encodes about 51 pages per second—roughly twice the throughput of common baselines under similar conditions. On ViDoRe v3, the 260M model sits on the quality‑size Pareto frontier, delivering strong nDCG@10 relative to its compact parameter count. Translation: you can index more, faster, with fewer GPUs, and still compete with larger models on relevance. Expect higher gains on image‑heavy corpora where layout, figure captions, and table structure materially influence retrieval accuracy.

Late-Interaction Without the Storage Tax

High‑resolution pages can produce thousands of token/patch vectors, turning indexes into storage hogs. NeoMME’s late‑interaction head pairs well with two compression levers: hierarchical token pooling to reduce vector counts per page, and asymmetric quantization to store documents at lower precision than queries. In tests, footprint dropped from roughly megabytes to kilobytes per page while retaining most of the baseline nDCG. Choose conservative pooling with int8 for quality‑sensitive archives, or aggressive pooling with binary documents when budget dominates and you still want strong reranking.

Implementation Playbook: Search and Visual RAG

Start by converting PDFs to page images and indexing with NeoMME‑Retriever: store dense vectors for fast ANN recall and late‑interaction vectors for reranking. Use dynamic image resolution to preserve detail on information‑dense pages. Calibrate compression to your storage budget (e.g., pooling 8–10, int8 queries, binary documents for aggressive savings). For visual RAG, fetch top‑k pages and prepend them as images to your VLM prompt. If your corpus is massive, run dense‑only recall on the full index, then rerank a narrow candidate set with late‑interaction. Fine‑tune domain heads via Sentence Transformers when your queries have consistent structure.

Risks, Costs, and Fit

Throughput depends on preprocessed tensors, I/O, and batch calibration; plan headroom for real pipelines. Long‑context encoders can raise activation memory needs at very high resolutions. Image‑based retrieval should respect privacy and security policies, especially for scanned IDs or signatures. NeoMME is not generative—pair it with a VLM for answer drafting in RAG. Finally, compression choices can subtly bias which page regions dominate similarity; validate settings on your actual tasks before committing to an index format.

Frequently Asked Questions

When should I use dense versus late‑interaction embeddings with NeoMME?

Use dense vectors for fast, low‑cost recall across the entire corpus. Retrieve a small candidate set (e.g., top 200–1,000) with ANN, then apply late‑interaction for final ranking to capture token‑to‑region matches. For small corpora or high‑stakes queries, you can directly search with late‑interaction if storage allows.

What hardware footprint should I plan for large‑scale indexing?

As a reference point, the 260M model encodes roughly 51 pages/sec per L40S at 2048×2048 with optimized batching. That implies ~1M pages in ~5–6 hours of pure model time per GPU. Real pipelines face I/O and preprocessing overhead; budget 2–3× wall‑clock. Scale linearly with more GPUs or reduce image resolution where acceptable.

How do I blend image‑page retrieval with text‑only content?

Index page images with NeoMME and keep your text embeddings for content without reliable imagery. Route queries through the same retriever for mixed‑modality scoring. For PDFs with weak scans, store both late‑interaction page vectors and dense text chunk vectors; recall densely, then rerank with late‑interaction when a page screenshot exists.