FlowtandBriefAI · Software · ResearchMeet Flowtand

Sentence Transformers adds multimodal finetuning for visual document retrieval

Announcement

Hugging Face

Browse Developer tools topic page →

A Hugging Face blog post by Tom Aarsen walks through training and finetuning multimodal embedding models with Sentence Transformers, using a visual document retrieval example. Finetuning Qwen/Qwen3-VL-Embedding-2B on English VDR data reportedly raised NDCG@10 from 0.888 to 0.947 on the author's own evaluation data.

What the post describes

Tom Aarsen's Hugging Face blog post explains how to train or finetune multimodal embedding and reranker models in Sentence Transformers, the Python library for embedding and reranker models used in semantic search and retrieval. The running example is Visual Document Retrieval: matching a text query to document page screenshots that keep charts, tables and layout intact. The pipeline reuses the existing SentenceTransformerTrainer; the difference is that datasets carry images (or other modalities) and the model's processor handles preprocessing automatically. In the example, Qwen/Qwen3-VL-Embedding-2B is finetuned on an English subset of the LlamaIndex VDR data (53,512 samples, with four hard negatives per query resolved to images). The loss is CachedMultipleNegativesRankingLoss, which combines supplied hard negatives with in-batch negatives and uses gradient caching to allow large effective batch sizes on limited GPU memory. On the author's own evaluation data, the finetuned model reaches NDCG@10 of 0.947 versus 0.888 for the base model, and is reported to beat every recent multimodal model tested, including ones up to four times larger. These are the author's measurements, not independent results.

A scenario for a retrieval engineer

Suppose a retrieval engineer maintains search over scanned product manuals, financial filings or policy PDFs where page layout matters. A team could try finetuning a general-purpose multimodal embedding model such as the one in the post on its own query-to-page pairs, so the model learns domain-specific layout, table and chart patterns rather than relying on general image-text matching. The blog's argument for this: general models are trained broadly, so they are rarely the best fit for a single task. A reasonable first move is a small slice of in-domain pages with one hard negative per query, the same minimal configuration the author used, before scaling the corpus.

How to trial and measure it

A team could hold out a fixed evaluation set of queries with known relevant pages before training, record base-model NDCG@10 on it, then finetune and re-measure on the same set. Success means a clear gain over that baseline, not just a higher training loss. Watch for overfitting to synthetically generated queries, a small memory footprint mismatch if the cached loss mini-batch is set too high, and any drop on out-of-domain queries. If memory permits, compare one hard negative against several, since the post notes more negatives strengthen the signal at extra cost. Keep a human spot-check of retrieved pages before switching production traffic, and treat the published 0.888-to-0.947 figure as a reference point rather than an expected result.

Trade-offs and when to skip it

Finetuning needs a labelled query-page dataset, GPU capacity and evaluation effort; the post's example leans on a public data release whose queries were generated by VLMs, which may not reflect real user phrasing. The cached loss trades memory savings for training time. Publishing or deploying a finetuned model brings its own licensing and hosting questions this excerpt does not address. A team without representative in-domain data, or whose task is plain text search, could reasonably wait; the gain reported here is specific to visual document retrieval on the author's data. Treat the result as a prompt to test on your own corpus, not as a guaranteed improvement.

Context and limitations

  • Figures come from the author's own evaluation data, not an independent benchmark.
  • Based on a short source excerpt; the full article, reranker results and configuration details were not reviewed.
  • The reported VDR setup uses an English subset of a public dataset, so domain transfer is untested.
  • Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.