Intel-Stack RAG Walkthrough: Xeon Embeddings, Gaudi 2 Generation
Hugging Face
- Infrastructure
- Developer tools
- Industry
- #gaudi
- #xeon
- #tgi
- #rag
Browse Infrastructure topic page →
A Hugging Face blog post by Julien Simon, Haihao Shen, Antony Vance Jeyaraj and Intel co-authors describes a RAG chatbot built from LangChain's rag-redis template, with the embedding model on an Intel Granite Rapids CPU and the LLM served by TGI on Gaudi 2. The authors report their own benchmark comparing Llama2-70B throughput and performance per dollar on four Gaudi 2 versus four H100 GPUs.
What the post actually ships
The authors describe a retrieval-augmented generation chatbot assembled from LangChain's rag-redis template: Nike 10-K filings are loaded as PDFs, split into chunks, embedded with BAAI/bge-base-en-v1.5 and stored in Redis as the vector database. The retriever, a prompt template and a Hugging Face endpoint are chained together, with the model served by Hugging Face Text Generation Inference on Gaudi 2 through the Optimum Habana interface. The embedding model runs on an Intel Granite Rapids CPU. Authors note FP8 quantization may raise throughput and that Llama Guard can be added for content moderation. The benchmarks (Llama2-70B, 16 concurrent users, four Gaudi 2 vs four H100) are the authors' own; the post says H100 showed 1.13x the throughput but 0.44x the performance per dollar of Gaudi 2, with the caveat that discounts vary.
Application scenario: platform engineer at a regulated firm
Consider an enterprise platform engineer who must keep document Q&A inside the company perimeter and already has Intel-based servers. A team could try the described split: embed internal policy or financial documents on Xeon CPUs, write vectors to Redis and serve an open-weight chat model through TGI on Gaudi 2. This maps naturally to workloads where ingestion is steady but generation latency tolerates batch queues, and where data cannot leave the datastore the firm controls. Because the post uses public filings as sample data, a firm would substitute its own gated corpus and confirm access controls around the Redis index and the TGI endpoint before any pilot. This is a trial proposal, not a claim that the stack meets any compliance regime.
How to trial it without fooling yourself
A team could start with the post's own pipeline shape on a small document set, then measure two baselines: retrieval quality (are the right chunks returned for a fixed question set?) and end-to-end throughput at a chosen concurrency level. Run the same questions before and after any change — FP8 quantization, a different embedding model, shard count for a larger model — and record latency, tokens per second and any change in answer accuracy scored by a human reviewer. Failure checks should include retrieval misses where the answer exists in the corpus but not in the returned context, and citation mismatches where the model cites a chunk that does not contain the claim. Keep humans reviewing a sample of answers throughout; the authors' throughput figures say nothing about answer quality.
Trade-offs, prerequisites and when to hold off
Prerequisites include access to Gaudi 2 hardware, the TGI Gaudi container and the Habana runtime flags shown in the post, plus a Redis instance and the LangChain template's dependencies. The performance-per-dollar comparison depends on publicly listed pricing and the post notes it varies with customer discounts, so it should not be treated as a general cost claim. Larger models may need multiple accelerators and sharding parameters. A team should skip this route if it has no Intel accelerator footprint and no appetite to maintain Habana tooling, if strict latency at high concurrency is the primary requirement (the H100 comparison favors raw throughput), or if a managed RAG service already satisfies data-handling constraints. The post is a vendor-authored walkthrough from May 2024; treat its benchmark as a starting hypothesis to replicate.
Context and limitations
- Based on a short source excerpt; the full article, including complete benchmark configuration notes, was not analyzed.
- Benchmark results are the authors' own experiments; throughput and performance-per-dollar claims were not independently verified.
- Hardware access, pricing discounts and FP8 gains depend on cloud, procurement and configuration specifics not covered here.
- Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.