RAG retrieval benchmark: keyword vs vector vs hybrid search on BEIR SciFact

The numbers, first

• 0.699: nDCG@10 for vector search BAAI/bge-small-en-v1.5 embeddings, cosine KNN in KiviDB. The model's published SciFact score is about 0.71.
• 0.688: nDCG@10 for hybrid search Keyword and vector results fused by reciprocal rank. On this dataset, slightly below vectors alone.
• 2.6 ms: Vector query, median Top 50 of 5,183 by cosine, timed from a client in the same AWS region as the database. 95th percentile 2.8 ms.

Free-tier database (shared host, 100 MB) in AWS eu-central-1. Timings are from a client in the same region, second of two passes over the 300 queries.

How we ran it

• The data SciFact from the BEIR benchmark: 5,183 scientific abstracts and 300 test claims, each with judgements of which abstracts support or refute it. It is a standard test for retrieval in RAG systems.
• Embeddings BAAI/bge-small-en-v1.5 (384 dimensions) on a laptop CPU: about 13 minutes for the abstracts, 4 seconds for the claims. Claims get the model's search instruction; abstracts are title plus text.
• Load and index One hash per abstract (title, text, vector), sent in pipelines of 250: 47 seconds from a laptop in India. FT.CREATE then indexed all 5,183 in 15.5 seconds.
• Three retrievals Keyword: the claim's words, stop words removed, OR'd together over title and text. Vector: KNN 50 by cosine. Hybrid: both top-50 lists fused by reciprocal rank (k = 60) in the client.
• Scoring nDCG@10 and recall@10 against SciFact's judgements, the measures BEIR reports. Latency: the same 300 queries from a task in eu-central-1, timed per command.

Vectors led on every measure

Mean over the 300 test claims. Higher is better.

Where the time goes

Milliseconds per query from a client in the same AWS region, drawn to one scale. Hybrid runs both searches at once, so it costs as much as its slower half.

Network round trip to the database: 1.8 ms. A free-tier database shares its host, so a dedicated database's timings can differ.

Three findings worth keeping

• Meaning beat matching words SciFact's claims are phrased differently from the abstracts that answer them, so an embedding that captures meaning outscores shared words. Our 0.699 through KiviDB is close to the model's published 0.71.
• Hybrid is not free accuracy Fusing in keyword results moved some keyword-only matches above better vector ones, and nDCG fell by 0.011. Hybrid tends to help where exact names, codes and identifiers matter. Measure it on your own questions before adopting it.
• Keyword cost follows matches Keyword time grows with the documents its words match: one common word that matched 1,826 abstracts took 54 ms. Searching only a claim's 8 rarest words cut the median from 171 to 110 ms, with hybrid nDCG unchanged at 0.687.

The fusion, in the client

Reciprocal rank fusion needs only the two ranked lists. This is the function we scored.

def rrf(lists, k=10, c=60):
    score = {}
    for ranked in lists:
        for rank, doc in enumerate(ranked):
            score[doc] = score.get(doc, 0) + 1 / (c + rank + 1)
    return sorted(score, key=score.get, reverse=True)[:k]

keyword = search(f"({'|'.join(words)})", limit=50)
vector  = search("*=>[KNN 50 @vec $v AS d]", v=embedding)
top10   = rrf([keyword, vector])

• Both searches go to the same index in one database.
• Run them concurrently: the hybrid query then takes as long as the slower of the two.

Every figure in one table

Recall was not recomputed for the 8-word variants. Latency is from a client in AWS eu-central-1, the database's region.

Questions

Was this a production-sized database?

No. It ran on a free-tier database: a shared host with a 100 MB memory limit. Retrieval quality does not depend on the hardware; latency does, and a dedicated database can be faster or slower than these figures.

Why a small embedding model?

bge-small-en-v1.5 is open, runs on a laptop CPU and is widely used, so anyone can repeat this without a GPU or an API key. A larger model would change the scores, not the method.

Can I repeat it?

Yes. SciFact is public through BEIR, the model is on Hugging Face, and the commands on this page are the ones we ran. Any KiviDB database with search, including a free one, can hold the index.

Whose data is this?

SciFact (Wadden et al., 2020), distributed in BEIR (Thakur et al., 2021). We report scores only and do not republish the corpus.

Other case studies

• Semantic cache on Banking77: 13,083 support questions: model calls saved against wrong answers.
• Agent memory on LoCoMo: 1,540 questions from retrieved memories against the whole history.
• PostgreSQL to KiviDB, live: Change data capture timed from commit to readable, and a fix it led to.

Index your own documents

Create a free database, run FT.CREATE with a vector field, and query it from the Redis client you already use.