Semantic cache benchmark: hit rate vs wrong answers on Banking77
We replayed 13,083 real banking support questions through a semantic cache. At a 0.90 similarity threshold 74% were answered from the cache, and 6.3% of those answers belonged to a different question.
A semantic cache answers a question from a stored answer to one that means the same. We replayed 13,083 real customer-support questions from Banking77 through one, at eight similarity thresholds, and counted both the model calls it saved and the answers it gave to the wrong question.
At a 0.90 threshold, over 13,083 questions
- 74%: answered from the cache 9,674 of 13,083 questions found an earlier question at cosine similarity 0.90 or more, so no model call was needed.
- 6.3%: of those answers were wrong The stored question had a different intent, so its answer would not have fitted. That is 4.6% of all questions.
- 100%: agreement with an exact search Through the KiviDB Cloud API, the cache made the same hit-or-miss decision as a brute-force search on all 2,000 questions we sent it.
Embeddings: BAAI/bge-small-en-v1.5. Each intent stands in for an answer: a hit is right when the stored question has the same intent as the new one.
How we ran it
- The data Banking77: 13,083 questions that customers sent a bank's support channel, each labelled with one of 77 intents such as card_arrival or exchange_rate. We used train and test together, shuffled once (seed 7), as the stream a support bot would see.
- The cache For each question: look for the most similar question stored so far. At or above the threshold it is a hit and returns that question's answer; below it, the question is a miss and is stored with its own answer.
- Right or wrong Banking77 has no answers, so each intent stands for one. A hit is correct when the stored question's intent matches the new question's, and wrong otherwise.
- Eight thresholds From 0.70 to 0.95, replayed with an exact nearest-neighbour search over the same embeddings, to show the whole trade-off.
- The real service The first 2,000 questions at 0.90 went through the KiviDB Cloud semantic cache API on a free database, with our own vectors (provider none), to check that the service decides as the replay does.
The cache, as it filled
Each miss was stored with its question and the intent that stands for its answer. The newest entries are the questions no earlier one matched.
Semantic cache · banking77 · Entries
Every threshold, both sides
Share of all 13,083 questions. A lower threshold answers more from the cache, and more of those answers belong to another question.
| Answered from the cache | Answered wrongly | |
|---|---|---|
| 0.70 | 99.1% | 62.9% |
| 0.80 | 95.5% | 23.7% |
| 0.85 | 89.1% | 11.3% |
| 0.88 | 81.4% | 6.4% |
| 0.90 | 73.9% | 4.6% |
| 0.92 | 63.5% | 2.8% |
| 0.95 | 42.6% | 1.1% |
0.75 is in the table below. Wrong answers are a share of all questions here; the table also gives them as a share of hits.
A cache earns its rate over time
Hit rate at 0.90 over the questions seen so far. Early on the cache is nearly empty and most questions are new.
| Hit rate so far | |
|---|---|
| First 1,000 | 40.1% |
| First 2,000 | 51.0% |
| First 5,000 | 63.1% |
| First 10,000 | 71.5% |
| All 13,083 | 73.9% |
The live run through the API reached 51.0% after 2,000 questions, the same as the replay, with 981 entries stored.
Three findings worth keeping
- The threshold belongs to the model Similarity scores depend on the embedding model. With bge-small, 0.70 gave the wrong answer to 63% of hits. KiviDB's default of 0.6 is tuned for OpenAI's text-embedding-3-small; with another model, measure before you choose.
- Wrong answers never reach zero Even at 0.95, 2.5% of hits crossed intents. The commonest: 'My top-up was rejected' (top_up_failed) matched 'My top up was rejected. Why?' (top_up_reverted) at 0.976. Some of these a person would answer alike; others they would not. Keep a cache away from answers that must never be wrong.
- The rate depends on repetition Banking77 repeats a small set of needs, which is why three in four questions hit. A workload of one-off questions would hit far less. The warm-up curve shows how quickly your own traffic would get there.
The full sweep
| Threshold | From the cache | Wrong, of hits | Wrong, of all | Entries stored |
|---|---|---|---|---|
| 0.70 | 99.1% | 63.4% | 62.9% | 117 |
| 0.75 | 98.0% | 40.1% | 39.4% | 256 |
| 0.80 | 95.5% | 24.8% | 23.7% | 587 |
| 0.85 | 89.1% | 12.7% | 11.3% | 1,426 |
| 0.88 | 81.4% | 7.8% | 6.4% | 2,440 |
| 0.90 | 73.9% | 6.3% | 4.6% | 3,409 |
| 0.92 | 63.5% | 4.4% | 2.8% | 4,779 |
| 0.95 | 42.6% | 2.5% | 1.1% | 7,514 |
13,083 questions in one fixed order. Entries stored is the number of misses: what the cache holds at the end.
What a lookup costs
- 32.5 ms: median lookup, in the API Measured by the KiviDB Cloud API for each lookup: expired entries cleared, the nearest stored question found, its answer read and its hit counted. 95th percentile 45 ms.
- 34 ms: median for a hit A hit reads the stored answer and counts the hit, so it does slightly more work than a miss (30 ms).
- 981: entries after 2,000 questions The cache holds only the questions that missed. Over all 13,083 at 0.90 it would hold 3,409.
Free-tier database (shared host) in AWS eu-central-1, timed inside the API in the same region. Your network time to the API comes on top.
Questions
Why count a different intent as a wrong answer?
Because in a support bot each intent gets its own reply. If 'Where is my card?' is answered with the reply stored for 'How do I activate my card?', the customer got the wrong answer even though the questions look alike.
Does this say how much money a cache saves?
It says how many model calls are avoided: 74 in every 100 at 0.90 on this traffic. Multiply by what one call costs you, and subtract what a wrong answer costs you, which only you can price.
Why our own embeddings and not a hosted provider?
So the study runs without an API key and anyone can repeat it. The cache also works with OpenAI or any OpenAI-compatible embedding endpoint, which embeds the prompt for you.
Whose data is this?
Banking77 (Casanueva et al., 2020), published by PolyAI under CC BY 4.0.
Other case studies
- RAG retrieval on SciFact: Keyword, vector and hybrid search scored on 300 scientific claims.
- Agent memory on LoCoMo: 1,540 questions from retrieved memories against the whole history.
- PostgreSQL to KiviDB, live: Change data capture timed from commit to readable, and a fix it led to.
Put a cache in front of your model
Create a free database, add a semantic cache namespace, and measure the threshold on your own questions.