Semantic Caching for LLM Apps with a Redis-Compatible Vector Database

Murtaza Shajapurwala ·

Every LLM app eventually hits the same wall. The model call is the slowest and most expensive part of the request, and a large share of the prompts coming in are the same question asked slightly differently. "How do I reset my password?", "forgot my password, what now", and "password reset steps" all deserve the same answer, but a normal cache treats them as three different keys and pays for three model calls.

Semantic caching fixes that. Instead of matching prompts character for character, you match them by meaning, using vector search. This post walks through how a semantic cache works, how to build one on a Redis-compatible database with a few dozen lines of Python, and the details that decide whether it helps or quietly serves wrong answers: similarity thresholds, scoping, and invalidation.

What is semantic caching?

A semantic cache stores LLM responses keyed by the embedding of the prompt rather than the prompt text. When a new prompt arrives, you embed it, look for the nearest previously answered prompt in vector space, and if it's close enough, return the stored answer without calling the model.

The flow looks like this:

  1. Embed the incoming prompt with an embedding model.
  2. Run a nearest-neighbour search against the cache's vector index.
  3. If the closest match is within your similarity threshold, return its cached response. That's a cache hit.
  4. Otherwise call the LLM, return the answer, and store the prompt embedding with the response so the next similar question hits.

Flow of a semantic cache: prompt, embed, KNN lookup, then either a hit that returns the cached answer or a miss that calls the LLM and stores the result with a TTL

An embedding call costs a small fraction of a completion and usually returns in tens of milliseconds, so a hit replaces a multi-second, expensive generation with an embedding call plus a vector lookup.

Why exact-match caching falls short for LLM apps

Exact-match caching (hash the prompt, GET the hash) is still worth doing, because it's nearly free. But in real traffic it hits rarely. People rephrase, add typos, change word order, and wrap the same question in different context. Any of those changes the hash.

Semantic caching catches the paraphrases. The trade-off is that "similar" is a judgement call, and a cache that is too generous will return an answer to a question that wasn't asked. Most of the engineering effort in a semantic cache goes into drawing that line correctly.

Why a Redis-compatible database is a good fit

You can build a semantic cache on a dedicated vector database, but most teams already run Redis for sessions, rate limits, and ordinary caching. A cache is a Redis-shaped problem: you want hashes for the payload, TTLs for expiry, and low-latency reads. The only missing piece has traditionally been the vector index.

KiviDB is an in-memory database written in Rust and compatible with the Redis protocol, so redis-py, ioredis, go-redis and the rest work unchanged. It implements the RediSearch-style FT.* commands with VECTOR fields (HNSW and FLAT), KNN queries, and metadata pre-filtering on TAG and NUMERIC fields, alongside the rest of the Redis command surface. That means the cache entries, their TTLs, and the vector index all live in one place, reached with one client you already have.

Two properties matter specifically for caching:

  • New entries are searchable as soon as they're written. KiviDB indexes vectors on write rather than in a background job. For a cache, that means the answer you just paid for can serve the very next paraphrase, instead of waiting for an indexer to catch up.
  • Reads scale across cores. KiviDB uses a shard-per-core architecture, so lookup throughput grows with the machine rather than being capped by a single thread.

Build a semantic cache in Python

The example below uses redis-py, numpy, and the OpenAI SDK for embeddings and completions. Any embedding model works, as long as you use the same one for writes and reads and set DIM to its output size.

pip install redis numpy openai

1. Connect and create the vector index

import hashlib

import numpy as np
import redis
from openai import OpenAI
from redis.commands.search.query import Query

# Self-hosted KiviDB listens on 6380 by default; KiviDB Cloud gives you a full endpoint URL.
r = redis.Redis(host="localhost", port=6380, decode_responses=True)
oai = OpenAI()

INDEX = "llmcache"
PREFIX = "llmcache:"
DIM = 1536              # text-embedding-3-small
THRESHOLD = 0.10        # max cosine distance for a hit; tune this (see below)
TTL_SECONDS = 24 * 3600


def ensure_index() -> None:
    try:
        r.ft(INDEX).info()
    except redis.ResponseError:
        r.execute_command(
            "FT.CREATE", INDEX,
            "ON", "HASH",
            "PREFIX", "1", PREFIX,
            "SCHEMA",
            "scope", "TAG",
            "embedding", "VECTOR", "HNSW", "6",
            "TYPE", "FLOAT32",
            "DIM", str(DIM),
            "DISTANCE_METRIC", "COSINE",
        )

Only two fields are indexed. embedding is the prompt vector. scope is a tag used to keep cache entries from leaking across contexts where the same question should get different answers; more on that below. The prompt text and response are stored in the same hash but don't need to be indexed.

Vectors are stored as raw FLOAT32 bytes, so always convert embeddings with dtype=np.float32 before writing or querying.

2. Embed prompts

def embed(text: str) -> np.ndarray:
    resp = oai.embeddings.create(model="text-embedding-3-small", input=text)
    return np.asarray(resp.data[0].embedding, dtype=np.float32)

3. Look up the nearest cached prompt

def lookup(vec: np.ndarray, scope: str) -> str | None:
    q = (
        Query(f"(@scope:{{{scope}}})=>[KNN 1 @embedding $vec AS dist]")
        .sort_by("dist")
        .return_fields("response", "dist")
        .dialect(2)
    )
    res = r.ft(INDEX).search(q, query_params={"vec": vec.tobytes()})
    if res.docs and float(res.docs[0].dist) <= THRESHOLD:
        return res.docs[0].response
    return None

The query filters to the current scope first, then finds the single nearest neighbour. With cosine distance, 0 means identical direction and larger means less similar, so a hit is anything at or below the threshold.

4. Store new answers

def store(prompt: str, response: str, vec: np.ndarray, scope: str) -> None:
    key = PREFIX + hashlib.sha256(f"{scope}\x00{prompt}".encode()).hexdigest()
    pipe = r.pipeline()
    pipe.hset(key, mapping={
        "scope": scope,
        "prompt": prompt,
        "response": response,
        "embedding": vec.tobytes(),
    })
    pipe.expire(key, TTL_SECONDS)
    pipe.execute()

Hashing the scope and prompt into the key means an exact repeat overwrites its own entry instead of piling up duplicates.

5. Wrap the model call

def cached_completion(prompt: str, scope: str, model: str = "gpt-4o-mini") -> str:
    vec = embed(prompt)

    hit = lookup(vec, scope)
    if hit is not None:
        return hit

    out = oai.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    answer = out.choices[0].message.content
    store(prompt, answer, vec, scope)
    return answer


ensure_index()
print(cached_completion("How do I reset my password?", scope="support_v1"))
print(cached_completion("forgot my password, what do I do", scope="support_v1"))  # likely a hit

That's a working semantic cache. The rest of this post is about making it safe.

Choosing a similarity threshold

A new prompt at the centre of a dashed threshold circle; nearby cached prompts inside the circle are hits, farther ones outside are misses

The threshold is the most important number in the system, and there's no universal right value. It depends on your embedding model, your domain, and how costly a wrong answer is compared to a missed hit.

A practical way to set it:

  1. Collect a few hundred real prompt pairs from your logs and label each as "same answer" or "different answer".
  2. Embed them and compute the cosine distance for each pair.
  3. Pick the largest threshold that keeps false positives (different-answer pairs under the threshold) at a level you can live with.

Start conservative. A missed hit costs you one model call; a false hit serves a confident wrong answer. Pairs like "cancel my order" and "don't cancel my order" can sit surprisingly close in embedding space, which is why the threshold should be validated on your own data rather than copied from a blog post, this one included.

Scope entries so answers don't leak

Two identical questions can deserve different answers. Build the scope tag from everything that changes the correct response:

  • Model and prompt template version. When you change the system prompt or switch models, bump the version so old answers stop matching.
  • Tenant or user, for anything personalised or permission-dependent.
  • Language or locale, if you serve more than one.

A scope like support_v3_en or tenant_4821_v2 keeps these apart at query time. Because the filter is applied before the nearest-neighbour search, a scoped lookup only ever considers entries in that scope.

Keep tag values to letters, digits, and underscores. Characters like - and . have special meaning in the query syntax and need escaping.

Invalidation and TTLs

Cached LLM answers go stale when the underlying facts change. Three tools cover most cases:

  • TTLs on every entry, sized to how fast your content changes. Hours for product and pricing questions, days or weeks for stable reference material.
  • Version bumps in the scope for deploys that change prompts, models, or retrieval sources. Old entries simply stop matching and age out through their TTL.
  • Targeted deletes when you know a specific answer is wrong. Because entries are ordinary hashes, a DEL or a scan over the prefix removes them.

What not to cache

Semantic caching works best for informational, repeatable questions. Skip it for:

  • Responses that depend on live data such as account balances, order status, or anything fetched by a tool call.
  • Multi-turn conversations where the right answer depends on earlier turns. If you do cache these, include a summary or hash of the conversation state in the scope.
  • Creative generation where users expect a different output each time.

Why write-to-queryable time matters for a cache

In a semantic cache, every miss becomes a write, and the value of that write depends on how quickly it becomes searchable. If the index lags behind writes, a burst of similar questions (after a launch, an outage, or a viral post) all miss and all pay for generation, which is exactly when the cache should be helping most.

In our vector benchmark on AWS Graviton3 (c7gn.12xlarge) using the glove-25-angular dataset of 1,183,514 vectors, KiviDB v1.0.3 ingested at about 5,262 vectors per second against about 2,606 for Redis Stack 7.4.7. The full dataset was queryable in 225 seconds on KiviDB and 455 seconds on Redis Stack. The full recall and latency breakdown is on the KiviDB vector database page, and the benchmark harness is public at github.com/kividbio/kivi-benchmark, so you can reproduce the numbers on your own hardware.

Measuring whether it's working

Track four numbers from day one:

  • Hit rate, overall and per scope. This tells you what the cache is saving.
  • Distance distribution of hits. If most hits sit right at the threshold, you're close to the edge and should review samples.
  • Sampled hit quality. Periodically log a sample of hits with the original prompt, the matched prompt, and the response, and have someone check them.
  • End-to-end latency for hits and misses separately, so the embedding call and lookup don't quietly erode the gains.

FAQ

What is the difference between semantic caching and regular caching?

Regular caching returns a stored value only when the key matches exactly. Semantic caching compares prompt embeddings and returns a stored response when a new prompt is close enough in meaning, so paraphrased questions can hit the cache.

Can I build a semantic cache on Redis?

Yes, if your deployment supports vector indexes through the FT.* search commands. KiviDB supports them natively, along with the standard Redis data types and TTLs, so the cache payload and vector index live in the same database and use the same client.

What similarity threshold should I use for a semantic cache?

It depends on your embedding model and domain. Start strict, measure false positives on labelled prompt pairs from your own traffic, and loosen only while the error rate stays acceptable.

Does semantic caching work with any embedding model?

Yes. Use the same model for storing and querying, set the index dimension to match its output, and store vectors as float32. If you change embedding models, rebuild the cache or use a new scope, since vectors from different models aren't comparable.

How much does semantic caching save?

It depends almost entirely on how repetitive your traffic is. Support bots, documentation assistants, and internal FAQ tools tend to see heavy repetition. Open-ended chat sees much less. Log your prompts for a week and measure how many cluster together before committing to a target.

Try it

KiviDB is compatible with the Redis protocol, so the code above runs against it without changes. Point your existing Redis client at a KiviDB instance, create the index, and wrap your model calls. You can get started at kividb.io.