LiteLLM with KiviDB: A Redis-Compatible Backend for Caching, Routing and Semantic Cache
Murtaza Shajapurwala ·
LiteLLM is one of the most widely used LLM gateways: one OpenAI-style API in front of more than a hundred model providers. Behind it, Redis does three jobs. It holds the response cache, it keeps the TPM/RPM counters that usage-based routing reads, and it powers the semantic cache that answers reworded prompts without paying for another model call.
KiviDB is a Redis-compatible database with search and vector similarity built in. So we asked a simple question: can KiviDB be LiteLLM's Redis? We tested it the way LiteLLM uses Redis, with Redis Stack running the same tests alongside it.
The result: KiviDB 1.0.5 passed all 17 checks, with the same results as Redis Stack, and LiteLLM needed no code changes.
How we tested
We wrote a harness that runs every Redis feature LiteLLM depends on and records PASS or FAIL. A local stub stands in for the LLM provider, so the tests need no API keys and cost nothing to run.
| Component | Version |
|---|---|
| LiteLLM | 1.103.2 |
| RedisVL (semantic cache) | 0.27.2 |
| redis-py | 8.1.0 |
| KiviDB | 1.0.5, official release binary |
| Redis Stack (reference) | 7.4.7 with RediSearch 2.10.20 |
Both databases ran on the same machine with the same harness. The checks cover three layers:
- Redis primitives LiteLLM relies on, from
SET EXto Lua scripts. - The RediSearch vector API:
FT.CREATE, KNN search with a TAG pre-filter,FT.INFO,FT.DROPINDEX. - LiteLLM end to end: the exact cache, router counters and the semantic cache, driven through LiteLLM itself.
Results: KiviDB 1.0.5 vs Redis Stack
| Check | Redis Stack | KiviDB 1.0.5 |
|---|---|---|
PING | PASS | PASS |
SET EX / GET / TTL | PASS | PASS |
INCR / EXPIRE / MGET | PASS | PASS |
MULTI / EXEC pipeline | PASS | PASS |
SCAN MATCH | PASS | PASS |
SET NX EX (locks) | PASS | PASS |
HSET / HGETALL | PASS | PASS |
RPUSH / LRANGE / LPOP | PASS | PASS |
EVAL + EVALSHA (Lua) | PASS | PASS |
INFO | PASS | PASS |
FT.CREATE (HNSW + TAG, cosine) | PASS | PASS |
FT.SEARCH KNN with TAG pre-filter | PASS | PASS |
FT.INFO | PASS | PASS |
FT.DROPINDEX | PASS | PASS |
LiteLLM exact cache (type: redis) | PASS | PASS |
| LiteLLM router TPM/RPM counters | PASS | PASS |
LiteLLM semantic cache (type: redis-semantic) | PASS | PASS |
| Total | 17 / 17 | 17 / 17 |
We ran the full suite against KiviDB four times, including back-to-back runs. Every run passed every check.
What each LiteLLM check proves:
- Exact cache: the same prompt sent twice reached the provider once. The second answer came from KiviDB.
- Router counters: three routed calls raised the deployment's RPM counter by exactly 3, and its TPM counter grew with them.
- Semantic cache: a reworded prompt ("how can I reset my password" after "how do I reset my password") was served from the cache, and an unrelated prompt ("what is the capital of france") went to the model.
Why KiviDB, if Redis already works?
Compatibility is the entry ticket. These are the reasons to switch.
1. Throughput headroom on a single node
A gateway touches Redis on almost every request: a cache lookup, router counter updates, and for semantic caching a vector search. Redis executes commands on one thread. KiviDB is multi-threaded and spreads work across all cores, so one node absorbs far more traffic before you reach for sharding or a cluster.
KiviDB's published 1.0.5 benchmark, run on 3 October 2026:
| Pipelined workload | KiviDB 1.0.5 | Redis 8.6 | KiviDB vs Redis |
|---|---|---|---|
| Write | ~14.7M ops/s | ~750K ops/s | ~19.6× |
| Read | ~17.7M ops/s | ~974K ops/s | ~18.2× |
| Mixed 1:1 | ~15.9M ops/s | ~913K ops/s | ~17.4× |
memtier_benchmark, AWS Graviton3 c7gn.12xlarge (48 vCPU) server, client on a separate c7gn.16xlarge in the same AZ, plaintext connections, -c 5 --pipeline=10. Over TLS the same build measured about 1.1M ops/s per workload in that run. Methodology and scripts: kivi-benchmark. This is a general key-value benchmark, not a LiteLLM workload; measure your own traffic before sizing.
2. A drop-in, not a migration
LiteLLM's standard redis and redis-semantic settings work unchanged: point the host and port at KiviDB. KiviDB 1.0.5 tracks Redis 8.10 commands, replies and error messages, and in our tests it returned the same results as Redis Stack down to the vector distances.
3. Search in the same binary
The semantic cache needs RediSearch's FT.* commands. In KiviDB they're part of the database itself, so there's no module to load and no separate Redis Stack image to build, patch and upgrade. The same database holds your cache, your routing counters and your vectors.
4. Run it the way you run everything else
Self-host the binary or container, deploy with the Kubernetes operator, manage databases with the Terraform provider, or use KiviDB Cloud on AWS, Azure and Google Cloud.
Semantic caching with RedisVL on KiviDB
The semantic cache is the demanding part. LiteLLM hands it to RedisVL, and RedisVL speaks RediSearch: it creates an index with FT.CREATE and looks up similar prompts with a vector range query.
FT.CREATE litellm_semantic_cache_index ON HASH PREFIX 1 litellm_semantic_cache_index SCORE 1.0 SCHEMA
prompt TEXT WEIGHT 1 response TEXT WEIGHT 1 inserted_at NUMERIC updated_at NUMERIC
prompt_vector VECTOR FLAT 6 TYPE FLOAT32 DIM 256 DISTANCE_METRIC COSINE
litellm_cache_key TAG SEPARATOR ,
FT.SEARCH litellm_semantic_cache_index
"@prompt_vector:[VECTOR_RANGE $distance_threshold $vector]=>{$YIELD_DISTANCE_AS: vector_distance} (@litellm_cache_key:{…})"
RETURN 8 entry_id prompt response inserted_at updated_at metadata litellm_cache_key vector_distance
SORTBY vector_distance ASC DIALECT 2 LIMIT 0 1
KiviDB runs these natively. We also compared the details a client can see, side by side with Redis Stack:
| Behaviour | Redis Stack | KiviDB 1.0.5 |
|---|---|---|
RedisVL's FT.CREATE schema (WEIGHT, SEPARATOR) | OK | OK |
VECTOR_RANGE distance for the same vectors | 0.00496286153793 | 0.0049628615 |
| KNN distance for the same vectors | 0.00496286153793 | 0.0049628615 |
RETURN / NOCONTENT | fields as asked | fields as asked |
Negated and OR pre-filters | correct matches | correct matches |
FT.SEARCH reply under RESP3 (redis-py 8 default) | map | same map |
| Malformed query | Syntax error at offset … | Syntax error at offset … |
| Unknown argument | error | error |
The only differences we found were cosmetic: KiviDB prints distances to 10 significant digits instead of 15, and its error messages name the offending token.
Configure LiteLLM to use KiviDB
Point LiteLLM's Redis settings at KiviDB. This is the configuration our harness ran, through the LiteLLM Python SDK:
import litellm
from litellm import Router
from litellm.caching.caching import Cache
# Exact-match response cache
litellm.cache = Cache(type="redis", host="localhost", port="6380", ttl=60)
# Semantic cache (RedisVL under the hood)
litellm.cache = Cache(
type="redis-semantic",
redis_url="redis://localhost:6380",
similarity_threshold=0.8,
ttl=60,
redis_semantic_cache_embedding_model="text-embedding-ada-002",
)
# Router with usage-based routing; TPM/RPM counters live in KiviDB
router = Router(
model_list=[{"model_name": "m", "litellm_params": {"model": "openai/gpt-4o-mini"}}],
redis_host="localhost",
redis_port=6380,
routing_strategy="usage-based-routing-v2",
)
Running the LiteLLM proxy instead? The same settings go under cache_params in config.yaml:
litellm_settings:
cache: true
cache_params:
type: redis-semantic
redis_url: redis://localhost:6380
similarity_threshold: 0.8
redis_semantic_cache_embedding_model: text-embedding-ada-002
Use the -lua or -full KiviDB build if you want Lua scripting (EVAL / EVALSHA); our harness tested the -full build.
Tips for testing any LiteLLM Redis backend
- Check the logs once. LiteLLM logs semantic-cache errors and keeps serving, so a misconfigured cache looks like a cache that never hits. Look for
Redis semantic-cache index build failed. - Give each test run its own semantic-cache index with
redis_semantic_cache_index_name. The default index is shared, and entries left by a run less thanttlseconds earlier can make an unrelated prompt hit. - Expect RESP3 with redis-py 8.
FT.SEARCHreplies arrive as a map instead of an array, on Redis Stack and on KiviDB alike.
FAQ
Does LiteLLM work with KiviDB? Yes. In our tests, LiteLLM 1.103.2's exact cache, usage-based routing counters and RedisVL semantic cache all worked on KiviDB 1.0.5, with the same results as Redis Stack.
Why use KiviDB instead of Redis for LiteLLM? For throughput headroom on a single node: KiviDB is multi-threaded, and in KiviDB's published benchmark it handled roughly 17 to 20 times Redis 8.6's pipelined throughput on the same hardware. You get that without changing LiteLLM's configuration beyond the host and port, with search built into the same database.
Do I need RediSearch or Redis Stack for LiteLLM's semantic cache?
You need a server that supports RediSearch's FT.* commands. KiviDB has them built in, so there's no module to install.
Do I need to change LiteLLM's code?
No. Use the standard redis and redis-semantic cache types and point them at KiviDB's host and port.
What did you not test? We drove LiteLLM through its Python SDK, not the proxy process, and tested synchronous calls only. The compatibility tests ran on one small machine, so the throughput figures above come from KiviDB's separate benchmark setup, not from this test.
Try it
- Download KiviDB 1.0.5 and point the configuration above at it.
- Or start a database on KiviDB Cloud and use its endpoint.
- New to semantic caching? Read Semantic Caching for LLM Apps with a Redis-Compatible Vector Database.