LiteLLM with KiviDB: A Redis-Compatible Backend for Caching, Routing and Semantic Cache

Murtaza Shajapurwala ·

LiteLLM is one of the most widely used LLM gateways: one OpenAI-style API in front of more than a hundred model providers. Behind it, Redis does three jobs. It holds the response cache, it keeps the TPM/RPM counters that usage-based routing reads, and it powers the semantic cache that answers reworded prompts without paying for another model call.

KiviDB is a Redis-compatible database with search and vector similarity built in. So we asked a simple question: can KiviDB be LiteLLM's Redis? We tested it the way LiteLLM uses Redis, with Redis Stack running the same tests alongside it.

The result: KiviDB 1.0.5 passed all 17 checks, with the same results as Redis Stack, and LiteLLM needed no code changes.

Table of LiteLLM features on KiviDB 1.0.5 and Redis Stack: exact response cache, router TPM/RPM counters and RedisVL semantic cache all PASS on both. 17 of 17 checks on each, no LiteLLM code changes.

How we tested

We wrote a harness that runs every Redis feature LiteLLM depends on and records PASS or FAIL. A local stub stands in for the LLM provider, so the tests need no API keys and cost nothing to run.

ComponentVersion
LiteLLM1.103.2
RedisVL (semantic cache)0.27.2
redis-py8.1.0
KiviDB1.0.5, official release binary
Redis Stack (reference)7.4.7 with RediSearch 2.10.20

Both databases ran on the same machine with the same harness. The checks cover three layers:

  1. Redis primitives LiteLLM relies on, from SET EX to Lua scripts.
  2. The RediSearch vector API: FT.CREATE, KNN search with a TAG pre-filter, FT.INFO, FT.DROPINDEX.
  3. LiteLLM end to end: the exact cache, router counters and the semantic cache, driven through LiteLLM itself.

Results: KiviDB 1.0.5 vs Redis Stack

CheckRedis StackKiviDB 1.0.5
PINGPASSPASS
SET EX / GET / TTLPASSPASS
INCR / EXPIRE / MGETPASSPASS
MULTI / EXEC pipelinePASSPASS
SCAN MATCHPASSPASS
SET NX EX (locks)PASSPASS
HSET / HGETALLPASSPASS
RPUSH / LRANGE / LPOPPASSPASS
EVAL + EVALSHA (Lua)PASSPASS
INFOPASSPASS
FT.CREATE (HNSW + TAG, cosine)PASSPASS
FT.SEARCH KNN with TAG pre-filterPASSPASS
FT.INFOPASSPASS
FT.DROPINDEXPASSPASS
LiteLLM exact cache (type: redis)PASSPASS
LiteLLM router TPM/RPM countersPASSPASS
LiteLLM semantic cache (type: redis-semantic)PASSPASS
Total17 / 1717 / 17

We ran the full suite against KiviDB four times, including back-to-back runs. Every run passed every check.

What each LiteLLM check proves:

  • Exact cache: the same prompt sent twice reached the provider once. The second answer came from KiviDB.
  • Router counters: three routed calls raised the deployment's RPM counter by exactly 3, and its TPM counter grew with them.
  • Semantic cache: a reworded prompt ("how can I reset my password" after "how do I reset my password") was served from the cache, and an unrelated prompt ("what is the capital of france") went to the model.

Why KiviDB, if Redis already works?

Compatibility is the entry ticket. These are the reasons to switch.

1. Throughput headroom on a single node

A gateway touches Redis on almost every request: a cache lookup, router counter updates, and for semantic caching a vector search. Redis executes commands on one thread. KiviDB is multi-threaded and spreads work across all cores, so one node absorbs far more traffic before you reach for sharding or a cluster.

KiviDB's published 1.0.5 benchmark, run on 3 October 2026:

Pipelined workloadKiviDB 1.0.5Redis 8.6KiviDB vs Redis
Write~14.7M ops/s~750K ops/s~19.6×
Read~17.7M ops/s~974K ops/s~18.2×
Mixed 1:1~15.9M ops/s~913K ops/s~17.4×

memtier_benchmark, AWS Graviton3 c7gn.12xlarge (48 vCPU) server, client on a separate c7gn.16xlarge in the same AZ, plaintext connections, -c 5 --pipeline=10. Over TLS the same build measured about 1.1M ops/s per workload in that run. Methodology and scripts: kivi-benchmark. This is a general key-value benchmark, not a LiteLLM workload; measure your own traffic before sizing.

2. A drop-in, not a migration

LiteLLM's standard redis and redis-semantic settings work unchanged: point the host and port at KiviDB. KiviDB 1.0.5 tracks Redis 8.10 commands, replies and error messages, and in our tests it returned the same results as Redis Stack down to the vector distances.

3. Search in the same binary

The semantic cache needs RediSearch's FT.* commands. In KiviDB they're part of the database itself, so there's no module to load and no separate Redis Stack image to build, patch and upgrade. The same database holds your cache, your routing counters and your vectors.

4. Run it the way you run everything else

Self-host the binary or container, deploy with the Kubernetes operator, manage databases with the Terraform provider, or use KiviDB Cloud on AWS, Azure and Google Cloud.

Semantic caching with RedisVL on KiviDB

The semantic cache is the demanding part. LiteLLM hands it to RedisVL, and RedisVL speaks RediSearch: it creates an index with FT.CREATE and looks up similar prompts with a vector range query.

FT.CREATE litellm_semantic_cache_index ON HASH PREFIX 1 litellm_semantic_cache_index SCORE 1.0 SCHEMA
  prompt TEXT WEIGHT 1 response TEXT WEIGHT 1 inserted_at NUMERIC updated_at NUMERIC
  prompt_vector VECTOR FLAT 6 TYPE FLOAT32 DIM 256 DISTANCE_METRIC COSINE
  litellm_cache_key TAG SEPARATOR ,

FT.SEARCH litellm_semantic_cache_index
  "@prompt_vector:[VECTOR_RANGE $distance_threshold $vector]=>{$YIELD_DISTANCE_AS: vector_distance} (@litellm_cache_key:{…})"
  RETURN 8 entry_id prompt response inserted_at updated_at metadata litellm_cache_key vector_distance
  SORTBY vector_distance ASC DIALECT 2 LIMIT 0 1

KiviDB runs these natively. We also compared the details a client can see, side by side with Redis Stack:

BehaviourRedis StackKiviDB 1.0.5
RedisVL's FT.CREATE schema (WEIGHT, SEPARATOR)OKOK
VECTOR_RANGE distance for the same vectors0.004962861537930.0049628615
KNN distance for the same vectors0.004962861537930.0049628615
RETURN / NOCONTENTfields as askedfields as asked
Negated and OR pre-filterscorrect matchescorrect matches
FT.SEARCH reply under RESP3 (redis-py 8 default)mapsame map
Malformed querySyntax error at offset …Syntax error at offset …
Unknown argumenterrorerror

The only differences we found were cosmetic: KiviDB prints distances to 10 significant digits instead of 15, and its error messages name the offending token.

Configure LiteLLM to use KiviDB

Point LiteLLM's Redis settings at KiviDB. This is the configuration our harness ran, through the LiteLLM Python SDK:

import litellm
from litellm import Router
from litellm.caching.caching import Cache

# Exact-match response cache
litellm.cache = Cache(type="redis", host="localhost", port="6380", ttl=60)

# Semantic cache (RedisVL under the hood)
litellm.cache = Cache(
    type="redis-semantic",
    redis_url="redis://localhost:6380",
    similarity_threshold=0.8,
    ttl=60,
    redis_semantic_cache_embedding_model="text-embedding-ada-002",
)

# Router with usage-based routing; TPM/RPM counters live in KiviDB
router = Router(
    model_list=[{"model_name": "m", "litellm_params": {"model": "openai/gpt-4o-mini"}}],
    redis_host="localhost",
    redis_port=6380,
    routing_strategy="usage-based-routing-v2",
)

Running the LiteLLM proxy instead? The same settings go under cache_params in config.yaml:

litellm_settings:
  cache: true
  cache_params:
    type: redis-semantic
    redis_url: redis://localhost:6380
    similarity_threshold: 0.8
    redis_semantic_cache_embedding_model: text-embedding-ada-002

Use the -lua or -full KiviDB build if you want Lua scripting (EVAL / EVALSHA); our harness tested the -full build.

Tips for testing any LiteLLM Redis backend

  • Check the logs once. LiteLLM logs semantic-cache errors and keeps serving, so a misconfigured cache looks like a cache that never hits. Look for Redis semantic-cache index build failed.
  • Give each test run its own semantic-cache index with redis_semantic_cache_index_name. The default index is shared, and entries left by a run less than ttl seconds earlier can make an unrelated prompt hit.
  • Expect RESP3 with redis-py 8. FT.SEARCH replies arrive as a map instead of an array, on Redis Stack and on KiviDB alike.

FAQ

Does LiteLLM work with KiviDB? Yes. In our tests, LiteLLM 1.103.2's exact cache, usage-based routing counters and RedisVL semantic cache all worked on KiviDB 1.0.5, with the same results as Redis Stack.

Why use KiviDB instead of Redis for LiteLLM? For throughput headroom on a single node: KiviDB is multi-threaded, and in KiviDB's published benchmark it handled roughly 17 to 20 times Redis 8.6's pipelined throughput on the same hardware. You get that without changing LiteLLM's configuration beyond the host and port, with search built into the same database.

Do I need RediSearch or Redis Stack for LiteLLM's semantic cache? You need a server that supports RediSearch's FT.* commands. KiviDB has them built in, so there's no module to install.

Do I need to change LiteLLM's code? No. Use the standard redis and redis-semantic cache types and point them at KiviDB's host and port.

What did you not test? We drove LiteLLM through its Python SDK, not the proxy process, and tested synchronous calls only. The compatibility tests ran on one small machine, so the throughput figures above come from KiviDB's separate benchmark setup, not from this test.

Try it