Benchmarking KiviDB
This is the runbook from github.com/kividbio/kivi-benchmark, where the Terraform and the scripts live. It aligns instance types and topology with the public…
This is the runbook from github.com/kividbio/kivi-benchmark, where the Terraform and the scripts live. It aligns instance types and topology with the public Dragonfly c7gn numbers: Dragonfly on c7gn.12xlarge (48 vCPU) and memtier_benchmark on a separate c7gn.16xlarge in the same Availability Zone. Infrastructure is provisioned with the Terraform in this repository; the engines are started and the runs made with the scripts under scripts/.
Results: KiviDB 1.1.0 vs Dragonfly 2.0.2 vs Redis 8.10
These are the numbers on the Benchmarks page and in the Performance section of the KiviDB README, which is the canonical copy.
Setup:
- Server: c7gn.metal (64 vCPU, Graviton3), each engine confined to 32 cores (
CORES=32, so 32 threads). Load generator: a second c7gn.metal in the same placement group. (This runbook's Terraform defaults to c7gn.12xlarge and c7gn.16xlarge; setserver_instance_typeandclient_instance_typetoc7gn.metalto repeat this capture.) - One engine at a time, in one session, with the same
memtier_benchmarkparameters. run-matrix.sh: 10 million keys of 100 bytes,DURATION=30seconds a scenario, the mean of two rounds; 300 connections with pipeline 16 (-t 60 -c 5), 1,200 without pipelining (-t 60 -c 20).
| ops/s | KiviDB 1.1.0 | Dragonfly 2.0.2 | Redis 8.10 | Redis Cluster (32 shards) |
|---|---|---|---|---|
| GET, pipeline 16 | 21.2M | 9.1M | 2.4M | 4.7M |
| SET, pipeline 16 | 19.2M | 8.5M | 2.0M | 4.7M |
| GET, no pipelining | 3.07M | 3.35M | 1.32M | 1.46M |
| SET, no pipelining | 3.12M | 3.38M | 1.54M | 1.46M |
| p99 latency, GET / SET, pipeline 16 | 0.36 / 0.42 ms | 0.88 / 1.07 ms | 2.2 / 2.7 ms | |
| Memory per key | 177 B | 134 B | 178 B | 183 B |
How to read it: with pipelining KiviDB served more than twice Dragonfly's operations and about nine times a single Redis; one request at a time per connection, Dragonfly was 8 to 9% ahead; Dragonfly used the least memory per key.
The exact commands are in section 5.
Dragonfly-published reference (c7gn, memtier defaults unless noted)
| Test | Ops/sec (approx.) | Avg. latency (µs) | P99.9 (µs) |
|---|---|---|---|
Write-only (ratio 1:0, -t 60 -c 20 -n 200000) | ~5.2M | ~250 | ~631 |
Read-only (ratio 0:1, same) | ~6M | ~271 | ~623 |
Pipelined read (-c 5, --pipeline=10) | ~8.9M | ~323 | ~839 |
Note: For the same pipelined-read command, Dragonfly publishes ~8.9M ops/sec. An earlier capture on this runbook's default c7gn.12xlarge measured Dragonfly v1.37.0 at 8.8M ops/sec, so our Dragonfly runs match its own published figures. We benchmark all three stores in the same session on the same instance, so the comparison is fair. Expect 5–15% run-to-run variance from AWS capacity and NIC state. Treat every published figure, including Dragonfly's own, as one controlled capture, not a universal guarantee.
Commands from their write-up:
# Writes
memtier_benchmark -s $SERVER_PRIVATE_IP --distinct-client-seed --hide-histogram --ratio 1:0 -t 60 -c 20 -n 200000
# Reads
memtier_benchmark -s $SERVER_PRIVATE_IP --distinct-client-seed --hide-histogram --ratio 0:1 -t 60 -c 20 -n 200000
# Pipelined reads
memtier_benchmark -s $SERVER_PRIVATE_IP --ratio 0:1 -t 60 -c 5 -n 200000 --distinct-client-seed --hide-histogram --pipeline=10
1. Prerequisites
- AWS account with service quotas allowing c7gn.12xlarge and c7gn.16xlarge in the chosen region.
- An EC2 key pair in that region (for SSH).
- Terraform
>= 1.3, AWS CLI configured (aws configureor environment variables).
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
# Optional: AWS_SESSION_TOKEN for assumed roles
export AWS_DEFAULT_REGION="us-east-1"
2. Provision instances
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars: key_name, ssh_cidr (recommended: your /32), region.
terraform init
terraform plan
terraform apply
Put key_name (and other variables) in terraform.tfvars so terraform apply does not prompt interactively.
If the server never appears in the EC2 console
-
Canceled apply — If you press Ctrl+C while Terraform says
aws_instance.server: Creating..., AWS may never finish creating the instance. Runterraform applyagain and leave it running. Large c7gn instances can stay inpendingfor several minutes; "Still creating… 1m0s" is normal. -
Stale plan — If you only see the client, the server apply likely never completed.
terraform state listshould showaws_instance.serveronly after a successful apply. If it is missing, runterraform applyagain. -
Insufficient capacity — The console error "We currently do not have sufficient c7gn.12xlarge capacity in the Availability Zone you requested" is the definitive explanation. Fix: set
availability_zoneinterraform.tfvarsto an AZ AWS lists (e.g. in us-east-1:us-east-1a,us-east-1b,us-east-1d,us-east-1fwhen us-east-1c fails), or setsubnet_idto a subnet in a good AZ. After changing AZ, runterraform applyso both instances use the same subnet/AZ. -
Placement group — If apply fails or hangs unusually long, try
use_placement_group = falseinterraform.tfvars, thenterraform applyagain.
Note outputs:
server_private_ip→ use asSERVERfor memtier from the client.server_public_ip/client_public_ip→ SSH access.availability_zone/subnet_id→ confirm you are in an AZ with c7gn capacity.
Bootstrap logs:
- Server:
/var/log/user-data-server.log - Client:
/var/log/user-data-client.log
Wait until cloud-init finishes on both (cloud-init status --wait). The client builds memtier from source, which takes a few minutes.
3. The server instance (c7gn.12xlarge)
user_data/server.sh installs the three engines at pinned versions and starts
none of them. Its log, /var/log/user-data-server.log, ends with the version
of each.
| Engine | Version | Source |
|---|---|---|
| KiviDB | v1.0.5, default build | official release binary |
| Dragonfly | v1.37.0 | its GitHub release |
| Redis | the newest in Redis's apt repository, or the one pinned in server.sh | packages.redis.io |
Start one engine at a time with scripts/start-server.sh. It stops
whatever was running and checks that it is gone, starts the engine in an empty
directory, and appends the exact command and the engine's version to
~/logs/launch.log.
~/kivi-benchmark/scripts/start-server.sh kividb
~/kivi-benchmark/scripts/start-server.sh dragonfly
~/kivi-benchmark/scripts/start-server.sh redis
~/kivi-benchmark/scripts/start-server.sh redis-cluster
Valkey, KeyDB and Garnet are not installed by user_data/server.sh. Install
them once with scripts/install-more-engines.sh (Valkey 9.1.2 from its
binary package, KeyDB v6.3.4 built from its release tag, Garnet v2.2.1 on the
.NET 10 runtime), and start them the same way:
~/kivi-benchmark/scripts/install-more-engines.sh
~/kivi-benchmark/scripts/start-server.sh valkey
~/kivi-benchmark/scripts/start-server.sh keydb
~/kivi-benchmark/scripts/start-server.sh garnet
What each is started with, on a 48-vCPU instance (all on port 6379, the cluster on 7000 upwards):
| Engine | Command |
|---|---|
| KiviDB | kividb --port 6379 --bind 0.0.0.0 --threads 46 |
| Dragonfly | dragonfly --port 6379 --bind 0.0.0.0 --proactor_threads 46 --dbfilename "" --logtostderr |
| Redis | redis-server --port 6379 --bind 0.0.0.0 --protected-mode no --save "" --appendonly no --io-threads 8 |
| Redis Cluster | 46 x redis-server --port 70NN --cluster-enabled yes --save "" --appendonly no, no replicas |
| Valkey | valkey-server --port 6379 --bind 0.0.0.0 --protected-mode no --save "" --appendonly no --io-threads 8 |
| KeyDB | keydb-server --port 6379 --bind 0.0.0.0 --protected-mode no --save "" --appendonly no --server-threads 16 |
| Garnet | GarnetServer --port 6379 --bind 0.0.0.0 --index 1g |
The same for every engine:
- Persistence is off. No snapshot and no append-only file.
- No memory limit, and an empty dataset at the start.
- Threads. KiviDB and Dragonfly get the same number, vCPUs minus two,
which is KiviDB's own default. Set
THREADSto change it for both. KiviDB takes its thread count from--threadsonly. - Redis executes commands on one thread, whatever
--io-threadsis. A single Redis against a multi-threaded engine on 48 cores is one core against many.redis-clusteris the comparison that gives Redis the whole machine: one primary per core, and memtier in cluster mode. Valkey is the same:--io-threadsmoves socket I/O off the main thread, not command execution. - KeyDB gets 16 server threads (
KEYDB_THREADS), and Garnet sizes its own thread pool. Garnet's hash index is raised from its 128 MB default to 1 GB, for the 10 million keys of the matrix.
Core scaling. CORES=<n> confines the engine to the first n CPUs with
taskset and sets its thread or shard count to match:
CORES=4 ~/kivi-benchmark/scripts/start-server.sh kividb
CORES=16 ~/kivi-benchmark/scripts/start-server.sh redis-cluster # 16 primaries
scripts/stop-server.sh kills everything and fails if anything is left
running or listening. Engines are killed, not asked to stop: the data is
disposable, and an engine asked nicely may first write it all to disk.
4. The client instance (c7gn.16xlarge)
user_data/client.sh builds memtier_benchmark from its newest release tag
and installs redis-cli. Its log is /var/log/user-data-client.log.
Latency check (same AZ; should be well under a millisecond):
ping <server-private-ip>
5. Runs (from the client)
SERVER=<server-private-ip>
# With the engine started on the server:
~/kivi-benchmark/scripts/run-matrix.sh kividb $SERVER 6379
~/kivi-benchmark/scripts/run-matrix.sh dragonfly $SERVER 6379
~/kivi-benchmark/scripts/run-matrix.sh redis $SERVER 6379
~/kivi-benchmark/scripts/run-matrix.sh redis-cluster $SERVER 7000 cluster
For each value size, run-matrix.sh empties the server, writes every key
once, and then runs each scenario for a fixed time over the whole keyspace:
| Value sizes | 100 bytes and 1 KB |
| Pipeline depth | 1 (60 threads x 20 connections) and 16 (60 threads x 5 connections) |
| Workloads | write-only, read-only, and 1 write to 10 reads |
| Keys | 10 million, uniformly random |
| Duration | 60 seconds a scenario, after the load |
Two choices matter more than the rest:
- Scenarios are timed, not counted (
--test-time, not-n). With a fixed number of requests, a fast engine is finished in seconds and is measured while it is still warming up, and a slow one runs for minutes. - The value size is always given. memtier's default is 32 bytes, which is the most flattering case for any engine.
Results are written to ~/results/<label>/: memtier's text and JSON output
for every scenario, the server's memory and key count after each load
(bytes per key), and summary.csv. Garnet does not report used_memory, so
its bytes-per-key column is empty.
Multi-key commands. run-multikey.sh loads the same 10 million keys
(100-byte values) and runs MSET and MGET of 10 keys a command for 60
seconds each, at pipeline depth 1. Its rows are added to the same
summary.csv, counted in commands a second, not keys. It is not for a
cluster: ten random keys do not share a hash slot.
~/kivi-benchmark/scripts/run-multikey.sh kividb $SERVER 6379
Core scaling. With the engine started under CORES=<n>, a shorter matrix
is enough to see the curve:
OUT=~/results/scale-kividb-8 DURATION=30 SIZES=100 PIPELINES=16 RATIOS="1:0 0:1" \
~/kivi-benchmark/scripts/run-matrix.sh scale-kividb-8 $SERVER 6379
The published capture
The numbers at the top of this page come from these scripts: on the server, CORES=32 scripts/start-server.sh <engine> for each engine in turn; on the client,
DURATION=30 scripts/run-matrix.sh <engine> $SERVER $PORT, twice, averaged. For the
published scenarios that runs, after loading 10 million 100-byte keys:
COMMON="-s $SERVER -p $PORT --distinct-client-seed --hide-histogram --key-minimum=1 --key-maximum=10000000 --key-pattern R:R --test-time=30 -t 60 -d 100"
# GET and SET, pipeline 16 (300 connections)
memtier_benchmark $COMMON --ratio 0:1 -c 5 --pipeline 16
memtier_benchmark $COMMON --ratio 1:0 -c 5 --pipeline 16
# GET and SET, no pipelining (1,200 connections)
memtier_benchmark $COMMON --ratio 0:1 -c 20 --pipeline 1
memtier_benchmark $COMMON --ratio 1:0 -c 20 --pipeline 1
The Redis Cluster runs add --cluster-mode against 32 primaries started with
CORES=32 scripts/start-server.sh redis-cluster.
6. Teardown
terraform destroy
7. Reproducibility notes
Results are environment-specific. Repeat runs on your own hardware and workload shape. Key factors that affect numbers:
- NIC state — c7gn ENA adapters can vary in throughput across runs by 5–15%.
- NUMA / CPU frequency — Graviton3 has a single NUMA node; frequency scaling is minimal but not zero.
- Kernel scheduler — io_uring submission batching varies with load.
- Run order — Always benchmark all three stores in the same session on the same instance to control for environmental drift.
Treat all published figures as one controlled capture, not a universal guarantee. The scripts and the Terraform are in this repository.
8. Comparing to Dragonfly README (other scenarios)
The upstream Dragonfly README also documents m5.large vs m5.xlarge
comparisons and c6gn.16xlarge peak throughput with -d 256 and
tunable -t / --pipeline. Those require different instance types
and memtier flags than this c7gn runbook; reproduce them by changing
server_instance_type / client_instance_type and the memtier command
lines to match the specific README row you care about.
Vector search
The vector search benchmark against Redis Stack has its own page: Vector search benchmark.