Benchmarking KiviDB

This is the runbook from github.com/kividbio/kivi-benchmark, where the Terraform and the scripts live. It aligns instance types and topology with the public…

This is the runbook from github.com/kividbio/kivi-benchmark, where the Terraform and the scripts live. It aligns instance types and topology with the public Dragonfly c7gn numbers: Dragonfly on c7gn.12xlarge (48 vCPU) and memtier_benchmark on a separate c7gn.16xlarge in the same Availability Zone. Infrastructure is provisioned with the Terraform in this repository; the engines are started and the runs made with the scripts under scripts/.

Results: KiviDB 1.1.0 vs Dragonfly 2.0.2 vs Redis 8.10

These are the numbers on the Benchmarks page and in the Performance section of the KiviDB README, which is the canonical copy.

Setup:

  • Server: c7gn.metal (64 vCPU, Graviton3), each engine confined to 32 cores (CORES=32, so 32 threads). Load generator: a second c7gn.metal in the same placement group. (This runbook's Terraform defaults to c7gn.12xlarge and c7gn.16xlarge; set server_instance_type and client_instance_type to c7gn.metal to repeat this capture.)
  • One engine at a time, in one session, with the same memtier_benchmark parameters.
  • run-matrix.sh: 10 million keys of 100 bytes, DURATION=30 seconds a scenario, the mean of two rounds; 300 connections with pipeline 16 (-t 60 -c 5), 1,200 without pipelining (-t 60 -c 20).
ops/sKiviDB 1.1.0Dragonfly 2.0.2Redis 8.10Redis Cluster (32 shards)
GET, pipeline 1621.2M9.1M2.4M4.7M
SET, pipeline 1619.2M8.5M2.0M4.7M
GET, no pipelining3.07M3.35M1.32M1.46M
SET, no pipelining3.12M3.38M1.54M1.46M
p99 latency, GET / SET, pipeline 160.36 / 0.42 ms0.88 / 1.07 ms2.2 / 2.7 ms
Memory per key177 B134 B178 B183 B

How to read it: with pipelining KiviDB served more than twice Dragonfly's operations and about nine times a single Redis; one request at a time per connection, Dragonfly was 8 to 9% ahead; Dragonfly used the least memory per key.

The exact commands are in section 5.

Dragonfly-published reference (c7gn, memtier defaults unless noted)

TestOps/sec (approx.)Avg. latency (µs)P99.9 (µs)
Write-only (ratio 1:0, -t 60 -c 20 -n 200000)~5.2M~250~631
Read-only (ratio 0:1, same)~6M~271~623
Pipelined read (-c 5, --pipeline=10)~8.9M~323~839

Note: For the same pipelined-read command, Dragonfly publishes ~8.9M ops/sec. An earlier capture on this runbook's default c7gn.12xlarge measured Dragonfly v1.37.0 at 8.8M ops/sec, so our Dragonfly runs match its own published figures. We benchmark all three stores in the same session on the same instance, so the comparison is fair. Expect 5–15% run-to-run variance from AWS capacity and NIC state. Treat every published figure, including Dragonfly's own, as one controlled capture, not a universal guarantee.

Commands from their write-up:

# Writes
memtier_benchmark -s $SERVER_PRIVATE_IP --distinct-client-seed --hide-histogram --ratio 1:0 -t 60 -c 20 -n 200000

# Reads
memtier_benchmark -s $SERVER_PRIVATE_IP --distinct-client-seed --hide-histogram --ratio 0:1 -t 60 -c 20 -n 200000

# Pipelined reads
memtier_benchmark -s $SERVER_PRIVATE_IP --ratio 0:1 -t 60 -c 5 -n 200000 --distinct-client-seed --hide-histogram --pipeline=10

1. Prerequisites

  • AWS account with service quotas allowing c7gn.12xlarge and c7gn.16xlarge in the chosen region.
  • An EC2 key pair in that region (for SSH).
  • Terraform >= 1.3, AWS CLI configured (aws configure or environment variables).
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
# Optional: AWS_SESSION_TOKEN for assumed roles
export AWS_DEFAULT_REGION="us-east-1"

2. Provision instances

cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars: key_name, ssh_cidr (recommended: your /32), region.

terraform init
terraform plan
terraform apply

Put key_name (and other variables) in terraform.tfvars so terraform apply does not prompt interactively.

If the server never appears in the EC2 console

  1. Canceled apply — If you press Ctrl+C while Terraform says aws_instance.server: Creating..., AWS may never finish creating the instance. Run terraform apply again and leave it running. Large c7gn instances can stay in pending for several minutes; "Still creating… 1m0s" is normal.

  2. Stale plan — If you only see the client, the server apply likely never completed. terraform state list should show aws_instance.server only after a successful apply. If it is missing, run terraform apply again.

  3. Insufficient capacity — The console error "We currently do not have sufficient c7gn.12xlarge capacity in the Availability Zone you requested" is the definitive explanation. Fix: set availability_zone in terraform.tfvars to an AZ AWS lists (e.g. in us-east-1: us-east-1a, us-east-1b, us-east-1d, us-east-1f when us-east-1c fails), or set subnet_id to a subnet in a good AZ. After changing AZ, run terraform apply so both instances use the same subnet/AZ.

  4. Placement group — If apply fails or hangs unusually long, try use_placement_group = false in terraform.tfvars, then terraform apply again.

Note outputs:

  • server_private_ip → use as SERVER for memtier from the client.
  • server_public_ip / client_public_ip → SSH access.
  • availability_zone / subnet_id → confirm you are in an AZ with c7gn capacity.

Bootstrap logs:

  • Server: /var/log/user-data-server.log
  • Client: /var/log/user-data-client.log

Wait until cloud-init finishes on both (cloud-init status --wait). The client builds memtier from source, which takes a few minutes.


3. The server instance (c7gn.12xlarge)

user_data/server.sh installs the three engines at pinned versions and starts none of them. Its log, /var/log/user-data-server.log, ends with the version of each.

EngineVersionSource
KiviDBv1.0.5, default buildofficial release binary
Dragonflyv1.37.0its GitHub release
Redisthe newest in Redis's apt repository, or the one pinned in server.shpackages.redis.io

Start one engine at a time with scripts/start-server.sh. It stops whatever was running and checks that it is gone, starts the engine in an empty directory, and appends the exact command and the engine's version to ~/logs/launch.log.

~/kivi-benchmark/scripts/start-server.sh kividb
~/kivi-benchmark/scripts/start-server.sh dragonfly
~/kivi-benchmark/scripts/start-server.sh redis
~/kivi-benchmark/scripts/start-server.sh redis-cluster

Valkey, KeyDB and Garnet are not installed by user_data/server.sh. Install them once with scripts/install-more-engines.sh (Valkey 9.1.2 from its binary package, KeyDB v6.3.4 built from its release tag, Garnet v2.2.1 on the .NET 10 runtime), and start them the same way:

~/kivi-benchmark/scripts/install-more-engines.sh
~/kivi-benchmark/scripts/start-server.sh valkey
~/kivi-benchmark/scripts/start-server.sh keydb
~/kivi-benchmark/scripts/start-server.sh garnet

What each is started with, on a 48-vCPU instance (all on port 6379, the cluster on 7000 upwards):

EngineCommand
KiviDBkividb --port 6379 --bind 0.0.0.0 --threads 46
Dragonflydragonfly --port 6379 --bind 0.0.0.0 --proactor_threads 46 --dbfilename "" --logtostderr
Redisredis-server --port 6379 --bind 0.0.0.0 --protected-mode no --save "" --appendonly no --io-threads 8
Redis Cluster46 x redis-server --port 70NN --cluster-enabled yes --save "" --appendonly no, no replicas
Valkeyvalkey-server --port 6379 --bind 0.0.0.0 --protected-mode no --save "" --appendonly no --io-threads 8
KeyDBkeydb-server --port 6379 --bind 0.0.0.0 --protected-mode no --save "" --appendonly no --server-threads 16
GarnetGarnetServer --port 6379 --bind 0.0.0.0 --index 1g

The same for every engine:

  • Persistence is off. No snapshot and no append-only file.
  • No memory limit, and an empty dataset at the start.
  • Threads. KiviDB and Dragonfly get the same number, vCPUs minus two, which is KiviDB's own default. Set THREADS to change it for both. KiviDB takes its thread count from --threads only.
  • Redis executes commands on one thread, whatever --io-threads is. A single Redis against a multi-threaded engine on 48 cores is one core against many. redis-cluster is the comparison that gives Redis the whole machine: one primary per core, and memtier in cluster mode. Valkey is the same: --io-threads moves socket I/O off the main thread, not command execution.
  • KeyDB gets 16 server threads (KEYDB_THREADS), and Garnet sizes its own thread pool. Garnet's hash index is raised from its 128 MB default to 1 GB, for the 10 million keys of the matrix.

Core scaling. CORES=<n> confines the engine to the first n CPUs with taskset and sets its thread or shard count to match:

CORES=4  ~/kivi-benchmark/scripts/start-server.sh kividb
CORES=16 ~/kivi-benchmark/scripts/start-server.sh redis-cluster   # 16 primaries

scripts/stop-server.sh kills everything and fails if anything is left running or listening. Engines are killed, not asked to stop: the data is disposable, and an engine asked nicely may first write it all to disk.


4. The client instance (c7gn.16xlarge)

user_data/client.sh builds memtier_benchmark from its newest release tag and installs redis-cli. Its log is /var/log/user-data-client.log.

Latency check (same AZ; should be well under a millisecond):

ping <server-private-ip>

5. Runs (from the client)

SERVER=<server-private-ip>

# With the engine started on the server:
~/kivi-benchmark/scripts/run-matrix.sh kividb    $SERVER 6379
~/kivi-benchmark/scripts/run-matrix.sh dragonfly $SERVER 6379
~/kivi-benchmark/scripts/run-matrix.sh redis     $SERVER 6379
~/kivi-benchmark/scripts/run-matrix.sh redis-cluster $SERVER 7000 cluster

For each value size, run-matrix.sh empties the server, writes every key once, and then runs each scenario for a fixed time over the whole keyspace:

Value sizes100 bytes and 1 KB
Pipeline depth1 (60 threads x 20 connections) and 16 (60 threads x 5 connections)
Workloadswrite-only, read-only, and 1 write to 10 reads
Keys10 million, uniformly random
Duration60 seconds a scenario, after the load

Two choices matter more than the rest:

  • Scenarios are timed, not counted (--test-time, not -n). With a fixed number of requests, a fast engine is finished in seconds and is measured while it is still warming up, and a slow one runs for minutes.
  • The value size is always given. memtier's default is 32 bytes, which is the most flattering case for any engine.

Results are written to ~/results/<label>/: memtier's text and JSON output for every scenario, the server's memory and key count after each load (bytes per key), and summary.csv. Garnet does not report used_memory, so its bytes-per-key column is empty.

Multi-key commands. run-multikey.sh loads the same 10 million keys (100-byte values) and runs MSET and MGET of 10 keys a command for 60 seconds each, at pipeline depth 1. Its rows are added to the same summary.csv, counted in commands a second, not keys. It is not for a cluster: ten random keys do not share a hash slot.

~/kivi-benchmark/scripts/run-multikey.sh kividb $SERVER 6379

Core scaling. With the engine started under CORES=<n>, a shorter matrix is enough to see the curve:

OUT=~/results/scale-kividb-8 DURATION=30 SIZES=100 PIPELINES=16 RATIOS="1:0 0:1" \
  ~/kivi-benchmark/scripts/run-matrix.sh scale-kividb-8 $SERVER 6379

The published capture

The numbers at the top of this page come from these scripts: on the server, CORES=32 scripts/start-server.sh <engine> for each engine in turn; on the client, DURATION=30 scripts/run-matrix.sh <engine> $SERVER $PORT, twice, averaged. For the published scenarios that runs, after loading 10 million 100-byte keys:

COMMON="-s $SERVER -p $PORT --distinct-client-seed --hide-histogram --key-minimum=1 --key-maximum=10000000 --key-pattern R:R --test-time=30 -t 60 -d 100"

# GET and SET, pipeline 16 (300 connections)
memtier_benchmark $COMMON --ratio 0:1 -c 5 --pipeline 16
memtier_benchmark $COMMON --ratio 1:0 -c 5 --pipeline 16

# GET and SET, no pipelining (1,200 connections)
memtier_benchmark $COMMON --ratio 0:1 -c 20 --pipeline 1
memtier_benchmark $COMMON --ratio 1:0 -c 20 --pipeline 1

The Redis Cluster runs add --cluster-mode against 32 primaries started with CORES=32 scripts/start-server.sh redis-cluster.


6. Teardown

terraform destroy

7. Reproducibility notes

Results are environment-specific. Repeat runs on your own hardware and workload shape. Key factors that affect numbers:

  • NIC state — c7gn ENA adapters can vary in throughput across runs by 5–15%.
  • NUMA / CPU frequency — Graviton3 has a single NUMA node; frequency scaling is minimal but not zero.
  • Kernel scheduler — io_uring submission batching varies with load.
  • Run order — Always benchmark all three stores in the same session on the same instance to control for environmental drift.

Treat all published figures as one controlled capture, not a universal guarantee. The scripts and the Terraform are in this repository.


8. Comparing to Dragonfly README (other scenarios)

The upstream Dragonfly README also documents m5.large vs m5.xlarge comparisons and c6gn.16xlarge peak throughput with -d 256 and tunable -t / --pipeline. Those require different instance types and memtier flags than this c7gn runbook; reproduce them by changing server_instance_type / client_instance_type and the memtier command lines to match the specific README row you care about.


Vector search

The vector search benchmark against Redis Stack has its own page: Vector search benchmark.