Load into KiviDB: migrate from Amazon ElastiCache to KiviDB

Self-hosted KiviDB

One RDB file (cluster mode disabled, or Serverless). Download it and start KiviDB with it:

aws s3 cp s3://my-migration-bucket/kividb-move-0001.rdb /backups/dump.rdb
kividb --dir /var/lib/kividb --requirepass "$KIVIDB_PASSWORD" --preload-file rdb:/backups/dump.rdb

--preload-file loads that file instead of what is in --dir. If the file cannot be read, KiviDB does not start, so you never get an empty server by mistake. From then on KiviDB writes its own dump.kdb; leave the flag off on later restarts.

One file per shard (cluster mode enabled, MemoryDB). --preload-file takes one file, so load the files over the network with RedisShake's rdb_reader, one run per file, into a running KiviDB:

# shake-0001.toml
[rdb_reader]
filepath = "/backups/kividb-move-0001.rdb"

[redis_writer]
address = "kividb.internal:6380"
password = "..."
# cluster = true     # when the target is a KiviDB cluster: any node's address

[advanced]
dir = "data-kividb-move-0001"

for f in /backups/kividb-move-*.rdb; do
  n=$(basename "$f" .rdb)
  sed "s#kividb-move-0001#$n#g" shake-0001.toml > "shake-$n.toml"
  ./redis-shake "shake-$n.toml" || break
done

Each run prints all done at the end. The keys of different shards do not overlap, so the files can be loaded in any order.

Over commands (Option 2). Point the writer at KiviDB and run it:

[redis_writer]
address = "kividb.internal:6380"
password = "..."
tls = false

[advanced]
rdb_restore_command_behavior = "rewrite"   # a rerun overwrites keys already copied

./redis-shake shake.toml

Without ksn, it stops by itself after all done. With ksn, it keeps running and copying changes until you stop it with Ctrl-C.

KiviDB Cloud

A KiviDB Cloud database cannot replicate from another server, so data arrives as a file or over normal commands. The database's endpoint, username (kividbapp) and password are on its overview; see Connect to your instance. Add the address of any machine that runs a copy tool to the database's allowed IPs.

One RDB file: seed a new database from S3. Make a link that is valid long enough for the database to start and fetch it (several minutes after you order; give it some hours):

aws s3 presign s3://my-migration-bucket/kividb-move-0001.rdb --expires-in 43200

A link signed with temporary credentials (an assumed role, SSO, an EC2 instance role) stops working when those credentials expire, whatever --expires-in says. Sign it with credentials that last long enough.

Then create the database. On the review page, under Seed data, choose HTTPS URL and RDB snapshot, and paste the link. The file can be no larger than the database's data size.

One RDB file: upload it. Under Snapshots → Import a snapshot, upload the .rdb. Create a database from it, or restore it into an existing database.

One file per shard, or a copy over commands. Use RedisShake as for self-hosted, with the Cloud database as the writer:

[redis_writer]
address = "my-db.cloud.kividb.io:6380"
username = "kividbapp"
password = "..."
tls = true

For a sharded database, set cluster = true and give the first shard's address, my-db-s1.cloud.kividb.io:6380. A sharded database cannot be created from an imported file, so this is the way to fill one from ElastiCache.

Valkey 9 and later

Valkey 9.0 writes its files with a new header (VALKEY080) and a new DUMP version. KiviDB 1.2.0 refuses both: --preload-file stops with bad magic, and RESTORE (which RedisShake uses) answers ERR DUMP payload version or checksum are wrong.

Copy with RIOT's --struct mode instead. It reads each key with type-specific commands, so no file format is involved. From the EC2 instance:

curl -LO https://github.com/redis/riot/releases/download/v4.3.0/riot-standalone-4.3.0-linux-x86_64.zip
unzip riot-standalone-4.3.0-linux-x86_64.zip

riot-standalone-4.3.0-linux-x86_64/bin/riot replicate --struct \
  --source-tls --source-user kividb-migrator --source-pass "$MIGRATOR_PASSWORD" \
  --target-tls --target-cacert kividb-ca.pem --target-user kividbapp --target-pass "$KIVIDB_PASSWORD" \
  --ttl-tolerance 10s \
  rediss://master.my-cache.xxxxxx.use1.cache.amazonaws.com:6379 \
  rediss://my-db.cloud.kividb.io:6380

Add --source-cluster for a cluster-mode or Serverless source, and --mode LIVE to keep copying changes (it uses keyspace notifications, as above). At the end, RIOT compares the two sides; --ttl-tolerance stops it reporting the seconds that pass during the copy as differences.

--struct leaves two things behind: stream consumer groups (the entries come across, the groups do not; recreate them with XGROUP CREATE) and per-field TTLs on hashes (the fields come across without their expiry).