Load into KiviDB: migrate from Google Memorystore to KiviDB

Self-hosted KiviDB

One RDB file:

kividb --dir /var/lib/kividb --requirepass "$KIVIDB_PASSWORD" --preload-file rdb:/backups/kividb-move.rdb

--preload-file loads that file instead of what is in --dir; if it cannot be read, KiviDB does not start. Leave the flag off on later restarts.

One file per shard. --preload-file takes one file, so load each one with RedisShake's rdb_reader into a running KiviDB:

[rdb_reader]
filepath = "/backups/shard-1.rdb"

[redis_writer]
address = "kividb.internal:6380"
password = "..."
# cluster = true     # when the target is a KiviDB cluster: any node's address

./redis-shake shake.toml   # once per file, changing filepath; each run ends with "all done"

Over commands (Option 2): add the writer to the scan configuration:

[redis_writer]
address = "kividb.internal:6380"
password = "..."

[advanced]
rdb_restore_command_behavior = "rewrite"   # a rerun overwrites keys already copied

./redis-shake shake.toml

Without ksn it stops after all done; with ksn it keeps running until you stop it.

KiviDB Cloud

A KiviDB Cloud database cannot replicate from another server, so data arrives as a file or over normal commands. The endpoint, the username (kividbapp) and the password are on the database's overview; see Connect to your instance. Add the address of any VM that runs a copy tool to the database's allowed IPs.

One RDB file: seed a new database from Cloud Storage. Make a signed link that stays valid while the database starts and fetches it:

gcloud storage sign-url gs://my-migration-bucket/kividb-move.rdb --duration=12h \
  --impersonate-service-account=signer@my-project.iam.gserviceaccount.com

Twelve hours is the longest a link signed this way lasts; signing with a service account key file (--private-key-file) allows up to seven days. The account that signs must be able to read the object.

Create the database. On the review page, under Seed data, choose HTTPS URL and RDB snapshot, and paste the https://storage.googleapis.com/... link (a gs:// path is not accepted). The file can be no larger than the database's data size.

One RDB file: upload it. Under Snapshots → Import a snapshot, upload it. Create a database from it, or restore it into an existing one.

One file per shard, or a copy over commands. Use RedisShake as for self-hosted, with the Cloud database as the target:

[redis_writer]
address = "my-db.cloud.kividb.io:6380"
username = "kividbapp"
password = "..."
tls = true

For a sharded database, set cluster = true and give my-db-s1.cloud.kividb.io:6380. A sharded database cannot be created from an imported file, so this is how to fill one.

Valkey 9 and later

Valkey 9.0 writes its files with a new header (VALKEY080) and a new DUMP version. KiviDB 1.2.0 refuses both: --preload-file stops with bad magic, and RESTORE (which RedisShake uses) answers ERR DUMP payload version or checksum are wrong.

Copy with RIOT's --struct mode instead, which reads each key with type-specific commands. From the VM:

curl -LO https://github.com/redis/riot/releases/download/v4.3.0/riot-standalone-4.3.0-linux-x86_64.zip
unzip riot-standalone-4.3.0-linux-x86_64.zip

riot-standalone-4.3.0-linux-x86_64/bin/riot replicate --struct --mode LIVE \
  --source-cluster \
  --target-tls --target-cacert kividb-ca.pem --target-user kividbapp --target-pass "$KIVIDB_PASSWORD" \
  --ttl-tolerance 10s \
  redis://10.0.0.5:6379 rediss://my-db.cloud.kividb.io:6380

Leave out --source-cluster for an instance with cluster mode disabled. Add --source-pass "$(gcloud auth print-access-token)" with IAM authentication, and --source-tls --source-cacert valkey-ca.pem with in-transit encryption. --mode LIVE keeps copying changes (it needs keyspace notifications); without it, RIOT copies once and then compares the two sides.

--struct leaves two things behind: stream consumer groups (the entries come across, the groups do not; recreate them with XGROUP CREATE) and per-field TTLs on hashes (the fields come across without their expiry).