Get the data out: migrate from Aiven to KiviDB

Two open-source tools copy over normal commands, and both have been tested with KiviDB 1.2.0 against a Valkey server with Aiven's disabled commands turned…

Option 1: a copy tool (works on every plan)

Two open-source tools copy over normal commands, and both have been tested with KiviDB 1.2.0 against a Valkey server with Aiven's disabled commands turned off:

ToolReads withKeepsNotes
RedisShake 4.x, scan modeSCAN, DUMP, PTTLEvery type, TTLs, stream consumer groups, every numbered databaseWrites with RESTORE. Not for Valkey 9 sources
RIOT 4.xSCAN, then Lua scripts (EVALSHA) that run TYPE, PTTL and DUMP (default), or type-specific reads (--struct)Every type and TTLsOne numbered database per run. Needs Lua on the source, which Aiven allows. --struct works for Valkey 9 sources

Download RedisShake on the copy machine:

curl -LO https://github.com/tair-opensource/RedisShake/releases/download/v4.7.0/redis-shake-v4.7.0-linux-amd64.tar.gz
tar xzf redis-shake-v4.7.0-linux-amd64.tar.gz

Write shake.toml. The reader is the Aiven service, with the host, port, user and password from the service URI:

[scan_reader]
address = "my-valkey-my-project.e.aivencloud.com:12691"
username = "default"
password = "aiven-password"
tls = true
dbs = []        # every database
ksn = false     # true to keep following changes (see the cutover)
count = 1000    # keys per SCAN

[redis_writer]
address = "your-name.cloud.kividb.io:6380"
username = "kividbapp"
password = "kividb-password"
tls = true

[advanced]
dir = "shake-data"
log_file = "shake.log"
rdb_restore_command_behavior = "rewrite"   # replace keys already on KiviDB

For a self-hosted KiviDB, put its address in [redis_writer] (port 6380 by default, or its --tls-port), and tls = false if it has no TLS. For a cluster plan, add cluster = true to the reader; for a KiviDB cluster or sharded Cloud database, add it to the writer.

Run it:

./redis-shake shake.toml

The log ends with scanStandaloneReader restore finished and all done when the copy is complete.

To copy with RIOT instead:

curl -LO https://github.com/redis/riot/releases/download/v4.3.0/riot-standalone-4.3.0-linux-x86_64.zip
unzip riot-standalone-4.3.0-linux-x86_64.zip

riot-standalone-4.3.0-linux-x86_64/bin/riot replicate \
  --source-tls --source-user default --source-pass "$AIVEN_PASSWORD" \
  --target-tls --target-cacert kividb-ca.pem --target-user kividbapp --target-pass "$KIVIDB_PASSWORD" \
  --ttl-tolerance 10s \
  rediss://my-valkey-my-project.e.aivencloud.com:12691 \
  rediss://your-name.cloud.kividb.io:6380

RIOT compares the two sides at the end; --ttl-tolerance stops it reporting the seconds that pass during the copy as differences. It copies database 0 only: for another database, run it again with /1, /2... at the end of both URIs.

The compare reads KiviDB through Lua scripts too. If the KiviDB has no Lua (the default build, or a Cloud database without the Lua add-on), the copy completes and the compare then fails with ERR this build of KiviDB has no Lua: add --compare NONE and use the verify script instead. Read RIOT's output rather than trusting its exit code, which can be 0 after a failed read.

Option 2: an RDB file over the replication protocol

valkey-cli --rdb asks the server for a full snapshot the way a replica does, with SYNC. Aiven's list of disabled commands does not include SYNC or PSYNC, but Aiven does not document this way out either. Try it; if it works on your service, you get a file you can load anywhere:

valkey-cli -u "$AIVEN_URI" --rdb dump.rdb

A successful run ends with Transfer finished with success after N bytes. If the command is refused, use option 1.

The file holds the dataset at that moment: writes made after it are not in it. On Valkey 9, see below for which files KiviDB can read.

What does not work

  • Downloading a backup. Aiven's backups are not available for download, and avn service backup-list only lists them.
  • SAVE, BGSAVE and MIGRATE: disabled on Aiven.
  • Making KiviDB a replica of the Aiven service (kividb --replicaof): this needs SYNC/PSYNC to be allowed and the service to be reachable as a primary. Test it the same way as option 2 if you want to try it into a self-hosted KiviDB. A KiviDB Cloud database can never be made a replica of an outside server.
  • Aiven's own migration tools: they move data into Aiven.

Valkey 9

Valkey 9 writes its RDB files with a new header (VALKEY080) and its DUMP payloads with a new version number. KiviDB 1.2.0 reads neither yet: loading such a file stops with bad magic, and RESTORE answers ERR DUMP payload version or checksum are wrong. That rules out RedisShake (it writes with RESTORE) and RIOT's default mode for a Valkey 9 service.

Two ways still work:

  • RIOT with --struct. It reads each key with type-specific commands (GET, HGETALL, LRANGE, XRANGE...) and writes them back the same way, so no file format is involved:

    riot-standalone-4.3.0-linux-x86_64/bin/riot replicate --struct \
      --source-tls --source-user default --source-pass "$AIVEN_PASSWORD" \
      --target-tls --target-cacert kividb-ca.pem --target-user kividbapp --target-pass "$KIVIDB_PASSWORD" \
      --ttl-tolerance 10s \
      rediss://my-valkey-my-project.e.aivencloud.com:12691 \
      rediss://your-name.cloud.kividb.io:6380
    

    --struct leaves three things behind: stream consumer groups (the entries come across, the groups do not: recreate them with XGROUP CREATE), per-field TTLs on hashes (the fields come across without them), and the order of fields inside a stream entry (the values are right, the order may change).

  • valkey-cli --rdb, if SYNC is allowed (option 2). A Valkey 9 server sends an older-format file (REDIS0011, which KiviDB reads) to a client that asks this way, as long as no hash has per-field TTLs. If one does, the transfer fails and the server logs Can't store key ... in RDB version 11.