Get the data out: migrate from Aiven to KiviDB
Two open-source tools copy over normal commands, and both have been tested with KiviDB 1.2.0 against a Valkey server with Aiven's disabled commands turned…
Option 1: a copy tool (works on every plan)
Two open-source tools copy over normal commands, and both have been tested with KiviDB 1.2.0 against a Valkey server with Aiven's disabled commands turned off:
| Tool | Reads with | Keeps | Notes |
|---|---|---|---|
| RedisShake 4.x, scan mode | SCAN, DUMP, PTTL | Every type, TTLs, stream consumer groups, every numbered database | Writes with RESTORE. Not for Valkey 9 sources |
| RIOT 4.x | SCAN, then Lua scripts (EVALSHA) that run TYPE, PTTL and DUMP (default), or type-specific reads (--struct) | Every type and TTLs | One numbered database per run. Needs Lua on the source, which Aiven allows. --struct works for Valkey 9 sources |
Download RedisShake on the copy machine:
curl -LO https://github.com/tair-opensource/RedisShake/releases/download/v4.7.0/redis-shake-v4.7.0-linux-amd64.tar.gz
tar xzf redis-shake-v4.7.0-linux-amd64.tar.gz
Write shake.toml. The reader is the Aiven service, with the host, port, user
and password from the service URI:
[scan_reader]
address = "my-valkey-my-project.e.aivencloud.com:12691"
username = "default"
password = "aiven-password"
tls = true
dbs = [] # every database
ksn = false # true to keep following changes (see the cutover)
count = 1000 # keys per SCAN
[redis_writer]
address = "your-name.cloud.kividb.io:6380"
username = "kividbapp"
password = "kividb-password"
tls = true
[advanced]
dir = "shake-data"
log_file = "shake.log"
rdb_restore_command_behavior = "rewrite" # replace keys already on KiviDB
For a self-hosted KiviDB, put its address in [redis_writer] (port 6380 by
default, or its --tls-port), and tls = false if it has no TLS. For a cluster
plan, add cluster = true to the reader; for a KiviDB cluster or sharded Cloud
database, add it to the writer.
Run it:
./redis-shake shake.toml
The log ends with scanStandaloneReader restore finished and all done when the
copy is complete.
To copy with RIOT instead:
curl -LO https://github.com/redis/riot/releases/download/v4.3.0/riot-standalone-4.3.0-linux-x86_64.zip
unzip riot-standalone-4.3.0-linux-x86_64.zip
riot-standalone-4.3.0-linux-x86_64/bin/riot replicate \
--source-tls --source-user default --source-pass "$AIVEN_PASSWORD" \
--target-tls --target-cacert kividb-ca.pem --target-user kividbapp --target-pass "$KIVIDB_PASSWORD" \
--ttl-tolerance 10s \
rediss://my-valkey-my-project.e.aivencloud.com:12691 \
rediss://your-name.cloud.kividb.io:6380
RIOT compares the two sides at the end; --ttl-tolerance stops it reporting
the seconds that pass during the copy as differences. It copies database 0
only: for another database, run it again with /1, /2... at the end of both
URIs.
The compare reads KiviDB through Lua scripts too. If the KiviDB has no Lua (the
default build, or a Cloud database without the Lua add-on), the copy completes
and the compare then fails with ERR this build of KiviDB has no Lua: add
--compare NONE and use the verify script instead. Read RIOT's
output rather than trusting its exit code, which can be 0 after a failed read.
Option 2: an RDB file over the replication protocol
valkey-cli --rdb asks the server for a full snapshot the way a replica does,
with SYNC. Aiven's list of disabled commands does not include SYNC or
PSYNC, but Aiven does not document this way out either. Try it; if it works
on your service, you get a file you can load anywhere:
valkey-cli -u "$AIVEN_URI" --rdb dump.rdb
A successful run ends with Transfer finished with success after N bytes. If
the command is refused, use option 1.
The file holds the dataset at that moment: writes made after it are not in it. On Valkey 9, see below for which files KiviDB can read.
What does not work
- Downloading a backup. Aiven's backups are not available for download, and
avn service backup-listonly lists them. SAVE,BGSAVEandMIGRATE: disabled on Aiven.- Making KiviDB a replica of the Aiven service (
kividb --replicaof): this needsSYNC/PSYNCto be allowed and the service to be reachable as a primary. Test it the same way as option 2 if you want to try it into a self-hosted KiviDB. A KiviDB Cloud database can never be made a replica of an outside server. - Aiven's own migration tools: they move data into Aiven.
Valkey 9
Valkey 9 writes its RDB files with a new header (VALKEY080) and its DUMP
payloads with a new version number. KiviDB 1.2.0 reads neither yet: loading
such a file stops with bad magic, and RESTORE answers
ERR DUMP payload version or checksum are wrong. That rules out RedisShake (it
writes with RESTORE) and RIOT's default mode for a Valkey 9 service.
Two ways still work:
-
RIOT with
--struct. It reads each key with type-specific commands (GET,HGETALL,LRANGE,XRANGE...) and writes them back the same way, so no file format is involved:riot-standalone-4.3.0-linux-x86_64/bin/riot replicate --struct \ --source-tls --source-user default --source-pass "$AIVEN_PASSWORD" \ --target-tls --target-cacert kividb-ca.pem --target-user kividbapp --target-pass "$KIVIDB_PASSWORD" \ --ttl-tolerance 10s \ rediss://my-valkey-my-project.e.aivencloud.com:12691 \ rediss://your-name.cloud.kividb.io:6380--structleaves three things behind: stream consumer groups (the entries come across, the groups do not: recreate them withXGROUP CREATE), per-field TTLs on hashes (the fields come across without them), and the order of fields inside a stream entry (the values are right, the order may change). -
valkey-cli --rdb, ifSYNCis allowed (option 2). A Valkey 9 server sends an older-format file (REDIS0011, which KiviDB reads) to a client that asks this way, as long as no hash has per-field TTLs. If one does, the transfer fails and the server logsCan't store key ... in RDB version 11.