Get the data out: migrate from DigitalOcean to KiviDB
Two open-source tools copy over normal commands. Both have been tested with KiviDB 1.2.0 against a Valkey server with DigitalOcean's disabled commands…
A copy tool
Two open-source tools copy over normal commands. Both have been tested with KiviDB 1.2.0 against a Valkey server with DigitalOcean's disabled commands turned off:
| Tool | Reads with | Keeps | Use it for |
|---|---|---|---|
| RedisShake 4.x, scan mode | SCAN, DUMP, PTTL | Every type, TTLs, stream consumer groups, every numbered database | Valkey 8 |
| RIOT 4.x | SCAN, then Lua scripts (EVALSHA) that run TYPE, PTTL and DUMP (default), or type-specific reads (--struct) | Every type and TTLs; one numbered database per run | Valkey 8; --struct for Valkey 9. Needs Lua on the source, which DigitalOcean allows |
RedisShake (Valkey 8)
On the copy machine:
curl -LO https://github.com/tair-opensource/RedisShake/releases/download/v4.7.0/redis-shake-v4.7.0-linux-amd64.tar.gz
tar xzf redis-shake-v4.7.0-linux-amd64.tar.gz
Write shake.toml. The reader is the DigitalOcean cluster:
[scan_reader]
address = "my-cache-do-user-1234567-0.db.ondigitalocean.com:25061"
username = "default"
password = "do-password"
tls = true
dbs = [] # every database
ksn = false # true to keep following changes (see the cutover)
count = 1000 # keys per SCAN
[redis_writer]
address = "your-name.cloud.kividb.io:6380"
username = "kividbapp"
password = "kividb-password"
tls = true
[advanced]
dir = "shake-data"
log_file = "shake.log"
rdb_restore_command_behavior = "rewrite" # replace keys already on KiviDB
For a self-hosted KiviDB, put its address in [redis_writer] (port 6380 by
default, or its --tls-port), and tls = false if it has no TLS. Run it:
./redis-shake shake.toml
The log ends with scanStandaloneReader restore finished and all done when the
copy is complete.
RIOT
curl -LO https://github.com/redis/riot/releases/download/v4.3.0/riot-standalone-4.3.0-linux-x86_64.zip
unzip riot-standalone-4.3.0-linux-x86_64.zip
riot-standalone-4.3.0-linux-x86_64/bin/riot replicate \
--source-tls --source-user default --source-pass "$DO_PASSWORD" \
--target-tls --target-cacert kividb-ca.pem --target-user kividbapp --target-pass "$KIVIDB_PASSWORD" \
--ttl-tolerance 10s \
rediss://my-cache-do-user-1234567-0.db.ondigitalocean.com:25061 \
rediss://your-name.cloud.kividb.io:6380
RIOT compares the two sides at the end; --ttl-tolerance stops it reporting
the seconds that pass during the copy as differences. It copies database 0
only: for another database, run it again with /1, /2... at the end of both
URIs.
The compare reads KiviDB through Lua scripts too. If the KiviDB has no Lua (the
default build, or a Cloud database without the Lua add-on), the copy completes
and the compare then fails with ERR this build of KiviDB has no Lua: add
--compare NONE and use the verify script instead. Read RIOT's
output rather than trusting its exit code, which can be 0 after a failed read.
Valkey 9
Valkey 9 writes its RDB files with a new header (VALKEY080) and its DUMP
payloads with a new version number. KiviDB 1.2.0 reads neither yet: RESTORE
answers ERR DUMP payload version or checksum are wrong. That rules out
RedisShake (it writes with RESTORE) and RIOT's default mode for a Valkey 9
cluster.
Use RIOT with --struct. It reads each key with type-specific commands (GET,
HGETALL, LRANGE, XRANGE...) and writes them back the same way, so no
file format is involved:
riot-standalone-4.3.0-linux-x86_64/bin/riot replicate --struct \
--source-tls --source-user default --source-pass "$DO_PASSWORD" \
--target-tls --target-cacert kividb-ca.pem --target-user kividbapp --target-pass "$KIVIDB_PASSWORD" \
--ttl-tolerance 10s \
rediss://my-cache-do-user-1234567-0.db.ondigitalocean.com:25061 \
rediss://your-name.cloud.kividb.io:6380
--struct leaves three things behind: stream consumer groups (the entries
come across, the groups do not: recreate them with XGROUP CREATE),
per-field TTLs on hashes (the fields come across without them), and the
order of fields inside a stream entry (the values are right, the order may
change).
An RDB file, if the cluster allows it
valkey-cli --rdb asks the server for a full snapshot the way a replica does,
with SYNC. DigitalOcean's list of disallowed commands does not include SYNC
or PSYNC, but DigitalOcean does not document this way out either. If it works
on your cluster, you get a file:
valkey-cli -u "$DO_URI" --rdb dump.rdb
A successful run ends with Transfer finished with success after N bytes. A
Valkey 9 server sends an older-format file (REDIS0011, which KiviDB reads) to
a client that asks this way, as long as no hash has per-field TTLs; if one does,
the transfer fails. The file holds the dataset at that moment: writes made after
it are not in it.