Get the data out: migrate from Redis Cloud to KiviDB

Free Essentials databases cannot back up or export: use Option 2.

Option 1: a backup to your storage (paid Essentials and Pro)

Free Essentials databases cannot back up or export: use Option 2.

1. Give Redis Cloud access to the storage.

For AWS S3, add a bucket policy for Redis Cloud's AWS account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RedisCloudBackupsAccess",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::168085023892:root" },
      "Action": ["s3:PutObject", "s3:getObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-migration-bucket/*"
    }
  ]
}

A bucket encrypted with a KMS key also needs that principal in the key policy.

For Google Cloud Storage, on the bucket's Permissions, Grant Access to service@redislabs-prod-clusters.iam.gserviceaccount.com with the role Storage Legacy Bucket Writer.

For Azure Blob Storage and FTP, the destination carries the credentials (below).

2. Set the destination and back up. Open the database, Edit, then the Configuration tab. Turn on Remote backup (under Performance & availability on Essentials, Durability on Pro), choose the storage type and the path:

Storage type	Path
AWS S3	`s3://my-migration-bucket/kividb-move`
Google Cloud Storage	`gs://my-migration-bucket/kividb-move`
Azure Blob Storage	`abs://ACCESS_KEY@STORAGE_ACCOUNT/CONTAINER/kividb-move`
FTP	`ftps://USER:PASSWORD@ftp.example.com:21/kividb-move/` (URL-encode special characters)

Save, then select Backup now on the database. The Redis Cloud API can do the same: POST /subscriptions/{subscriptionId}/databases/{databaseId}/backup on Pro, POST /fixed/subscriptions/{subscriptionId}/databases/{databaseId}/backup on Essentials.

3. Collect the files. A database with several shards produces one RDB file per shard. List the destination to see them, and decompress any that end in .gz:

aws s3 ls s3://my-migration-bucket/kividb-move/
aws s3 cp --recursive s3://my-migration-bucket/kividb-move/ ./backup/
gunzip ./backup/*.gz 2>/dev/null; ls -l ./backup

Redis Software. In the Cluster Manager UI, open the database, select Configuration, then the ⋮ menu, Export. It writes to FTP, SFTP, S3, Google Cloud Storage, Azure Blob Storage or a mount point, one file per shard. The REST API does the same:

curl -k -u "$RS_ADMIN:$RS_PASSWORD" -X POST \
  https://cluster.example.com:9443/v1/bdbs/1/actions/export \
  -H 'Content-Type: application/json' \
  -d '{"export_location": {"type": "url", "url": "sftp://user:password@10.0.0.5/exports/"}}'

If the database loads a module whose data cannot be exported, the export is refused with 406.

Option 2: copy over normal commands

This works on every plan, including Free, and on Redis Software. Run the tool on a machine that can reach both the Redis Cloud endpoint (public, or through your VPC peering or Private Service Connect) and KiviDB.

RedisShake (open source, MIT) in scan mode copies each key with DUMP and RESTORE, and with keyspace notifications keeps copying keys as they change.
RIOT (open source, Apache 2.0) with --struct reads and writes each type with its own commands. It is the one that carries time series across.

Behind the default endpoint, a database with several shards answers SCAN for all of them. With the OSS Cluster API turned on, SCAN covers one shard, so set cluster = true (RedisShake) or --source-cluster (RIOT).

Following changes. Turn on keyspace notifications for the copy (Redis Cloud allows this setting):

redis-cli ... CONFIG SET notify-keyspace-events KEA

Set it back to its old value (CONFIG GET notify-keyspace-events first) when you are done.

Install RedisShake:

curl -LO https://github.com/tair-opensource/RedisShake/releases/download/v4.7.0/redis-shake-v4.7.0-linux-amd64.tar.gz
tar xzf redis-shake-v4.7.0-linux-amd64.tar.gz

[scan_reader]
address = "redis-12345.c12345.us-east-1-mz.ec2.cloud.rlrcp.com:12345"
username = "default"
password = "..."
tls = true            # if TLS is on
ksn = true
count = 100
# cluster = true      # only with the OSS Cluster API on
skip_unknown_type = ["TSDB-TYPE", "MBbloom--", "MBbloomCF", "CMSk-TYPE", "TopK-TYPE", "TDIS-TYPE"]

skip_unknown_type leaves out the types KiviDB cannot take as a DUMP payload; time series then go across with RIOT (below). The [redis_writer] part is in Load it into KiviDB.