Get the data out: migrate from Amazon ElastiCache to KiviDB

Export works for node-based Redis OSS and Valkey clusters and for Serverless caches. It does not work for clusters that use data tiering. MemoryDB has the…

Option 1: export a backup to S3

Export works for node-based Redis OSS and Valkey clusters and for Serverless caches. It does not work for clusters that use data tiering. MemoryDB has the same feature with its own commands, below. The S3 bucket must be in the same region as the backup.

1. Let ElastiCache write to the bucket. Add this bucket policy (replace the region and bucket name). The principal is per region:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ElastiCacheExport",
      "Effect": "Allow",
      "Principal": { "Service": "us-east-1.elasticache-snapshot.amazonaws.com" },
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketAcl",
        "s3:ListMultipartUploadParts",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": ["arn:aws:s3:::my-migration-bucket", "arn:aws:s3:::my-migration-bucket/*"]
    }
  ]
}

The person running the export needs elasticache:CopySnapshot, and s3:ListAllMyBuckets on * as well as access to the bucket.

2. Stop writes, if you want the file to be the final copy. A backup is a point-in-time copy; anything written after it starts is not in it. If you are going to catch up with a copy tool afterwards, you can leave writes running.

3. Take a backup.

# Cluster mode disabled: name a node (with replicas, ElastiCache backs up from a replica)
aws elasticache create-snapshot --cache-cluster-id my-cache-001 --snapshot-name kividb-move

# Cluster mode enabled: name the replication group
aws elasticache create-snapshot --replication-group-id my-cluster --snapshot-name kividb-move

# Serverless
aws elasticache create-serverless-cache-snapshot \
  --serverless-cache-name my-cache --serverless-cache-snapshot-name kividb-move

Wait until the backup's status is available (aws elasticache describe-snapshots --snapshot-name kividb-move, or describe-serverless-cache-snapshots).

4. Export it. In the console: Backups, select the backup, Export, enter a name and the bucket, then Copy. Or:

# Node-based
aws elasticache copy-snapshot \
  --source-snapshot-name kividb-move \
  --target-snapshot-name kividb-move \
  --target-bucket my-migration-bucket

# Serverless
aws elasticache export-serverless-cache-snapshot \
  --serverless-cache-snapshot-name kividb-move \
  --s3-bucket-name my-migration-bucket

ElastiCache adds a node or shard number and .rdb to the name, so the bucket holds kividb-move-0001.rdb, and for a cluster with several shards kividb-move-0002.rdb and so on. List the bucket to see what you got:

aws s3 ls s3://my-migration-bucket/

MemoryDB. Take a snapshot and export it the same way:

aws memorydb create-snapshot --cluster-name my-memorydb --snapshot-name kividb-move
aws memorydb copy-snapshot \
  --source-snapshot-name kividb-move \
  --target-snapshot-name kividb-move \
  --target-bucket my-migration-bucket

MemoryDB's bucket policy uses the principal us-east-1.memorydb-snapshot.amazonaws.com (your region), and the bucket's Object Ownership must be ACLs enabled, bucket owner preferred. Clusters that use data tiering cannot export. MemoryDB writes one file per shard, named like ElastiCache's.

5. Check the file's format before you plan around it:

aws s3 cp s3://my-migration-bucket/kividb-move-0001.rdb .
head -c 9 kividb-move-0001.rdb; echo

REDIS0011 (or a lower number) loads into KiviDB. VALKEY080 comes from Valkey 9.0 or later and does not load yet: use Option 2 with RIOT.

Option 2: copy over normal commands from inside the VPC

ElastiCache node-based clusters, VPC Serverless caches and MemoryDB have no public address, so the copy runs on an EC2 instance in the same VPC (or a peered one, or over a VPN). That machine also needs to reach KiviDB.

The source allows SCAN, TYPE, PTTL and DUMP on every deployment type, which is what these tools use:

  • RedisShake (open source, MIT) in scan mode copies each key with DUMP and RESTORE. With keyspace notifications on, it keeps copying keys as they change until you stop it.
  • RIOT (open source, Apache 2.0) with --struct reads and writes each type with its own commands. Use it when the source is Valkey 9.0 or later.

Give the copy its own user, so you can remove it afterwards. With RBAC (the user group must be on a cache with in-transit encryption):

aws elasticache create-user --user-id kividb-migrator --user-name kividb-migrator \
  --engine valkey --passwords "$MIGRATOR_PASSWORD" \
  --access-string "on ~* &* +@read +@scripting +@pubsub +@connection +info +time"
aws elasticache modify-user-group --user-group-id my-user-group --user-ids-to-add kividb-migrator

Use --engine redis for a Redis OSS cache. That access string is what both tools need: RedisShake reads with SCAN, TYPE, DUMP and PTTL and subscribes for changes; RIOT reads through Lua scripts that call TIME. With an AUTH token instead of RBAC, use the token as the password and no username.

Do not run the copy as an IAM-authenticated user. Its token lasts 15 minutes and the connection is closed after 12 hours, and neither tool renews it.

Catching changes while you copy (node-based only). RedisShake's ksn mode listens for keyspace notifications. Set notify-keyspace-events to KEA in a custom parameter group (the default group cannot be changed), and apply it to the cluster; it takes effect at once. Serverless does not offer PSUBSCRIBE, so on Serverless, stop writes before the copy, or copy and then replay.

Install RedisShake on the EC2 instance:

curl -LO https://github.com/tair-opensource/RedisShake/releases/download/v4.7.0/redis-shake-v4.7.0-linux-amd64.tar.gz
tar xzf redis-shake-v4.7.0-linux-amd64.tar.gz

The [scan_reader] section for each source type:

# Cluster mode disabled: the primary endpoint
[scan_reader]
address = "master.my-cache.xxxxxx.use1.cache.amazonaws.com:6379"
username = "kividb-migrator"
password = "..."
tls = true
ksn = true        # needs notify-keyspace-events KEA
count = 100

# Cluster mode enabled, Serverless, MemoryDB: the configuration or cluster endpoint
[scan_reader]
cluster = true
address = "clustercfg.my-cluster.xxxxxx.use1.cache.amazonaws.com:6379"
username = "kividb-migrator"
password = "..."
tls = true
ksn = true        # node-based only; leave false on Serverless
count = 100

The [redis_writer] section, and running it, are in Load it into KiviDB. count is how many keys each SCAN returns; DUMP costs the source CPU, so start low on a busy cache and watch EngineCPUUtilization.