Get the data out: migrate from Google Memorystore to KiviDB

Option 1: an RDB file in Cloud Storage

Make a bucket in the same region as the instance:

gcloud storage buckets create gs://my-migration-bucket --location=us-central1

Memorystore for Redis: export

The instance writes the file with its own service account. Find it and let it write to the bucket:

gcloud redis instances describe my-redis --region=us-central1 --format='value(persistenceIamIdentity)'
# serviceAccount:service-123456789@cloud-redis.iam.gserviceaccount.com

gcloud storage buckets add-iam-policy-binding gs://my-migration-bucket \
  --member=serviceAccount:service-123456789@cloud-redis.iam.gserviceaccount.com \
  --role=roles/storage.admin

It needs storage.buckets.get and storage.objects.create on the bucket (and storage.objects.delete to overwrite a file); a custom role with those three is enough if you prefer it to Storage Admin.

Then export (in the console: the instance page, Export):

gcloud redis instances export gs://my-migration-bucket/kividb-move.rdb my-redis --region=us-central1

Reads and writes keep working during the export, which can make the instance slower; scaling and upgrades wait until it ends. The export can take minutes to hours, and it can fail if heavy writes during it double the memory in use, so run it at a quiet time with memory to spare.

Memorystore for Redis Cluster: back up, then export the backup

gcloud redis clusters create-backup my-cluster --backup-id=kividb-move --region=us-central1

gcloud redis clusters backup-collections list --region=us-central1
gcloud redis clusters backups list --backup-collection=BACKUP_COLLECTION --region=us-central1

gcloud storage buckets add-iam-policy-binding gs://my-migration-bucket \
  --member=serviceAccount:service-123456789@cloud-redis.iam.gserviceaccount.com \
  --role=roles/storage.admin

gcloud redis clusters backups export kividb-move \
  --backup-collection=BACKUP_COLLECTION --region=us-central1 --gcs-bucket=my-migration-bucket

--gcs-bucket takes the bucket name without gs://, and the bucket must be a regional bucket in the backup's region.

Memorystore for Valkey: back up, then export the backup

gcloud memorystore instances backup my-valkey --backup-id=kividb-move --location=us-central1

gcloud memorystore backup-collections list --location=us-central1
gcloud memorystore backup-collections backups list --backup-collection=BACKUP_COLLECTION --location=us-central1

gcloud storage buckets add-iam-policy-binding gs://my-migration-bucket \
  --member=serviceAccount:service-123456789@gcp-sa-memorystore.iam.gserviceaccount.com \
  --role=roles/storage.admin

gcloud memorystore backup-collections backups export kividb-move \
  --backup-collection=BACKUP_COLLECTION --location=us-central1 --gcs-bucket=my-migration-bucket

Check what you got

A cluster's backup holds one RDB file per shard. List the bucket and check each file's header:

gcloud storage ls -r gs://my-migration-bucket/
gcloud storage cp -r gs://my-migration-bucket/ ./backup/
for f in $(find ./backup -name '*.rdb'); do printf '%s: ' "$f"; head -c 9 "$f"; echo; done

REDIS0011 (or a lower number) loads into KiviDB. VALKEY080 comes from Valkey 9.0 or later and does not load yet: use Option 2 with RIOT.

Option 2: copy over normal commands from inside the VPC

Memorystore has only private addresses, so the copy runs on a Compute Engine VM in the VPC that reaches the instance (for Redis Cluster and Valkey, the VPC with the Private Service Connect endpoints, and the firewall must allow ports 6379 and 11000 to 13047). The VM also needs to reach KiviDB.

• RedisShake (open source, MIT) in scan mode copies each key with DUMP and RESTORE, and with keyspace notifications keeps copying keys as they change.
• RIOT (open source, Apache 2.0) with --struct reads and writes each type with its own commands. Use it for Valkey 9.0 and later.

Following changes. Turn on keyspace notifications for the copy (CONFIG SET is blocked on Memorystore, so it goes through gcloud):

gcloud redis instances update my-redis --region=us-central1 --update-redis-config=notify-keyspace-events=KEA
gcloud redis clusters update my-cluster --region=us-central1 --update-redis-config=notify-keyspace-events=KEA
gcloud memorystore instances update my-valkey --location=us-central1 --update-engine-configs=notify-keyspace-events=KEA

Set it back afterwards.

Credentials. With an AUTH string (Memorystore for Redis), use it as the password:

gcloud redis instances get-auth-string my-redis --region=us-central1

With IAM authentication (Redis Cluster, Valkey), the password is an access token for an account with roles/redis.dbConnectionUser (Redis Cluster) or roles/memorystore.dbConnectionUser (Valkey):

gcloud auth print-access-token

The token expires after an hour. A connection that has signed in keeps working, but a copy tool that reconnects after that needs a new one, so start long copies with a fresh token and check the tool's log for authentication errors.

TLS. With in-transit encryption on, Memorystore for Redis takes TLS on port 6378; Redis Cluster and Valkey on 6379. RedisShake does not check the server's certificate, so it needs no CA. RIOT does, with --source-cacert. Get the CA from the console (Memorystore for Redis: the instance's details, TLS server certificate, Download) or with gcloud:

gcloud redis instances describe my-redis --region=us-central1   # the serverCaCerts field
gcloud redis clusters get-cluster-certificate-authority my-cluster --region=us-central1
gcloud memorystore instances get-certificate-authority my-valkey --location=us-central1

Install RedisShake:

curl -LO https://github.com/tair-opensource/RedisShake/releases/download/v4.7.0/redis-shake-v4.7.0-linux-amd64.tar.gz
tar xzf redis-shake-v4.7.0-linux-amd64.tar.gz

# Memorystore for Redis (Basic or Standard tier)
[scan_reader]
address = "10.0.0.3:6379"          # 6378 with in-transit encryption, and tls = true
password = "AUTH_STRING"
ksn = true
count = 100

# Redis Cluster, or Valkey with cluster mode enabled: the discovery endpoint
[scan_reader]
cluster = true
address = "10.0.0.5:6379"
password = "ACCESS_TOKEN"          # only with IAM authentication on
tls = true                         # only with in-transit encryption on
ksn = true
count = 100

The [redis_writer] part is in Load it into KiviDB. DUMP costs the source CPU: start with a low count and watch the CPU metric.