Cut over: migrate from Aerospike to KiviDB

Aerospike Community Edition cannot stream its changes, so the safe order is: send new writes to both stores first, copy the rest, then fix up what the copy…

Aerospike Community Edition cannot stream its changes, so the safe order is: send new writes to both stores first, copy the rest, then fix up what the copy raced with.

  1. Prepare KiviDB. Start it (with --aof on a self-hosted server), or create the Cloud database. Create the search indexes.

  2. Dual-write. Deploy a version of the application that writes every change to Aerospike and to KiviDB (the same conversion as the loader: Changing your application), deletes from both, and still reads from Aerospike. Note the time it went live everywhere as T0 (Unix seconds). A failed KiviDB write should be logged, not fail the request: the catch-up pass repairs it.

  3. Bulk copy. Run the loader. Records written during the copy may be copied from an older read, after the application already wrote the newer value.

  4. Catch up. Re-copy everything written since T0; this puts back the newest value of every record touched during the copy:

    python as2kivi.py --aerospike 10.0.0.5:3000 --namespace app --kividb "$KIVIDB_URL" --mode json --since "$T0"
    

    It uses its own checkpoint file, as2kivi-since-<T0>.done.

  5. Prune. Remove keys whose record was deleted while the copy ran (the copy could have read a record just before the application deleted it):

    python as2kivi.py --aerospike 10.0.0.5:3000 --namespace app --kividb "$KIVIDB_URL" --prune
    
  6. Verify. Run the verify script. Counts should match per set, and the sample should show no problems.

  7. Switch reads. Move reads to KiviDB, a service or a percentage at a time, while still writing to both. Compare results if you can.

  8. Stop writing to Aerospike. Once reads have run on KiviDB long enough to trust it, remove the Aerospike writes. Keep Aerospike running, unwritten, for a while.

Rollback. Until step 8, Aerospike still has every write: move reads back. After step 8, rolling back means copying KiviDB back into Aerospike, so keep dual-writing until you are sure.

If you can stop writes instead. Put the application in maintenance, run the loader, run verify, and switch. No dual-write, catch-up or prune needed; the downtime is the copy time, so measure it on a copy of your data first.

Enterprise Edition with XDR or Kafka. If the changes already flow to Kafka through Aerospike Connect, start a consumer that applies them to KiviDB before step 3, and skip steps 2, 4 and 5.