Data integration: PostgreSQL and MySQL into KiviDB

Keep a live copy of PostgreSQL or MySQL tables in a KiviDB database with change data capture: every table copied once, then each change applied within seconds. $0.04 per pipeline-hour, with 10 GB...

Point a pipeline at a PostgreSQL or MySQL database and choose the tables. KiviDB copies each one, then applies every change committed in the source, usually within a couple of seconds, so your application reads from KiviDB while the source stays the system of record.

A copy, then every change

  • Change data capture PostgreSQL through logical replication, MySQL and MariaDB through the row-based binary log, read as a replica would. The source sees one more reader.
  • Rows become keys Each row is stored under a key you name with a template, such as customer:{id}, as a hash or a JSON document, with the columns you choose, renamed if you like, and an expiry if you want one.
  • Nothing missed A pipeline saves where it has got to in the source's log after every transaction and writes whole keys, so after a restart it picks up where it stopped and applying a change twice changes nothing.

Pipelines you can see into

  • Checked before it starts Saving a pipeline connects to the source and reads each table, so a wrong password or a missing table is reported at once, not later in the background.
  • Progress and lag Rows copied, changes applied, how far behind the source it is, and the last error, in the console and the API.
  • Cleans up after itself Deleting a PostgreSQL pipeline drops the replication slot it made, so the source does not keep its write-ahead log for a reader that is gone.

By the pipeline-hour

  • $0.04 an hour For each hour, or part of an hour, a pipeline runs: about $29.20 a month for one that runs all month. A stopped pipeline is not charged.
  • 10 GB a month included Each pipeline may write 10 GB to its database in a calendar month; after that, $0.10 per GB, counted from the keys, field names and values it writes.
  • Ten per organization, on top of the database Up to ten pipelines of up to fifty tables each. The keys a pipeline writes live in a KiviDB database you run, at that database's price.

Questions

What does the source need?

PostgreSQL needs wal_level = logical, a user with the REPLICATION attribute and a primary key on each table. MySQL needs the binary log on in ROW format with full row images, and a user with replication and SELECT privileges. The docs give the settings for Amazon RDS too.

Can it reach a private database?

Not yet. The source needs a public address that accepts connections from KiviDB Cloud; private and reserved addresses are refused.

Does it follow schema changes?

Not automatically. After adding a column, save the pipeline again and it copies the tables afresh.

Put your tables in memory

Create a database, then add a pipeline in the console's Data integration section.