Unverified Commit 718e3177 authored by Dimitri Fontaine's avatar Dimitri Fontaine Committed by GitHub
Browse files

WIP: add support for logical decoding (receiver) (#12)

* First batch of work to implement support for logical decoding.

The idea is to enable Change data capture while the initial copy of the
source database is happening, in a way that we can then later catch-up with
the changes and reduce the maintenance window for the migration.

The trouble with using Logical Replication directly is that:

  - logical replication has no support for DDLs, so the schema parts needs a
    separate handling anyways

  - during the initial copy of the data the logical replication subscriber
    does not consume from the replication slot, meaning that a high traffic
	source system might end-up filling up the WAL disk entirely when the
	initial copy takes too long

  - using logical replication requires creating new SQL objects in both the
    source and the target database, and the subscription on the target
	database requires superuser access.

All in all there is a strong case to be made for writing a client-side
logical decoding client that replays the changes from the source system into
the target system.

* Improve logical decoding support, refactor code.

* Add a quick metadata parsing stage to the receiver.

This provides some stats and a better output, but that's about it really.

* Fix compilation with gcc.

* Create WAL.json files in XDG_DATA_HOME (~/.local/share).

* Implement fsync() on a regular basis and report flush_lsn accordingly.

* Install the Unix signal handlers.

* Fix the logical decoding write/flush/close callbacks.

Include calling the flush callback when a signal has been received, and the
close callback before exiting the pgsql_stream_logical function.

* Implement resuming operations from the latest LSN.

* Error handling enhancements.

This includes a retry policy for streaming changes, and also better logging
of error messages.

* Move signal handlers init to logical streaming only at the moment.

Checking that the rest of the code is ready to handle interrupts is not
within the realm of this PR. Let's just install the signal handlers for the
logical streaming command and review the general case separately.

* Add some docs for the new commands.

Just a skeleton of the needed documentation, will be improved on later.

* Refactor StreamDBoptions to re-use the common CopyDBOptions.

* Clean-up inter-modules dependencies.

* Implement pgcopydb stream receive --endpos.

* Implement a new test suite for Change Data Capture.

* Review docs.

* Improve docs.

* Typo fix the github workflow branch names (master/main)

* Add tests/cdc to our test matrix.

* Patch review.
parent 8ab548cb
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment