Fix how snapshot are exported when using CDC. (#279)
It turns out that we can't just call pg_create_logical_replication_slot() in
a transaction that uses a snapshot to sync the initial copy of the data with
the change data capture. We need to use the CREATE_REPLICATION_SLOT logical
replication command with the RESERVE_WAL true SNAPSHOT 'export' options.
* Add a new test case that fails because of missing data around replication slot creation.
* Implement tracking of --plugin, --origin, and --slot-name options.
We already tracked --snapshot so that we can --resume and continue operating
with the same snapshot as before, and we should really have the same
facility for the rest of the replication based options.
* In --follow mode, export the snapshot via CREATE_REPLICATION_SLOT.
The previous code would set transaction snapshot to a previously exported
snapshot and then create the logical replication slot, but that creates a
replication gap. Postgres supports only one way to avoid that gap, which
consists of exporting the snapshot used to create the replication slot.
In passing, register the full information we get from creating the
replication slot to file, in such a way that we can re-use that file and
information later, including the consistent_point LSN that is associated
with the replication slot... and the snapshot.
As a result, the following commands have been removed from the CLI:
pgcopydb stream create slot
pgcopydb stream create origin
pgcopydb stream drop slot
pgcopydb stream drop origin
The only way to create a replication slot with pgcopydb is now the command
`pgcopydb snapshot --follow`, and the replication origin on the target must
be created with the consistent_point LSN returned by the same command.
* Fix unit tests to adjust to the new API/CLI.
The creation of a replication slot is not done in the pgcopydb steam setup
command anymore, the command pgcopydb snapshot --follow must be used now.
* Fix installing default values for --plugin and --slot-name.
In particular the option --plugin also is supported via the environment
variable PGCOPYDB_OUTPUT_PLUGIN and we must refrain from overwritting the
value we might have read from there.
Then we need to adjust the tests to wait until the coproc snapshot command
has created its snapshot and files on-disk to be able to properly read the
values from there (snapshot, replication slot name and lsn, etc).
parent
2343d74a
Please register or sign in to comment