Unverified Commit b90dd58f authored by Dimitri Fontaine's avatar Dimitri Fontaine Committed by GitHub
Browse files

Automatically switch from prefetch/catchup to live replay mode. (#199)

* Automatically switch from prefetch/catchup to live replay mode.

When pgcopydb clone --follow is used, it is important to prefetch changes
while the initial COPY is ongoing, and switch to catchup mode thereafter.
That said, when we're all caught-up, we could switch to live replay mode
where we have:

  stream | transform | replay

* Fix pgcopydb stream commands internal service names.

Some commands can run either as a service, when processing stdin/stdout
streams instead of files.

* pgcopydb stream catchup now needs the sentinel apply to be set.

Part of the refactoring and changes is that now the low-level commands also
require the `pgcopydb stream sentinel set apply` command.

* Skip creating the main pidfile in auxilliary services.

That said, still refuse to run auxilliary services when the main service is
already running, except for the snapshot service which has been designed to
run concurrently to the main pgcopydb clone --follow service.

* Review follow mode subprocess management.

Instead of sending a TERM signal to other sub-processes as soon as the first
one has finished, make sure that we detect feof() on the stream those
processes are reading from.

Also make sure to propagate the --endpos to the pgcopydb.sentinel table
before starting the sub-processes, so that they can all have the same
information to share. In passing, if the endpos has been reached already at
startup, skip starting the sub-processes entirely.

* Refactoring, bug fixes, and actually stream KEEPALIVE and SWITCH.

Only the "main" JSON messages received from the stream receive process were
forwarded to our Unix pipe for the other processes (transform, then replay).
All of our messages should go through the pipe!

* Have pgcopydb follow use the prefetch/catchup - stream loop.

* Documentation coverage.
parent b97b977a
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment