Automatically switch from prefetch/catchup to live replay mode. (#199)
* Automatically switch from prefetch/catchup to live replay mode. When pgcopydb clone --follow is used, it is important to prefetch changes while the initial COPY is ongoing, and switch to catchup mode thereafter. That said, when we're all caught-up, we could switch to live replay mode where we have: stream | transform | replay * Fix pgcopydb stream commands internal service names. Some commands can run either as a service, when processing stdin/stdout streams instead of files. * pgcopydb stream catchup now needs the sentinel apply to be set. Part of the refactoring and changes is that now the low-level commands also require the `pgcopydb stream sentinel set apply` command. * Skip creating the main pidfile in auxilliary services. That said, still refuse to run auxilliary services when the main service is already running, except for the snapshot service which has been designed to run concurrently to the main pgcopydb clone --follow service. * Review follow mode subprocess management. Instead of sending a TERM signal to other sub-processes as soon as the first one has finished, make sure that we detect feof() on the stream those processes are reading from. Also make sure to propagate the --endpos to the pgcopydb.sentinel table before starting the sub-processes, so that they can all have the same information to share. In passing, if the endpos has been reached already at startup, skip starting the sub-processes entirely. * Refactoring, bug fixes, and actually stream KEEPALIVE and SWITCH. Only the "main" JSON messages received from the stream receive process were forwarded to our Unix pipe for the other processes (transform, then replay). All of our messages should go through the pipe! * Have pgcopydb follow use the prefetch/catchup - stream loop. * Documentation coverage.
parent
b97b977a
Please register or sign in to comment