This script works very well with development versions but I couldn't yet
figure out how to properly manage released (tagged) version numbers when
building in a docker container. Oops.

* Implement pgcopydb clone --follow.

* Review and improve some replay LSN log messages.

* Review process organisation for pgcopydb clone --follow.

We need to separate the clone process tree (with its calls to waipid()
blocking until all the sub-processes are done) and the follow process tree
that creates two extra sub-processes (prefetch and catchup).

As a result `pgcopydb clone` process tree is impacted too, and the main
process only exports a snapshot now. The rest of the top-level processing is
actually implemented in the "main" (or "clone") sub-process, even when a
"follow" sub-process is not started.

* Improve docs for pgcopydb clone --follow.

Fixes #70.

* Implement a sentinel.

The sentinel is a table on the source database with 3 columns to share the
LSN start position for the logical decoding receiver, the LSN end position
where to stop receiving and applying logical decoding changes, and finally
the apply boolean which can be set to true when the catching-up process is
allowed to start processing pre-fetched events.

The idea is to set the sentinel apply boolean as soon as the base copy
finishes, and to allow pgcopydb users and test suite to set and endpos after
the start of the follow processes.

* Assorted fixes.

  - review the test files to use the new commands facilities
  - only use the sentinel in the catchup command, not the apply command
  - filter out CDC for pgcopydb.* tables (sentinel)
  - allow retrying until the streaming context files have been created

* Use the pgcopydb sentinel table startpos and endpos in follow command.

Rather than manually fetching the current LSN on the source database and
then passing that as an argument to pgcopydb follow --endpos, we can now use
the pgcopydb stream sentinel set endpos --current command.

* Use the pgcopydb sentinel to track progress, and sync as we go.

We need to sync with the sentinel changes in case an endpos has been set
concurrently to the pgcopydb follow processing. We can also use the sentinel
shared table as a way to exchange information between the replay process and
the receive process, such as the replay_lsn.

* Publish replay_lsn feedback to the replication stream.

* Implement support for early WAL file switching, improve tests.

Now the tests are running a separate instance for the injection of SQL
commands, and the sentinel endpos is driven from outside the pgcopydb follow
process and command.

* Review and adjust documentation for pgcopydb follow etc.

This PR introduces a bunch of new commands and also changes in how some
commands are made available. Finally edit the docs to reflect the new CLI.

* Implement pgcopydb follow.

* Add some documentation coverage.

* Add some unit tests coverage.

* Fix pgcopydb drop commands.

* Fix tests.

Unless when using the --endpos option, the `pgcopydb stream catchup` command
should keep looping and reading the same file again when it's not complete,
allowing concurrent activity with other processes such as `pgcopydb stream
prefetch`.

This command knows how to catch-up from the current tracked position found
at the target database. This position should be created from the LSN
returned by the creation of the logical replication slot, which the other
command `pgcopydb stream setup` already does.

* Implement pgcopydb stream prefetch.

The prefetch command setup the logical decoding operations in a way that
allows transforming each JSON file received into an SQL file. It's a kind of
integration of the two following commands:

    $ pgcopydb stream receive
    $ pgcopydb stream transform

Where the transform process is started in the background as soon as needed.

* Implement pgcopydb stream setup.

Also implement two other new commands to create and drop a replication
origin, so that the command

    $ pgcopydb stream setup

is equivalent to the commands

    $ pgcopydb create slot
    $ pgcopydb create origin --startpos <slot creation lsn>

* Implement support for copying roles.

Postgres roles are global objects, which means they do not belong to any
specific database in an instance. To be able to dump and restore the roles,
we need to use pg_dumpall --roles-only.

* Manually send SQL commands from the pg_dumpall --roles-only script.

This allows to implement "create role if not exists" our own way, accepting
roles that already exists as they are.

* Have tests/pagila use a non-postgres role.

We now create a "pagila" role and make it owner of the database and its
objects, and then run the pgcopydb commands using that new non-superuser
role.

The creation of the new role on the target system is made by calling into
pgcopydb dump roles and pgcopydb restore roles.

* Skip ALTER ROLE commands too when skipping CREATE ROLE.

* Imnplement pgcopydb copy roles.

* Implement pgcopydb clone --roles.

This includes a STEP 0 that copies the roles from the source instance to the
target instance.

Fixes #53.

* Review the pgcopydb commands.

  pgcopydb
    clone    Clone an entire database from source to target
    fork     Clone an entire database from source to target
    follow   Replay changes from the source database to the target database
    copy-db  Copy an entire database from source to target

* Assorted fixes.

Applying the SQL is as easy as reading the prepared SQL file one line at a
time and then sending that SQL query to the target system, though it also
requires using the Postgres Replication Progress Tracking facility to skip
previously applied transactions.

This patch does not implement cleaning the replication origin tracking on
the target system.

The autopkgtest would barf at the existing files left around from the
build-time test.

* WIP: infrastructure to parse a JSON message in our internal structs.

An internal representation for the JSON message then allows to be smart
about multi-values INSERT statements or switching to COPY, and issuing a
single DELETE statement for many values too, etc.

At the moment only the infrastructure to support that is coded, we always
fill-in single tuple per statement.

* Actually use the "include-timestamp" wal2json option.

* First pass at producing the SQL queries from the JSON messages.

* Add some testing coverage.

* Fix an error spotted in the gcc output on debian.

* Implement UPDATE and DELETE output in SQL.

* Review an Assert() and make it a first-class error reporting.

* Update docs.

Because we might keep open a long-running transaction on the source
database, the one that keeps a snapshot open for the whole duration of the
migration, then it's probably best to set the tcp_keepalives_idle to
something short.

This should help avoid some problems with firewalls shutting down long
running (idle) Postgres connections or transactions.

Large numbers are hard to read and we don't have that kind of precision anyway.

* First batch of work to implement support for logical decoding.

The idea is to enable Change data capture while the initial copy of the
source database is happening, in a way that we can then later catch-up with
the changes and reduce the maintenance window for the migration.

The trouble with using Logical Replication directly is that:

  - logical replication has no support for DDLs, so the schema parts needs a
    separate handling anyways

  - during the initial copy of the data the logical replication subscriber
    does not consume from the replication slot, meaning that a high traffic
	source system might end-up filling up the WAL disk entirely when the
	initial copy takes too long

  - using logical replication requires creating new SQL objects in both the
    source and the target database, and the subscription on the target
	database requires superuser access.

All in all there is a strong case to be made for writing a client-side
logical decoding client that replays the changes from the source system into
the target system.

* Improve logical decoding support, refactor code.

* Add a quick metadata parsing stage to the receiver.

This provides some stats and a better output, but that's about it really.

* Fix compilation with gcc.

* Create WAL.json files in XDG_DATA_HOME (~/.local/share).

* Implement fsync() on a regular basis and report flush_lsn accordingly.

* Install the Unix signal handlers.

* Fix the logical decoding write/flush/close callbacks.

Include calling the flush callback when a signal has been received, and the
close callback before exiting the pgsql_stream_logical function.

* Implement resuming operations from the latest LSN.

* Error handling enhancements.

This includes a retry policy for streaming changes, and also better logging
of error messages.

* Move signal handlers init to logical streaming only at the moment.

Checking that the rest of the code is ready to handle interrupts is not
within the realm of this PR. Let's just install the signal handlers for the
logical streaming command and review the general case separately.

* Add some docs for the new commands.

Just a skeleton of the needed documentation, will be improved on later.

* Refactor StreamDBoptions to re-use the common CopyDBOptions.

* Clean-up inter-modules dependencies.

* Implement pgcopydb stream receive --endpos.

* Implement a new test suite for Change Data Capture.

* Review docs.

* Improve docs.

* Typo fix the github workflow branch names (master/main)

* Add tests/cdc to our test matrix.

* Patch review.

To prevent limitations around lock management when running pgcopydb on a
source database with lots of objects, we now benefit from --not-consistent
as an opportunity to reduce long-running transactions.

It was already the case that --not-consistent wouldn't require exporting or
setting a snapshot, but the opportunity was not yet implemented fully.

* Implement filtering the pre-data section.

When installing the schema on the target database, we want to already filter
out the SQL objects that are filtered-out in the filtering setup. The
previous coding forgot about that parts.

See #43.

* Implement support for exclude-schemas.

The parsing was implemented already, but the filtering itself was shy of a
brick load. We need to both filter tables that belong to the schema (we did)
and also the schema object itself (we didn't).

* Add missed SQL objects types to parse from pg_restore archive list.

* Add ACL as a pg_restore archive catalog description entry.

Unless the PGAPPNAME environment variable has already been set, in which
case we would just use the existing value. This allows users to force their
own application_name either in the environment or in the connection string.

* A pass at improving pgcopydb docs.

* Add coverage for the new filtering commands.

We still have a problem with how the version string is computed in the
Docker image, where there is no .git environment and then no tags. The
script from the git git repository uses an extra "version" file for
releases, and that's not what we're doing at the moment.

We might want to produce the version file on the local machine and then send
it in the Docker environment, but at the moment, let's just keep it simple.