The configuration is written to file and can be edited with the new command
pgcopydb config set; now this command also signals the running process with
SIGHUP and the config file can be read and applied on-the-fly.

Those commands only work when the config.ini file can be found on-disk,
which is to say when a pgcopydb command has been started already. The
commands allow to review and edit the current settings being used.

At the moment editing the values still has no impact on the running
processes.

This allows to later review how the command was run, and more importantly
editing the configuration file (via new commands, not implemented yet) and
signal the already running pgcopydb to dynamically apply new settings.

It turns out that we can't just call pg_create_logical_replication_slot() in
a transaction that uses a snapshot to sync the initial copy of the data with
the change data capture. We need to use the CREATE_REPLICATION_SLOT logical
replication command with the RESERVE_WAL true SNAPSHOT 'export' options.

* Add a new test case that fails because of missing data around replication slot creation.

* Implement tracking of --plugin, --origin, and --slot-name options.

We already tracked --snapshot so that we can --resume and continue operating
with the same snapshot as before, and we should really have the same
facility for the rest of the replication based options.

* In --follow mode, export the snapshot via CREATE_REPLICATION_SLOT.

The previous code would set transaction snapshot to a previously exported
snapshot and then create the logical replication slot, but that creates a
replication gap. Postgres supports only one way to avoid that gap, which
consists of exporting the snapshot used to create the replication slot.

In passing, register the full information we get from creating the
replication slot to file, in such a way that we can re-use that file and
information later, including the consistent_point LSN that is associated
with the replication slot... and the snapshot.

As a result, the following commands have been removed from the CLI:

    pgcopydb stream create slot
    pgcopydb stream create origin
    pgcopydb stream drop slot
    pgcopydb stream drop origin

The only way to create a replication slot with pgcopydb is now the command
`pgcopydb snapshot --follow`, and the replication origin on the target must
be created with the consistent_point LSN returned by the same command.

* Fix unit tests to adjust to the new API/CLI.

The creation of a replication slot is not done in the pgcopydb steam setup
command anymore, the command pgcopydb snapshot --follow must be used now.

* Fix installing default values for --plugin and --slot-name.

In particular the option --plugin also is supported via the environment
variable PGCOPYDB_OUTPUT_PLUGIN and we must refrain from overwritting the
value we might have read from there.

Then we need to adjust the tests to wait until the coproc snapshot command
has created its snapshot and files on-disk to be able to properly read the
values from there (snapshot, replication slot name and lsn, etc).

That result status comes from our implementation of a logical decoding
client, where COPY BOTH protocol is used.

We have identified the root cause of the issue where double precision values were going out of range during the COPY operation. It turns out that we were missing the extra_float_digits GUC (grand unified configuration) while copying the float8 data. As a result, when extreme values like '1.7976931348623157E+308' were copied from the source, an overflow occurred.

When switching replay operating modes we want to replay all the changes that
has been streamed locally but have not been caught-up yet. This includes
some JSON to SQL transformation work that was missed before.

To be able to receive from the transform queue after having received a
signal to stop, we need to also reset the signals control flags, otherwise
all we get is an early exit in queue_receive.

Manage the connection internally in the stream_apply_send_sync_sentinel and
stream_apply_fetch_sync_sentinel functions, closing the connection each time
we have received the result.

* Update pgcopydb sentinel's replay_lsn asynchronously.

When in follow replay mode, the pgcopydb sentinel is updated quite
frequently so that we can see progress being made and be reactive about when
to implement the migration cutover.

Now, we might have to consider some latency with the source database system
where the replay_lsn is published, so rate-limiting those sentinel updates
is important. Also, rate limiting alone might not be enough to prevent
slowing down the replay process, so switch to using libpq async API.

* Make sure to update replay_lsn before sub-process exit.

The previous dynamic memory allocation fix was shy of a brick load. See #259.

The summary commands were also statically allocated with BUFSIZE bytes,
which is not always enough to fit the index definitions etc. Apply same
approach as in commit f05d75b6.

Also add unit test coverage this time.

* Assorted streaming fixes, including skipping of empty transactions.

* Fix memory management of JSON buffers when parsing Logical Messages.

Now that we need to keep around the previous LogicalMessageMetadata, we also
need to keep around its JSON buffer. This means we can't re-use the pointer
to the current context buffer, and we need to duplicate memory even when
using wal2json and keeping the same JSON buffer as received.

In passing, adjust the JSON files in the test suite according to the new
empty transactions filtering and synthetic KEEPALIVE situation.

* Atttempt to fix CI errors (exit code 141).

The exit code 141 is 128 + 13, and 13 is SIGPIPE. It means that in our
cdc-low-level test case when doing receive | transform | apply, one of the
processes is trying to write to its output pipe when the next process has
already exited, or at least closed its pipe input file descriptor.

By ignoring SIGPIPE we ask the system to return EPIPE at write time. Let's
now see how our code handles that.

* Fix wal2json bytea values.

The wal2json output plugin processes to remove the \x prefix from Postgres
bytea strings, so pgcopydb needs to put the prefix back when transforming
from the JSON to the SQL format.

* Update unit testing files with the wal2json "type" column.

Recent changes had the impact of discarding empty transactions entrirely
from the SQL file when transforming them from the JSON file. These
transactions act somewhat like a KEEPALIVE message of sorts, as they
represent activity in the source database that is not happening in the
database we're following.

In this commit we force the transform of these empty transactions to make
them part of the SQL file.

Rather than using a static BUFSIZE string buffer we call calloc() to
allocate memory dynamically for index columns, index definitions, constraint
definition, and collation descriptions.

The allocated memory is used throughout the pgcopydb program execution, and
at the moment the memory is not de-allocated by a call to free(). Instead,
we rely on the OS to take care of that for us when exiting pgcopydb.

Instead of escaping the characters at the C code level only, issuing queries
with actual newlines in the query text, arrange to embed escaped characters
in the SQL query text (\n and \r).

It could be that the role used to connect to the source and target databases
lacks privileges to access some Postgres schema and/or objects within that
schema, which means using --exclude-schema on the command line is the right
approach.

We need to find sequences that are associated with table columns either as
default values (through pg_depend entry between the sequence and pg_attrdef)
or as an identity column (through a pg_depend entry between the sequence and
the table column directly).

Also, when a sequence is filtered-out from our pg_restore activity, then we
need to also filter-out the archive TOC entry for the default value that
uses the sequence, in addition to filtering out the table and the sequence
themselves.

  ALTER TABLE ONLY public.bar ALTER COLUMN id SET DEFAULT nextval(...)

Otherwise we get ERROR: relation does not exist in pg_restore.

See
https://www.postgresql.org/docs/current/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-JSONLOG
for the list of key names that Postgres uses when using jsonlog.

This allows pgcopydb to log both to the terminal and also to a file, where
the format is not adjusted for terminal width (long time format, source file
and line always included).

Also implements logging in JSON, using a JSON object per line with a static
set of attributes that includes all the metadata we have about the log
message.

The control of this feature set is entirely based on using environment
variables, no new command line switch has been introduced. It is possible to
log both in plain text with colors on the terminal and to a JSON file at the
same time.

The --verbose increments the logLevel from “normal” to verbose (or notice),
then SQL, then DEBUG, then TRACE. Previous coding would skip the SQL level
when incrementing the level, this fixes it.

This allows postponing the first round of VACUUM ANALYZE on the target
tables after enabling the production workload again.

Rather than having to add so many parameters just re-use the options struct
as a parameter there.

We still want to have a separate command line option structure rather than
use the internal tracking CopyDataSpec structure at command line parsing
time, but we can still simplify the code a good deal.

This allows easier adding of new command line options in the future.

Careful reading of the Explicit Locking chapter of Postgres documentation
shows that VACUUM and ANALYZE take a SHARE UPDATE EXCLUSIVE lock against the
target table, and CREATE INDEX take a SHARE lock, which conflicts with the
former one.

  https://www.postgresql.org/docs/current/explicit-locking.html

So we can't actually VACUUM ANALYZE a table while building the indexes and
then constraints on top of that. The queueing of the VACUUM operations is
now done after the indexes and constraints are built.

In the apply process logging the SQL statements opens the door to logging
user data. For privacy concerns, reduce the apply module logging.

Also reduce the amount of logs sent by the receive and apply processes in
general, as this can lead to a very large volume of data and we do not
anciticape needing that level of details to be able to debug operations.

Even though the create index step must be skipped in some cases (when
Postgres doesn't have support for separate CREATE INDEX and ALTER TABLE
commands), we should still track the operation as if it was done, which
allows the code to later build the constraint with ALTER TABLE syntax.
That's when the index gets created actually.

This also allows to then skip the pg_restore entry for both the index and
the attached constraint.

* Implement a --fail-fast option.

This option allows pgcopydb to abort as soon as an error happens in one of
the sub-processes. For this, a TERM signal is sent to the process group. The
option can also be set from the environment variable PGCOPYDB_FAIL_FAST.

In passing, improve the sequence setting code for error management. As soon
as a sequence reset operation has failed, the Postgres transaction is
broken. Use savepoints to allow the error management code to continue then.

* Improve error handling.

* Fix previous improvements.

* Remove dead code.

Sytem V resources are created for sub-process communication within pgcopydb
run-time. At exit, we need to ensure that we clean-up those resources, which
are not automatically reclaimed by the OS.

We used to do that correctly with an atexit(3) handle function for the main
logging semaphore, now this is done systematically for all the semaphores
and queues created. This also ensures that the same process creates and
unlinks the resources, which wasn't always the case in the previous coding.

Add a new log level, SQL, that sits between NOTICE and DEBUG. In practice
the SQL queries and parameters are now logged when using --debug command
line, or when using --notice --verbose.

The option --notice is new and a synonym to --verbose. It has been added
because the --verbose option activates the NOTICE log level.

The logs time format was a short format including only the time, bypassing
the year/month/date parts because it takes a lot of terminal space and isn't
useful in an interactive context. This patch introduces support for the
environment variable PGCOPYDB_LOG_TIME_FORMAT which can host any format
string for the strftime(3). Also, the default log time format is
automatically switched to full date when logging to file (isatty is false).

Finally, when logging to file, then the FILE and LINE information are always
added to the output.