See
https://www.postgresql.org/docs/current/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-JSONLOG
for the list of key names that Postgres uses when using jsonlog.

This allows pgcopydb to log both to the terminal and also to a file, where
the format is not adjusted for terminal width (long time format, source file
and line always included).

Also implements logging in JSON, using a JSON object per line with a static
set of attributes that includes all the metadata we have about the log
message.

The control of this feature set is entirely based on using environment
variables, no new command line switch has been introduced. It is possible to
log both in plain text with colors on the terminal and to a JSON file at the
same time.

The --verbose increments the logLevel from “normal” to verbose (or notice),
then SQL, then DEBUG, then TRACE. Previous coding would skip the SQL level
when incrementing the level, this fixes it.

This allows postponing the first round of VACUUM ANALYZE on the target
tables after enabling the production workload again.

Rather than having to add so many parameters just re-use the options struct
as a parameter there.

We still want to have a separate command line option structure rather than
use the internal tracking CopyDataSpec structure at command line parsing
time, but we can still simplify the code a good deal.

This allows easier adding of new command line options in the future.

Careful reading of the Explicit Locking chapter of Postgres documentation
shows that VACUUM and ANALYZE take a SHARE UPDATE EXCLUSIVE lock against the
target table, and CREATE INDEX take a SHARE lock, which conflicts with the
former one.

  https://www.postgresql.org/docs/current/explicit-locking.html

So we can't actually VACUUM ANALYZE a table while building the indexes and
then constraints on top of that. The queueing of the VACUUM operations is
now done after the indexes and constraints are built.

In the apply process logging the SQL statements opens the door to logging
user data. For privacy concerns, reduce the apply module logging.

Also reduce the amount of logs sent by the receive and apply processes in
general, as this can lead to a very large volume of data and we do not
anciticape needing that level of details to be able to debug operations.

Even though the create index step must be skipped in some cases (when
Postgres doesn't have support for separate CREATE INDEX and ALTER TABLE
commands), we should still track the operation as if it was done, which
allows the code to later build the constraint with ALTER TABLE syntax.
That's when the index gets created actually.

This also allows to then skip the pg_restore entry for both the index and
the attached constraint.

* Implement a --fail-fast option.

This option allows pgcopydb to abort as soon as an error happens in one of
the sub-processes. For this, a TERM signal is sent to the process group. The
option can also be set from the environment variable PGCOPYDB_FAIL_FAST.

In passing, improve the sequence setting code for error management. As soon
as a sequence reset operation has failed, the Postgres transaction is
broken. Use savepoints to allow the error management code to continue then.

* Improve error handling.

* Fix previous improvements.

* Remove dead code.

Sytem V resources are created for sub-process communication within pgcopydb
run-time. At exit, we need to ensure that we clean-up those resources, which
are not automatically reclaimed by the OS.

We used to do that correctly with an atexit(3) handle function for the main
logging semaphore, now this is done systematically for all the semaphores
and queues created. This also ensures that the same process creates and
unlinks the resources, which wasn't always the case in the previous coding.

Add a new log level, SQL, that sits between NOTICE and DEBUG. In practice
the SQL queries and parameters are now logged when using --debug command
line, or when using --notice --verbose.

The option --notice is new and a synonym to --verbose. It has been added
because the --verbose option activates the NOTICE log level.

The logs time format was a short format including only the time, bypassing
the year/month/date parts because it takes a lot of terminal space and isn't
useful in an interactive context. This patch introduces support for the
environment variable PGCOPYDB_LOG_TIME_FORMAT which can host any format
string for the strftime(3). Also, the default log time format is
automatically switched to full date when logging to file (isatty is false).

Finally, when logging to file, then the FILE and LINE information are always
added to the output.

* Use OVERRIDING SYSTEM VALUE in INSERT statements in follow mode.

This allows bypassing GENERATED ALWAYS AS IDENTITY columns when replaying
changes from the source database to the target database. This is an INSERT
statement feature.

See https://www.postgresql.org/docs/current/sql-insert.html for details,
including:

  OVERRIDING SYSTEM VALUE

  If this clause is specified, then any values supplied for identity columns
  will override the default sequence-generated values.

  For an identity column defined as GENERATED ALWAYS, it is an error to
  insert an explicit value (other than DEFAULT) without specifying either
  OVERRIDING SYSTEM VALUE or OVERRIDING USER VALUE. (For an identity column
  defined as GENERATED BY DEFAULT, OVERRIDING SYSTEM VALUE is the normal
  behavior and specifying it does nothing, but PostgreSQL allows it as an
  extension.)

* Fix test files, add comments.

Previously, when `pgcopydb` encountered a FATAL error and called
`copydb_fatal_exit`, any wrapper processes around `pgcopydb` would also get
terminated due to the `if (kill(0, SIGTERM) == -1)` statement. To fix this
issue, a new process group is now created for `pgcopydb` during
initialization. This ensures that only the `pgcopydb` process will be
terminated in case of a FATAL error, leaving the wrapper processes intact.

In case of permission denied, we would break the whole transaction. It might
be that only some of the sequences are not accessible for the current user,
in which case we want to list them all while SELECT'ing from the ones we are
granted access to.

* Only create pgcopydb schema and table_size table when --cache is used.

Otherwise, create a temporary table instead, so that read-only roles can be
used on the source database connection to drive the database copy.

* Fix tests.

The pgcopydb list table-parts code needed to know about the new facility
where we may create a temporary table and then need to prepend pgcopydb to
the search_path.

Also refactor the new logic in schema_prepare_pgcopydb_table_size() to make
it easier to follow. It was getting too complex to read for what it does.

* Fix SWITCH WAL and KEEPALIVE message handling in transform.

The SWITCH WAL and KEEPALIVE messages both need to be written out by the
transform process as soon as it receives them when in streaming mode. That
includes flushing a current transaction out to the apply process and then
continuing with that same transaction across the WAL file boundary for
instance.

In passing, improve error handling on the return value of the fformat()
call, so that we stop processing messages early in case of a problem there.

Also, refrain from processing keepalive messages unless the server is
requesting for a reply. That reduces the amount of KEEPALIVE messages
drastically, which is a good thing.

Finally, the main follow loop must verify if we reached endpos after the
intermediate catchup at mode switching.

* Fix snapshot re-use from command line and workdir file.

In particular, make sure to remove the file on-disk when we close the
snapshot, in order to stop trying to re-use a snapshot that no longer
exists.

* Fix transform for transactions that spans multiple JSON files.

This option made it to pg_dumpall in Postgres 10 and allows to skip
passwords when migrating roles. That's quite useful in the context of
pgcopydb, and it's now available.

In case of early exit without retries from function copydb_copy_table we
would break out of the retry loop and return true, allowing pgcopydb to
continue working and report a successful return code to the OS.

In passing, also review error handling in lower-level pg_copy function to
prevent trying to COMMIT a transaction in an already failed connection.

Finally, we could have either a client-side connection error or a
server-side reported connection error: improve our code to take both the
situations into account.

* Automatically switch from prefetch/catchup to live replay mode.

When pgcopydb clone --follow is used, it is important to prefetch changes
while the initial COPY is ongoing, and switch to catchup mode thereafter.
That said, when we're all caught-up, we could switch to live replay mode
where we have:

  stream | transform | replay

* Fix pgcopydb stream commands internal service names.

Some commands can run either as a service, when processing stdin/stdout
streams instead of files.

* pgcopydb stream catchup now needs the sentinel apply to be set.

Part of the refactoring and changes is that now the low-level commands also
require the `pgcopydb stream sentinel set apply` command.

* Skip creating the main pidfile in auxilliary services.

That said, still refuse to run auxilliary services when the main service is
already running, except for the snapshot service which has been designed to
run concurrently to the main pgcopydb clone --follow service.

* Review follow mode subprocess management.

Instead of sending a TERM signal to other sub-processes as soon as the first
one has finished, make sure that we detect feof() on the stream those
processes are reading from.

Also make sure to propagate the --endpos to the pgcopydb.sentinel table
before starting the sub-processes, so that they can all have the same
information to share. In passing, if the endpos has been reached already at
startup, skip starting the sub-processes entirely.

* Refactoring, bug fixes, and actually stream KEEPALIVE and SWITCH.

Only the "main" JSON messages received from the stream receive process were
forwarded to our Unix pipe for the other processes (transform, then replay).
All of our messages should go through the pipe!

* Have pgcopydb follow use the prefetch/catchup - stream loop.

* Documentation coverage.

The KEEPALIVE and SWITCH WAL messages could be part of a
transaction (typically one that spans multiple WAL files), but also can be
found independently, between transactions.

Our previous internal representation insisted that every message be put in
the context of a transaction, but that doesn't fit well with the reality.

This refactoring is also necessary for streaming, where we don't want to
wait until the end of the next transaction before replaying the KEEPALIVE
messages. That kind of defeat the purpose of a KEEPALIVE message.

Make it easier to follow what happens in bare clone operations and what
happens in clone-and-follow operations.

* Implement pgcopydb stream replay.

The idea is to have an integrated command that does the same as the
following unix pipe command:

    $ pgcopydb stream receive --to-stdout
      | pgcopydb stream transform - -
	  | pgcopydb stream apply -

The command takes care of setting up the unix pipes and the process tree
internally. Makes it easier to use, and the implementation is needed for
intagration in the end-user API (pgcopydb clone --follow) anyway.

* Review follow module subprocess supervision and management.

* Review.

* Refactor the transform process management.

Make the transform process a sibling process to the receive and catchup
processes, where it was started by the receive process before. This
refactoring simplifies our process management and prepares for being able to
transition from the prefetch model to the live replay model.

* More refactoring, also name our queues.

Improve debugging and process management, and share more code to start the
follow sub-processes and supervise them, in a way that the same supervision
code applies to many of our modes of operations: prefetch, prefetch and
catchup, and soon, replay.

The main use case is in the test-suite where we can use our retry policy to
make sure the environment is ready for running the test script. Because it's
useful for the project in the context of its unit testing suite, it might be
useful also for any project using pgcopydb with some automation around it.

* Implement "live" streaming of changes from source to target.

This patch introduces support for Unix Pipes and an internal streaming API
where changes flow through the system rather than being batched in files.
The following command line can be used to replay changes in a "live" fashion
from the source to the target database.

  $ pgcopydb stream receive --to-stdout
    | pgcopydb stream transform - -
    | pgcopydb stream apply -

The command `pgcopydb stream replay` is scheduled to glue together those
internal parts, and then the `pgcopydb clone --follow` will be extended to
automatically switch from catchup mode to replay mode when suitable.

* Code refactoring to share more bits.

Share the code that reads through a unix pipe (stream) line by line,
introducing a user callback facility with a (void *) context.

* First pass at adding tests for our low-level API/CLI.

* Switch producers to line buffered output mode.

This avoids partial reads of a part of a line, now we read a line at a time,
unless it doesn't fit in our 128 MB read buffer, a case that is handled in
the code already (albeit quite poorly at the time being).

Also improve logging, install set -o pipefail in the test suite.

* Add the new test case to GitHub Actions CI.

* Fix internal API to the new refactored code.

This allows multiple unrelated subcommands to run at once, each with their
own service specific pidfile. Also, most sub-commands do not need a pidfile
at all and now can bypass checking for stale pids etc entirely.

The code was ready to be re-used already, but still copy-pasted. In the same
file. Well now at least it's cleaner.

Writing directly to the target JSON file creates a windows where the file
exists and can be read with partial content, either by pgcopydb itself or
from external tooling. Using a temp filename then switching to the actual
expected file protects again partial reads and parse failures.

Writing directly to the output file opens a window where the file exists and
can be read with partial content, either from pgcopydb itself, or from an
external tooling. Using a temp filename then switching to the actual
expected file protects again partial reads and parse failures.