Sytem V resources are created for sub-process communication within pgcopydb
run-time. At exit, we need to ensure that we clean-up those resources, which
are not automatically reclaimed by the OS.

We used to do that correctly with an atexit(3) handle function for the main
logging semaphore, now this is done systematically for all the semaphores
and queues created. This also ensures that the same process creates and
unlinks the resources, which wasn't always the case in the previous coding.

Add a new log level, SQL, that sits between NOTICE and DEBUG. In practice
the SQL queries and parameters are now logged when using --debug command
line, or when using --notice --verbose.

The option --notice is new and a synonym to --verbose. It has been added
because the --verbose option activates the NOTICE log level.

The logs time format was a short format including only the time, bypassing
the year/month/date parts because it takes a lot of terminal space and isn't
useful in an interactive context. This patch introduces support for the
environment variable PGCOPYDB_LOG_TIME_FORMAT which can host any format
string for the strftime(3). Also, the default log time format is
automatically switched to full date when logging to file (isatty is false).

Finally, when logging to file, then the FILE and LINE information are always
added to the output.

* Use OVERRIDING SYSTEM VALUE in INSERT statements in follow mode.

This allows bypassing GENERATED ALWAYS AS IDENTITY columns when replaying
changes from the source database to the target database. This is an INSERT
statement feature.

See https://www.postgresql.org/docs/current/sql-insert.html for details,
including:

  OVERRIDING SYSTEM VALUE

  If this clause is specified, then any values supplied for identity columns
  will override the default sequence-generated values.

  For an identity column defined as GENERATED ALWAYS, it is an error to
  insert an explicit value (other than DEFAULT) without specifying either
  OVERRIDING SYSTEM VALUE or OVERRIDING USER VALUE. (For an identity column
  defined as GENERATED BY DEFAULT, OVERRIDING SYSTEM VALUE is the normal
  behavior and specifying it does nothing, but PostgreSQL allows it as an
  extension.)

* Fix test files, add comments.

Previously, when `pgcopydb` encountered a FATAL error and called
`copydb_fatal_exit`, any wrapper processes around `pgcopydb` would also get
terminated due to the `if (kill(0, SIGTERM) == -1)` statement. To fix this
issue, a new process group is now created for `pgcopydb` during
initialization. This ensures that only the `pgcopydb` process will be
terminated in case of a FATAL error, leaving the wrapper processes intact.

In case of permission denied, we would break the whole transaction. It might
be that only some of the sequences are not accessible for the current user,
in which case we want to list them all while SELECT'ing from the ones we are
granted access to.

* Only create pgcopydb schema and table_size table when --cache is used.

Otherwise, create a temporary table instead, so that read-only roles can be
used on the source database connection to drive the database copy.

* Fix tests.

The pgcopydb list table-parts code needed to know about the new facility
where we may create a temporary table and then need to prepend pgcopydb to
the search_path.

Also refactor the new logic in schema_prepare_pgcopydb_table_size() to make
it easier to follow. It was getting too complex to read for what it does.

* Fix SWITCH WAL and KEEPALIVE message handling in transform.

The SWITCH WAL and KEEPALIVE messages both need to be written out by the
transform process as soon as it receives them when in streaming mode. That
includes flushing a current transaction out to the apply process and then
continuing with that same transaction across the WAL file boundary for
instance.

In passing, improve error handling on the return value of the fformat()
call, so that we stop processing messages early in case of a problem there.

Also, refrain from processing keepalive messages unless the server is
requesting for a reply. That reduces the amount of KEEPALIVE messages
drastically, which is a good thing.

Finally, the main follow loop must verify if we reached endpos after the
intermediate catchup at mode switching.

* Fix snapshot re-use from command line and workdir file.

In particular, make sure to remove the file on-disk when we close the
snapshot, in order to stop trying to re-use a snapshot that no longer
exists.

* Fix transform for transactions that spans multiple JSON files.

This option made it to pg_dumpall in Postgres 10 and allows to skip
passwords when migrating roles. That's quite useful in the context of
pgcopydb, and it's now available.

In case of early exit without retries from function copydb_copy_table we
would break out of the retry loop and return true, allowing pgcopydb to
continue working and report a successful return code to the OS.

In passing, also review error handling in lower-level pg_copy function to
prevent trying to COMMIT a transaction in an already failed connection.

Finally, we could have either a client-side connection error or a
server-side reported connection error: improve our code to take both the
situations into account.

* Automatically switch from prefetch/catchup to live replay mode.

When pgcopydb clone --follow is used, it is important to prefetch changes
while the initial COPY is ongoing, and switch to catchup mode thereafter.
That said, when we're all caught-up, we could switch to live replay mode
where we have:

  stream | transform | replay

* Fix pgcopydb stream commands internal service names.

Some commands can run either as a service, when processing stdin/stdout
streams instead of files.

* pgcopydb stream catchup now needs the sentinel apply to be set.

Part of the refactoring and changes is that now the low-level commands also
require the `pgcopydb stream sentinel set apply` command.

* Skip creating the main pidfile in auxilliary services.

That said, still refuse to run auxilliary services when the main service is
already running, except for the snapshot service which has been designed to
run concurrently to the main pgcopydb clone --follow service.

* Review follow mode subprocess management.

Instead of sending a TERM signal to other sub-processes as soon as the first
one has finished, make sure that we detect feof() on the stream those
processes are reading from.

Also make sure to propagate the --endpos to the pgcopydb.sentinel table
before starting the sub-processes, so that they can all have the same
information to share. In passing, if the endpos has been reached already at
startup, skip starting the sub-processes entirely.

* Refactoring, bug fixes, and actually stream KEEPALIVE and SWITCH.

Only the "main" JSON messages received from the stream receive process were
forwarded to our Unix pipe for the other processes (transform, then replay).
All of our messages should go through the pipe!

* Have pgcopydb follow use the prefetch/catchup - stream loop.

* Documentation coverage.

The KEEPALIVE and SWITCH WAL messages could be part of a
transaction (typically one that spans multiple WAL files), but also can be
found independently, between transactions.

Our previous internal representation insisted that every message be put in
the context of a transaction, but that doesn't fit well with the reality.

This refactoring is also necessary for streaming, where we don't want to
wait until the end of the next transaction before replaying the KEEPALIVE
messages. That kind of defeat the purpose of a KEEPALIVE message.

Make it easier to follow what happens in bare clone operations and what
happens in clone-and-follow operations.

* Implement pgcopydb stream replay.

The idea is to have an integrated command that does the same as the
following unix pipe command:

    $ pgcopydb stream receive --to-stdout
      | pgcopydb stream transform - -
	  | pgcopydb stream apply -

The command takes care of setting up the unix pipes and the process tree
internally. Makes it easier to use, and the implementation is needed for
intagration in the end-user API (pgcopydb clone --follow) anyway.

* Review follow module subprocess supervision and management.

* Review.

* Refactor the transform process management.

Make the transform process a sibling process to the receive and catchup
processes, where it was started by the receive process before. This
refactoring simplifies our process management and prepares for being able to
transition from the prefetch model to the live replay model.

* More refactoring, also name our queues.

Improve debugging and process management, and share more code to start the
follow sub-processes and supervise them, in a way that the same supervision
code applies to many of our modes of operations: prefetch, prefetch and
catchup, and soon, replay.

The main use case is in the test-suite where we can use our retry policy to
make sure the environment is ready for running the test script. Because it's
useful for the project in the context of its unit testing suite, it might be
useful also for any project using pgcopydb with some automation around it.

* Implement "live" streaming of changes from source to target.

This patch introduces support for Unix Pipes and an internal streaming API
where changes flow through the system rather than being batched in files.
The following command line can be used to replay changes in a "live" fashion
from the source to the target database.

  $ pgcopydb stream receive --to-stdout
    | pgcopydb stream transform - -
    | pgcopydb stream apply -

The command `pgcopydb stream replay` is scheduled to glue together those
internal parts, and then the `pgcopydb clone --follow` will be extended to
automatically switch from catchup mode to replay mode when suitable.

* Code refactoring to share more bits.

Share the code that reads through a unix pipe (stream) line by line,
introducing a user callback facility with a (void *) context.

* First pass at adding tests for our low-level API/CLI.

* Switch producers to line buffered output mode.

This avoids partial reads of a part of a line, now we read a line at a time,
unless it doesn't fit in our 128 MB read buffer, a case that is handled in
the code already (albeit quite poorly at the time being).

Also improve logging, install set -o pipefail in the test suite.

* Add the new test case to GitHub Actions CI.

* Fix internal API to the new refactored code.

This allows multiple unrelated subcommands to run at once, each with their
own service specific pidfile. Also, most sub-commands do not need a pidfile
at all and now can bypass checking for stale pids etc entirely.

The code was ready to be re-used already, but still copy-pasted. In the same
file. Well now at least it's cleaner.

Writing directly to the target JSON file creates a windows where the file
exists and can be read with partial content, either by pgcopydb itself or
from external tooling. Using a temp filename then switching to the actual
expected file protects again partial reads and parse failures.

Writing directly to the output file opens a window where the file exists and
can be read with partial content, either from pgcopydb itself, or from an
external tooling. Using a temp filename then switching to the actual
expected file protects again partial reads and parse failures.

* Allow bash scripts as test cases.

* Fix typo.

Address the failure when `stream cleanup` is used after `clone follow`
caused by `dir` is passed as `NULL` to `copydb_init_workdir`, which leads
`copydb_prepare_filepaths` to set `topdir` to an undesired location.

Includes cleanup of commands that do not require working directory, by
removing `copydb_init_workdir`. And, sets `dir` to appropriate value that
do requires working directory.

* Fix lsn for KEEPALIVE actions.

* Fix failing unit tests by adding fake K messages.

Should fix #172.

* Error out early when work dir does not exists and is expected to.

Auxilliary commands such as pgcopydb list progress expect the work directory
to have been created already. Check that early in the code.

* Fix copydb_init_workdir to specify if work dir should be created.

Then fix the callers and make most of them re-use a pre-existing directory
rather than create a default one and then use it.

Trivial: Write index OIDs as unsigned integers correctly to the summary file & fix partitioning for tables with bigint primary keys with values greater than 2^32 (#168)

* Write OIDs as unsigned integers correctly to the summary file

* Update partitioning logic to support bigint primary keys

* tests: Add test for scenario when endpos is set between BEGIN and COMMIT.

This test reproduces the issue mentioned in #154.

* Add new test to run-tests.yml

* Fix transform process to handle endpos in between transactions.

We might have a transaction that spans over multiple JSON files, and in that
case we always have a SWITCH WAL message at the end of the current file,
within the current transaction.

When we reach the end of file and we're in the middle of a transaction but
we don't have a SWITCH WAL message, it means that we have reached endpos. In
that case just ignore the currently opened transaction and replace its
contents by a synthetic KEEPALIVE message to advance the target system.

* Code cleanup.

* Review and fix to account for target case only...

Co-authored-by: Shubham Dhama <shubhamdhamaofficial@gmail.com>