To work with shorter timeouts in network stacks (firewall, NAT settings,
etc), adjust to TCP keepalive settings to quite short values.

* Set Process Titles as seen in ps/top/htop etc.

This helps understand the process hierarchy better and could be useful to
debug or watch long running processes.

* Append current table/index information when we have it.

* Fix escaping double-quotes in SQL identifiers.

When double-quoting SQL identifiers, any double-quote that is part of the
name should be doubled. Now, rather than implementing that ourselves yet
again in the C code, use format('%I', relname) as relname in the SQL
queries.

* Fix bugs.

* Update test files, the transform output has changed.

* Clean up PQExpBuffers in all usages

PQExpBuffers must be destroyed with `destroyPQExpBuffer`, otherwise the
bytes that they contain end up being leaked. In putting together this
patch I audited every call site of `createPQExpBuffer` and the subsequent
code in order to locate missing calls to `destroyPQExpBuffer`. Some
modifications may be classified as pedantic, but overall this patch
removes a number of real-world memory leaks.

* Fix memory leaks in wal2json transformation

The `stream_transform_file` function didn't free memory which was
allocated to contain the file contents.

The `FreeLogicalMessageTuple` function didn't correctly free all items
in its hierarchy.

* Implement same-table concurrency using Postgres CTID column.

The Postgres system column "ctid" is the physical location of the row
version within its table. It is encoded using the tid datatype:
`(pagenumber,rownumber)`.

Every single Postgres table has a "ctid" column and it is always possible to
split a table's contents by using ctid based ranges. The question of this
approach being good at reducing pgcopydb timings remains open: this PR is
meant for allowing more experimentations.

* Restrict CTID COPY partitionning to tables using "heap" am.

TID scan might not be supported by other Table AM (such as Citus Columnar).

This allows a way faster replay because we don't have to wait for Postgres
disk sync operation before sending the next SQL command. We need to be
careful with what replay_lsn value is sent back to the replication protocol,
and for that we need to introduce a tracking between the source LSN and the
replay insert LSN.

We still restart applying from the latest LSN that we manage to commit
durably using the replication origin API. The advantage of that API is that
it's as durable as the transactions replayed on the target system.

When a subprocess terminates with a successful return code, it might still
have been terminated by a signal, one signal would be SIGSEGV. Arrange our
code to report when that happens.

This happened in initialisation of the streaming module when trying to call
setvbuf on a un-assigned file descriptor. This is fixed in follow.c when
preparing the call.

* Introduce Bob Jenkins lookup3 hash function.

* Use PREPARE/EXECUTE statements for applying DML in follow mode.

When replaying INSERT/UPDATE/DELETE statements, switch to using PREPARE and
EXECUTE at the protocol level (using the libpq functions PQprepare and
PQexecPrepared). When a statement has already been prepared previously in
our session, we then only send the EXECUTE statement.

This shoud provide nice performance improvements.

* Fix ci/banned.h.sh.

Avoid using sscand(), use strtoull with a base 16 instead. In passing
clean-up some extra logging.

* Code review and refactoring.

* Suppress compilation warnings from Jenkins lookup3.c code.

Introduce a new C module "compare.c" and implement the work there, leaving
the "cli_compare.c" module with command line and output handling.

In passing, add the checksum information to the schema.json file at the end
of the pgcopydb compare data command, that might be useful.

That way pgcopydb compare data waits for as long as the slowest query that
are running concurrently on source and target instances, which is better
than serial execution on two different servers.

This changes the JSON format output, because we also now skip fetching the
target database catalogs.

First, the parsing of the schema and table names from the INI file was done
wrong, in a way that shows with long schema and table names.

Then, I just learned we can't use pg_restore --schema in our context,
because it would then skip the CREATE SCHEMA statement and pgcopydb relies
on pg_restore to create the schema on the target database,

* Implement concurrent workers for Large Objects data copy.

The default is 4 workers for Large Object data, and another process is
created to queue the Large Object metadata (oid) and allow workers to share
the workload. New option available: --large-objects-jobs.

* Fix blob summary.

Reinstall a blob summary file and change its format to JSON.

Use an output format that is stable in number of digits and can deal with a
bigint overflow (in Postgres a sum(bigint) is numeric): use an MD5 sum and
represent it as an UUID.

Also, include the row count in the MD5 computation to better protect against
collisions.

Finally, add support for pgcopydb compare data --json.

Implement a set of pgcopydb commands to compare databases, so that it's
possible to make sense of a success criteria for the migration. The current
implementation is pretty limited and provides a starting point in that
direction.

Despite the name, this filter is an exclusion filter and works exactly the
same as the existing "exclude-schema" filter. The new facility is syntactic
sugar and allows listing schemas to keep around, which is useful when
keeping a single schema and excluding the rest of them.

Instead of using a static buffer of BUFSIZE for building the SQL string, use
the PQExpBuffer API instead here too.

This option allows specifying which version of an extension to install on
the target database. The version might be different from the version on the
source, and in some cases installing the default version of an extension is
not desired. Examples are PostGIS v2 and v3 series, for instance.

A requirement file in JSON can be given to pgcopydb commands. The JSON file
should be an array of objects with the "name" and "version" keys, and can be
given to either of these commands:

    pgcopydb copy extensions --requirements requirements.json
    pgcopydb clone ... --requirements requirements.json

To list the available extension versions on the target database use the
following commands:

    pgcopydb list extension --available-versions
    pgcopydb list extension --available-versions --json

    pgcopydb list extension --requirements
    pgcopydb list extension --requirements --json

The last command will output all the extensions available on the target
database with their default version. It's then possible to edit the output
file and change the version numbers before giving it back to the copy and
clone commands.

A single sequence could be used by more than one table/column thanks to
calling the nextval() function in the default value definition of the
column. That said, a sequence is OWNED BY a single table.

In this patch we fix the case where the table that owns the sequence is
filtered out from the pgcopydb scope while another table that uses the same
sequence is to be copied over... with the sequence, then.

A simple way to create the situation is using the following DDL:

    create table app.foo(id bigserial, f1 text);
    create table copy.foo(like app.foo including all);

And then filter-out schema app from pgcopydb scope.

This resolves a problem encountered during the transition from prefetch to
replay mode, where the transform process identifies some incomplete
transactions. This issue also arises when we interrupt 'follow' and execute
a 'follow --resume'.

The root cause is upon resumption of streaming, we may encounter a
transaction where the initial messages have LSNs less than the startpos
calculated from the latest json file.

The implemented fix involves comparing the LSNs with 'startpos' only when
resuming streaming of partially complete transactions. Once 'startpos' is
reached, comparison of LSNs for subsequent messages is discontinued.

Fixes: #373

Should fix #317.

First, when interrupting the pgcopydb process interactively with C-c while
the transform process is running, we should exit the follow loop cleanup.

Second, when using pgcopydb clone --follow --restart we should do a
replication setup cleanup first, the same as the pgcopydb stream cleanup
command, so that we can actually restart from scratch as the command says.

First, Postgres type names might contain square brackets (arrays), which are
used as a delimited in the test_decoding format for typenames. Fix the code
to handle the following properly:

    f2[text[]]:'{incididunt,ut,labore,et,dolore,magna}'

Then, it turns out that test_decoding emits text in standard SQL form, which
is to say in single-quotes and with single-quotes doubled, but with the \n
and other characters not-escaped. We need to edit the representation a
little so that the ld_transform module processes the text correctly.

See #323.
Fixes #377.

The new commands allow listing extension versions in different ways and
formats (human-friendly table, JSON):

  $ pgcopydb list extensions --available-versions
  $ pgcopydb list extensions --requirements

The idea is to prepare for a feature where the extension versions to install
on the target database can be provided by the user. To prepare for that, the
new commands allow listing available extension versions on the target
database and preparing the current requirements file from the extensions
available on the target and their default version.

The connection strings are now dynamically allocated but cli_stream somehow
didn't get the memo so far.

* Implement multi-parts read from Unix PIPE.

A Unix PIPE is typically 64kB large and we might have JSON messages that
won't fit in that. The size of the buffer filled by the read() call used to
be fixed at 128MB, but that doesn't have an impact on how much data we can
receive in a single read() call actually.

Fix that properly and allocate the size of the PIPE buffer thanks to a call
to fnctl(fd, FIONREAD) which seems to be well supported. Then use the libpq
string buffer facility to build the messages in memory when receiving them a
piece at a time from the Unix PIPE buffer, in multiple read() calls.

* Ensure our read_from_stream buffer terminates properly.

C-string must be terminated by '\0' and the read() call is not ensuring
that.

The list command does not use a --target connection string, so we skip
preparing the target catalogs in that case. In passing, also fix a strange
way that PQgetisnull() was done for sequences, as the misleading error meant
this code got a free round of review.