This is the first step towards support for REFRESHing materialized views
within pgcopydb instead of in pg_restore.

* Fix working directory for pgcopydb stream cleanup.

* Add CHECK CONSTRAINT archive entry mapping.

* Fix success return of `copydb_target_finalize_schema` on failure.

* Add CHECK CONSTRAINT unit test

* Add LANGUAGE archive entry mapping

* Link issue in unit test

---------

Co-authored-by: Shubham Dhama <shubhamdhamaofficial@gmail.com>

* Fix typo: ther -> the

* Consistent casing in help messages

* Update versions in docs to latest

* Consistent forked process names in design docs

* Update package list in rockylinux

* Fix indentation in pgcopydb help message

* Fix typo: Create and exports -> Create and export

* Update some version outputs

* Fix typo

* Update version output using official docker image

The `read_from_stream` function in pgcopydb was encountering a data loss issue
when reading from a source system with transaction bodies containing more than
1024 inserts. The problem arose from the statically allocated array of size 1024
used to split lines within the function. When the buffer held more than 1024
lines, it caused subsequent lines to be ignored, resulting in data loss.

To resolve this issue, this commit introduces the following changes:
- We now count the number of lines in the buffer before splitting them.
- Dynamically allocate an array to hold the split lines, based on the line count.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

This change will make sure that pgcopydb can write to the database even
if the user has set the default_transaction_read_only GUC.

When an outside contributor submits a PR, the test runs fail because
they have the commits on a fork, and there is no easy way to get the
commits from the fork into the main repository. This change makes the
test runs work for outside contributors by always checking out the SHA
for the event.

We no longer need to have separate steps for PRs and non-PRs as
checkout@v3 has sane defaults for both. Below is an excerpt from the
docs of this action that explains the behavior of ref values:

  The branch, tag or SHA to checkout. When checking out the repository
  that triggered a workflow, this defaults to the reference or SHA for
  that event. Otherwise, uses the default branch.

* Switch to using libpq async API.

The easy to use API PQexec() and PQexecParams() are blocking, which means
those functions could fail to detect when a Postgres connection has been
lost, and now wait an indefinite amount of time.

Switch to using the async API PQsendQuery() or PQsendQueryParams() with the
PQgetResult() function, all with some lower-level socket API calls such as
select(2) and PQconsumeInput() and PQisBusy().

Note that when the server is no longer busy computing the result, we should
loop over all the available results... apparently.

* Code cleanup, add another check for interrupts.

* Remove extra check for interrupts.

The Postgres tooling for configuration variable settings that are specific
to a database is only activated when pg_restore --create is used, which
means that pg_restore is in charge of CREATE DATABASE.

In the context of pgcopydb, we do not want pg_restore to create the target
database for us, but we still want to copy over the database specific
settings that have been applied with either of the following commands:

    ALTER DATABASE SET
    ALTER ROLE IN DATABASE SET

To achieve that we need to copy some code from the pg_dump and pg_restore
implementation and along with it we also vendor-in some of the Postgres
common code (a part of string_utils.c) to avoid link-time issues on
different packaging systems.

* Improve error handling when COPY fails.

In the normal case we want the COPY process to continue with the rest of the
tables in the queue, and in the fail-fast case we want to bubble-up the
problem as fast as possible for a quick total failure.

The partial failure in the normal case can be resumed with the existing
option such as --resume, as long as the snapshot still is held, or when
using the option --not-consistent.

Also, ensure that when errors happen mid-flight in the COPY processes we
still send the STOP messages in the queues, or signal all the other
processes to terminate.

* Also review error handling for vacuum/create index queues.

Now that we grab the Postgres identifier names already quoted, we need to
expand our internal structures memory to handle more bytes: the quotes
around the names, and also any double-quote char within the name is going to
be doubled by the quoting rules.

This call should not be blocking in our implementation, because we want to
be able to react to network problems or server-side interruptions.

The test_decoding plugin escapes identifiers using double quotes for
column names. However, the pgcopydb transform process also escapes
identifiers in the same way, leading to double escaping and apply process
failures.

This commit addresses the issue by adding double quotes only when the
column name is not already quoted.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Add tests coverage for Postgres 16.

* Fix GitHub Actions workflow file.

* Rewrite the ad-hoc parser for Postgres Archive TOC items.

This time use a tokenizer and hand-roll a grammar/parser on-top of it,
expecting tokens in the right order. We have a pretty simple grammar at the
moment, because we don't extend the support to the ACL and COMMENT composite
types. We might want to do that later.

* Reformat parsing of ACL and COMMENTS too.

* Free allocated memory for parsing pg_restore archive catalogs.

* Fix rebase.

* Protect against NULL pointer dereferencing.

* Fix string size compute bug (include 2 bytes for prefix).

Co-authored-by: lospejos <lospejos@github.com>

The following command will now parse the pg_restore --list output file and
output to standard out the parsed contents, allowing to check the parser.

    pgcopydb restore parse-list pre.list

This commit optimizes multi-value inserts by transferring insert values
from a new statement to an existing one if they target the same table.
This optimization significantly reduces network latency when multiple consecutive
insert statements are executed within a transaction.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

That way it's possible to review the pg_restore file separately from the
filtered file that we process in pgcopydb, and because both the files are
left around at the end of the command, it's also possible to diff them.

Using the Postgres function format with the %I formatter allows our code to
bypass any formatting internally when re-using the SQL object names, both in
our logs and also in the SQL queries we then emit.

Generalize that approach to attribute names, index names, index table names,
index constraint names, and sequences names.

When failing to start a sub-process (out-of-memory can be reported when
calling the fork() system call), properly handle the error rather than
continue trying the migration.

In order to cancel the other sub-processes that were succesfully started we
then send a TERM signal to all processes in our process group.

* Improve connection string TCP keepalive parameters handling.

Allow overriding at the command-line level, also set the parameters in the
connection string so that they're effective even when connecting.

* Fix computing of the safe URI: always do it now.

To work with shorter timeouts in network stacks (firewall, NAT settings,
etc), adjust to TCP keepalive settings to quite short values.

* Set Process Titles as seen in ps/top/htop etc.

This helps understand the process hierarchy better and could be useful to
debug or watch long running processes.

* Append current table/index information when we have it.

* Fix escaping double-quotes in SQL identifiers.

When double-quoting SQL identifiers, any double-quote that is part of the
name should be doubled. Now, rather than implementing that ourselves yet
again in the C code, use format('%I', relname) as relname in the SQL
queries.

* Fix bugs.

* Update test files, the transform output has changed.

* Clean up PQExpBuffers in all usages

PQExpBuffers must be destroyed with `destroyPQExpBuffer`, otherwise the
bytes that they contain end up being leaked. In putting together this
patch I audited every call site of `createPQExpBuffer` and the subsequent
code in order to locate missing calls to `destroyPQExpBuffer`. Some
modifications may be classified as pedantic, but overall this patch
removes a number of real-world memory leaks.

* Fix memory leaks in wal2json transformation

The `stream_transform_file` function didn't free memory which was
allocated to contain the file contents.

The `FreeLogicalMessageTuple` function didn't correctly free all items
in its hierarchy.

* Implement same-table concurrency using Postgres CTID column.

The Postgres system column "ctid" is the physical location of the row
version within its table. It is encoded using the tid datatype:
`(pagenumber,rownumber)`.

Every single Postgres table has a "ctid" column and it is always possible to
split a table's contents by using ctid based ranges. The question of this
approach being good at reducing pgcopydb timings remains open: this PR is
meant for allowing more experimentations.

* Restrict CTID COPY partitionning to tables using "heap" am.

TID scan might not be supported by other Table AM (such as Citus Columnar).

This allows a way faster replay because we don't have to wait for Postgres
disk sync operation before sending the next SQL command. We need to be
careful with what replay_lsn value is sent back to the replication protocol,
and for that we need to introduce a tracking between the source LSN and the
replay insert LSN.

We still restart applying from the latest LSN that we manage to commit
durably using the replication origin API. The advantage of that API is that
it's as durable as the transactions replayed on the target system.

When a subprocess terminates with a successful return code, it might still
have been terminated by a signal, one signal would be SIGSEGV. Arrange our
code to report when that happens.

This happened in initialisation of the streaming module when trying to call
setvbuf on a un-assigned file descriptor. This is fixed in follow.c when
preparing the call.

* Introduce Bob Jenkins lookup3 hash function.

* Use PREPARE/EXECUTE statements for applying DML in follow mode.

When replaying INSERT/UPDATE/DELETE statements, switch to using PREPARE and
EXECUTE at the protocol level (using the libpq functions PQprepare and
PQexecPrepared). When a statement has already been prepared previously in
our session, we then only send the EXECUTE statement.

This shoud provide nice performance improvements.

* Fix ci/banned.h.sh.

Avoid using sscand(), use strtoull with a base 16 instead. In passing
clean-up some extra logging.

* Code review and refactoring.

* Suppress compilation warnings from Jenkins lookup3.c code.

Introduce a new C module "compare.c" and implement the work there, leaving
the "cli_compare.c" module with command line and output handling.

In passing, add the checksum information to the schema.json file at the end
of the pgcopydb compare data command, that might be useful.

That way pgcopydb compare data waits for as long as the slowest query that
are running concurrently on source and target instances, which is better
than serial execution on two different servers.

This changes the JSON format output, because we also now skip fetching the
target database catalogs.

First, the parsing of the schema and table names from the INI file was done
wrong, in a way that shows with long schema and table names.

Then, I just learned we can't use pg_restore --schema in our context,
because it would then skip the CREATE SCHEMA statement and pgcopydb relies
on pg_restore to create the schema on the target database,

* Implement concurrent workers for Large Objects data copy.

The default is 4 workers for Large Object data, and another process is
created to queue the Large Object metadata (oid) and allow workers to share
the workload. New option available: --large-objects-jobs.

* Fix blob summary.

Reinstall a blob summary file and change its format to JSON.

Use an output format that is stable in number of digits and can deal with a
bigint overflow (in Postgres a sum(bigint) is numeric): use an MD5 sum and
represent it as an UUID.

Also, include the row count in the MD5 computation to better protect against
collisions.

Finally, add support for pgcopydb compare data --json.