To prevent limitations around lock management when running pgcopydb on a
source database with lots of objects, we now benefit from --not-consistent
as an opportunity to reduce long-running transactions.

It was already the case that --not-consistent wouldn't require exporting or
setting a snapshot, but the opportunity was not yet implemented fully.

* Implement filtering the pre-data section.

When installing the schema on the target database, we want to already filter
out the SQL objects that are filtered-out in the filtering setup. The
previous coding forgot about that parts.

See #43.

* Implement support for exclude-schemas.

The parsing was implemented already, but the filtering itself was shy of a
brick load. We need to both filter tables that belong to the schema (we did)
and also the schema object itself (we didn't).

* Add missed SQL objects types to parse from pg_restore archive list.

* Add ACL as a pg_restore archive catalog description entry.

Unless the PGAPPNAME environment variable has already been set, in which
case we would just use the existing value. This allows users to force their
own application_name either in the environment or in the connection string.

* A pass at improving pgcopydb docs.

* Add coverage for the new filtering commands.

We still have a problem with how the version string is computed in the
Docker image, where there is no .git environment and then no tags. The
script from the git git repository uses an extra "version" file for
releases, and that's not what we're doing at the moment.

We might want to produce the version file on the local machine and then send
it in the Docker environment, but at the moment, let's just keep it simple.

The user in connection must have a superuser role to access the pg_authid
table, which may not be possible for managed services. The fix here is to
use publicly readable view pg_roles
https://www.postgresql.org/docs/current/view-pg-roles.html.

Filtering support is important to have to debug the pg_restore archive
catalog editing that pgcopydb implements.

In particular fix the debian package build, and use dpkg-parsechangelog in
that environment so that we report the debian/changelog version string.

* Add versioning information to the logs output.

For that to be useful, we also integrate with the git scripts from git
itself to compute a trustworthy version string such as 0.5.5.gaea0ff08.dirty
when we have local changes are are building from a git checkout.

* Implement a parser for filtering settings, and filter table lists.

* Apply filters to listing sequences.

We filter only those sequences that have a link dependency to the selected
tables. There is no way at the moment to select a sequence that's not
tracked to a specific default value for a column that doesn't belong to one
of the selected tables.

That said, when filtering is not used, all sequences are processed, even
those not attached to any table. Which means that `pgcopydb copy sequences`
in the use case where filtering is needed for tables, but not for sequences.

* Implement pgcopydb list ... --list-skipped.

This option allows debugging the filtering setup, and is also needed to
implement pg_restore catalog editing (--use-list) to avoid installing
objects that are filtered-out in the setup.

* Document filtering setup.

* Add a test for the filtering capabilities.

* Improve filtering of pg_restore list entries without an OID.

Some pg_restore --list entries such as INDEX ATTACH miss both the catalog
and the object oid, and then we need to match them by their pg_restore list
name, which is a compound of the schema name, object name, and owner name.

Adding to that, the way pg_restore builds that compound name is with using a
single space as a separator, and replacing \n and \r characters with a
single space. This makes the pg_restore list output unfriendly to machine
parsing, and so instead we generate the list name from the Postgres catalogs
in our catalog queries.

Then we can use an hash table on the OIDs and another one on the compound
names and find out if an INDEX ATTACH catalog entry refers to an index that
has been filtered-out by filtering rules, and comment it out then.

* Add a pg_depend recursive walker facility.

This allows filtering from pg_restore --list objects that depend on tables
that have been filtered out.

A good optimization when using COPY is to use TRUNCATE in the same
transaction, so as to be able to then use COPY FREEZE and avoid some VACUUM
activities after the bulk loading of data.

Trouble is that with inherited tables, the TRUNCATE command would also
remove all the data that was (maybe) just copied to the inheritance tree
nodes.

To fix, just use the TRUNCATE ONLY variant of the command.

This commit also adds full regression testing for this issue, thanks to the
test case provided in #33.

* Implement Unit/Regression testing suite.

At the moment it's quite a simple test suite, and only covers the exclusion
constraint issue #22. As more specific issues are opened, we can add to this
regression test suite.

* Add SQL based checking.

To ensure that the test is passing, it is now possible to run SQL tests and
compare the result from the registered expected output.

The default connection timeout was raised from 2s to 10s in a previous
PR (#28), but the change was not applied to the timeout that we set in the
environment before calling Postgres client applications.

In our context we need to connect through unknown network layers, the
default of 2s is better suited for a known local connection. 10s seems a
good trade-off for pgcopydb.

The auxiliary process for large objects would somehow piggyback on the main
snapshot connection instance, so make sure it's using its own private
connection.

Since re-using the same connection throughout a COPY process we switched to
multi statements connections, and those need to be closed explicitly. We
missed that in the previous coding.

Using SSL as in debian by default shows more errors, so make sure to run at
least one of our docker compose based testing in SSL too.

Move the copy of sequences data to a proper place that matches the comment
and refactor some function names to distinguish between async behaviour (use
_start_ in the name then) and sync behaviour. Recent changes made that
distinction hard to follow.

Use TMPDIR instead.

I picked a lot of the C lower level infrastructure from pg_auto_failover,
which I did under the Postgres license (so that's fine) and at Microsoft,
who has the copyright on that parts then.

* Fix support for exclusion constraints.

* Include ALTER TABLE ADD constraint timing to the summary.

* Improve a summary label.

* Bug fix: recognize index supporting constraints.

The "constraint" boolean arg is about our intend to either create the index
or the constraint (that maybe attached to the index). Our decision making
about the concurrent building of indexes should apply to building an index
that is baking a constraint that's neither PK or UNIQUE, we can't re-use our
"constraint" boolean for that decision making.

* Refactoring: fix a function's name.

Using the start_ prefix in the function name would imply async behavior
where copydb_create_table_indexes actually waits until all the indexes have
been built before returning. Change its name.

* Refactor code to make it easier to read.

* pgcopydb copy blobs

New command to support copying large objects from the source to the target
database instances, in a streaming fashion: we avoid spilling the large
object data to disk altogether.

* Integrate an auxiliary process in the main table-data part.

Now the large objects are automatically copied over from the source to the
target database. The concurrency is limited to 1 by the way the code is
written at the moment.

At the minimum we want to use the same client_encoding on both connections,
so that we know that we are dealing with the same data without having to
actually look into it.

Then, for bulk-loading and index creation, some settings are easy to set on
the target connection to get a better perf profile (e.g.
maintenance_work_mem).

* Allow using --resume with --snapshot without --not-consistent.

When the snapshot has been kept around in between two runs of pgcopydb we
can still ensure consistency and use --resume and --snapshot.

* Improve --resume --snapshot ... processing.

Make the processing more robust when using `pgcopydb copy ...` commands and
refactor some code to share more of the tooling. Specifically, all the
commands now share the same getopt structure and support the same
environment variables.

* Add proper support for --snapshot for pgcopydb copy ... subcommands.

* Fix pgcopydb copy constraints.

Instead of using the `pgcopydb copy data` approach focused on tables, and
then for each tables the indexes, and then when all the indexes of a given
table are finished creating the constraints; simplify and re-use the index
only code path from `pgcopydb copy indexes` taking care of constraints.

This means indexJobs processes are created and share the list of constraints
to build with the usual locking approach.