From their own documentation:

> The Boehm-Demers-Weiser conservative garbage collector can be used as a
> garbage collecting replacement for C malloc or C++ new. It allows you to
> allocate memory basically as you normally would, without explicitly
> deallocating memory that is no longer useful. The collector automatically
> recycles memory when it determines that it can no longer be otherwise
> accessed.

In pgcopydb C code base it's been hard to maintain proper malloc/free
concerns while also implementing proper error-handling, which is a common
problem when writing C code. In order to ease the maintenance of the code
and reduce production hazards, the best way to proceed looks like automating
the memory management altogether by using a Garbage Collector.

When registering the catalog setup, which includes the same-table
concurrency threshold, only consider that value when the table partitions
have been populated in the cache already. Otherwise the table partitions are
going to be computed again anyway.

* Improve SQLite write performance with journal_mode as WAL.

* Batch initial catalog setup write in a single transaction.

The SQLite patch series introduced a bug where we would skip extensions even
when the option --skip-extension is not used.

Also adjust the "pgcopydb copy extensions" to the recent changes where we
use a sub-process under a supervisor to simplify snapshot handling.

Make sure we don't try and allocate a zero-size memory area, which can lead
to "corrupted size vs. prev_size". Review all the calloc() call sites and
protect them better when they need to be.

Make it systematic to first count the lines and then allocate the right
amount of memory for the array of pointers to the start of each line, and
make sure we don't try and allocate a zero-size array, which can lead to
"corrupted size vs. prev_size" errors at run-time.

Unfortunately -Wshadow=compatible-local does not seem to be available when
using llvm as the compiler. Still, debian builds for Postgres community are
producing warnings that we can fix.

It turns out that when using just "exclude-index" in our filtering the code
would be confused as to how to implement that filtering option for listing
table and sequence objects from the Postgres catalogs.

* Use SQLite for LSN tracking.

* Ensure we open the catalogs in pgcopydb stream apply.

This makes it possible to see the timings while pgcopydb is operating,
including timing of top-level operations and cumulative timings.

Solution: Populate extension configuration objects along with their type
values. Based on the type value, perform either table copy or sequence copy.

At the moment we support only copying extension configuration tables & sequences.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Remove root_with_debug CommandLine that is identical to root

* Fix numerous issues with argument parsing

The following issues were fixed:
- some options were not recognized
- some nonexistent options were recognized
- some options with required arguments were not expecting an argument
- some commands did not print help messages when given invalid options

Solution: Use flush LSN from target to find the durable LSN.

To find a durable LSN with async commit, we maintain a list of LSN mapping
between source(sourceLSN) and target(insertLSN). New mapping will be
inserted into the list on COMMIT or KEEPALIVE message.

While reporting the replay_lsn to sentinel we have to get the flush lsn from
target and compare it against the insertLSN from the list to find the sourceLSN.

This commit introduces a shell script that runs all commands that accept
`--help` parameter and stores the result in template files that are
referenced in our documentation. You can use that script by running
`make update-docs` in top directory of the project.

Usage: `make DEBUG=1 install`

Introduce a new query to count the number of tables and indexes already
processed from the summary table, fixing the list progress output for items
"done".

* Use our SQLite catalogs for the pgcopydb sentinel table.

The main use-case for the sentinel table is to communicate progress made by
the three independent worker processes for stream, transform, and replay.
It is also needed to communicate new startpos, endpos and apply values with
those processes from another pgcopydb (interactive) command, a process
that's not in the process tree.

This was previously solved by creating and using a pgcopydb.sentinel table
on the source database. Now that we have a SQLite database internal to the
pgcopydb run, we can use that instead.

* More work on the sentinel table move to SQLite catalogs.

In particular, we need to share a common directory mount point in the test
and inject containers, because the inject container is setting the endpos
for the test container where the main tests are running.

For that, in my testing the best option is to use a host directory. This
also makes it very easy to then use sqlite3 and inspect what's going on from
the outside.

* Attempt to fix GitHub Action host workdir perms

* Fix follow-data-only tests: update sentinel at stream flush time.

* Assorted fixes, per self-review.

The copy-db command is deprecated and will get removed from pgcopydb on
next release, please upgrade your scripts and integrations.

The waitpid() API provides the process return code, which can still be zero
when the processed was terminated by a signal such as SIGSEGV. Improve our
sub-process supervision with taking that into account as an unexpected error
condition.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Remove summary files, use our SQLite database instead.

* Get rid of summary files for the indexing and constraints too.

This allows also to entirely remove the need for semaphores to handle
concurrency between worker processes, using SQLite insert-or-ignore for
concurrency control instead.

Also the idea of TablePaths and IndexPaths elements are gone, with the lock
files and the done files mechanisms.

* Protect SQLite concurrent write access with a semaphore.

SQLite has not been designed for write-write concurrency, so implement a
critical section in our code so that SQLite only sees a single writer at all
times. It may still see multiple readers though.

Because we have re-entrant queries (an SQLite iterator function may then
choose to run another SQLite query), this commit also adds support for
re-entrant semaphore in a way that doesn't call into semop(2) again when we
already are in the critical section.

* Assorted fixes.

When the COPY table-data workers fail before the iteration over the list of
tables is done, we have a race condition where the COPY supervisor would be
busy and is not using waitpid(), thus not reacting to worker process
termination.

To solve, limit the responsibilities of the COPY supervisor process even
more and create a new process that's responsible for iterating over the
table list and sending tables/parts to the queue, etc.

The code to call TRUNCATE ONLY was only active for tables setup with COPY
partitioning (--split-tables-larger-than), and was forced to "false" for the
other tables.

The code also evolved in a way that it would build a sub-SELECT query with
the table attribute list as the "srcQname" argument to the pgsql.c internal
bits for the copy, making it impossible to then re-use that bit in the sql
command for the TRUNCATE operation.

To fix, implement a new way to prepare the COPY query bits and pieces so
that we can re-use the qualified table name in the TRUNCATE command and also
build a full COPY (SELECT a, b, c FROM ONLY t WHERE ...) TO STDOUT; query.

In passing, because of the way we are changing the summary files to use our
internal SQLite database instead, also push the TRUNCATE command of COPY
partitionned table to the COPY supervisor process. This is a better way to
solve the concurrency issues and make sure that TRUNCATE is done only once,
and also finished before any of the copy-data worker processes get started.

* add quotes identifiers

* test changes

* Fix escaping of identifiers while transforming for wal2json plugin.

wal2json doesn't escape identifiers as test_decoding do. To fix this issue
we are using PQescapeIdentifier function. Even though this function doesn't
require PGconn to be connected, but creating an empty PGconn object was
unfeasible. Some fields of PGconn that are required by new versions of
libpq for PQescapeIdentifier are not available in older PGconn. So decided
to do this the right way, but creating PGconn with an actual server
connection. It turned out more simpler than managing a dummy PGconn object.

* Fix review comments.

* Fix style.

---------

Co-authored-by: VaibhaveS <vaibhavedavey@gmail.com>

test_decoding transform relies on source catalog to decode the update message.

Prior to this commit, update while using test_decoding would work only while doing `pgcopydb clone --follow`. This commit enables support for update message decoding
while using test_decoding plugin in `pgcopydb follow` too.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

We close the FILE which is used to log into file before calling
the `copydb_cleanup_sysv_resources` which still uses log functions leads
to crash due to invalid FILE* access.

This commit fixes the crash by closing the FILE after calling the
cleanup function.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

We track resources likes semaphore and queue into a global array and
deallocate the non freed objects using exit handler. The array tracking
these resources uses pointers, which could inadvertently track resource
handles allocated from the stack.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

The filter clause is not supported by my local SQLite for some reason,
so I needed to use a different approach to count the number of
constraints on a table.

This fixes the following error:

    ERROR  Failed to prepare SQLite statement: select count(1) as indexes, count(1) filter(where c.oid is not null) OVER () as constraints from s_index i left join s_constraint c on c.indexoid = i.oid where tableoid = $1
    ERROR  [SQLite] misuse of aggregate: count()
    ERROR  Failed to count indexes attached to table dist_tables.t1
    ERROR  Failed to copy data for table with oid 17226 and part number 0, see above for details

Relevant documentation at https://www.sqlite.org/lang_aggfunc.html says:

The count(X) function returns a count of the number of times that X is not NULL in a group