Co-authored-by: Hanefi Onaldi <Hanefi.Onaldi@microsoft.com>

Follow up of #595

This allows to ensure we're always using a known version of the SQLite lib
and allows pgcopydb code to use some "recent" features on all OS targets.

Such features include the RETURNING clause for DMLs, the FILTER clause for
aggregate function calls, and time formatting function or JSON functions.

See https://sqlite.org/amalgamation.html for details about that distribution
mode for SQLite.

Before this change we had 153 files on the main branch that did broke at
least one of our editorconfig rules. After this change, all of our files
honor the rules that are partially updated.

There are 2 places that are exempt from our rules:
- external libraries in src/bin/lib/
- debian directory in debian/
- test file from outside source in tests/extensions/

You can see the .editorconfig file in each of these directories for more
details.

I used a copy of Citus' CONTRIBUTING.md as a starting point.

I removed sections that are not relevant to pgcopydb, and added some
more sections about documentation and running tests.

I also added references to the CONTRIBUTING.md file in the README.md and
in documentation page about installing pgcopydb. While I was at it, I
also removed the outdated installation instructions from the README.md
file, and replaced them with a reference to the documentation page.

Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>

* Fix confusion around reporting operations in progress.

The SQLite refactor meant rewriting all the SourceTableArray loops to use a
query result iterator instead. Well there is a place for a SourceTableArray
loop still, and that's the pgcopydb list progress thing.

* Fix the table-in-copy SQLite query.

The query was meant to report actual duration and transmitted bytes for each
table in-progress, but the join and sum() operations are creating problems
in the SQLite query and the rest of the progress.c code was not prepared to
benefit from the extra information anyway.

The SQLite problem is that the sum(s.duration), sum(s.bytes) causes a single
row output when the dataset is empty, and the other columns are all set to
NULL. That might be because of the lack of a GROUP BY clause.

Simplify that aspect by removing the extra information for now.

Since the move of the summary and process information to the SQLite
catalogs, we don't have a progress JSON file on-disk with parts of the setup
anymore.

The code needed to be updated to behave correctly without having
--table-jobs and --index-jobs information.

* Use Boehm-Demers-Weiser Garbage Collector.

From their own documentation:

> The Boehm-Demers-Weiser conservative garbage collector can be used as a
> garbage collecting replacement for C malloc or C++ new. It allows you to
> allocate memory basically as you normally would, without explicitly
> deallocating memory that is no longer useful. The collector automatically
> recycles memory when it determines that it can no longer be otherwise
> accessed.

In pgcopydb C code base it's been hard to maintain proper malloc/free
concerns while also implementing proper error-handling, which is a common
problem when writing C code. In order to ease the maintenance of the code
and reduce production hazards, the best way to proceed looks like automating
the memory management altogether by using a Garbage Collector.

* Revert removal of regree() calls.

* Get rid of LinesBuffer.ownsBuffer.

Now that libgc is handling the memory for us, we don't need to keep track of
the ownership of the buffer anymore.

I observed frequent `[SQLite] PRAGMA journal_mode = WAL` when it should be
run only once as it's a sqlite database property, not something that has be
set for every connection. This might fix
https://github.com/dimitri/pgcopydb/actions/runs/7614380466/job/20736769276?pr=647

```
2024-01-22 16:45:01.923 44 ERROR  catalog.c:7129            [SQLite]: PRAGMA journal_mode = WAL failed: database is locked
```

- Adds CI to check that the documentation templates are up-to-date.
- Add a new `help` command to the documentation templates. This command
  prints the help message for all the sub-commands of `pgcopydb` in a
  single file. This is useful for users who want to see all the
  sub-commands in one place.
- Remove dependency on installing the binary before updating the
  documentation templates. This allows users to update the documentation
  templates without installing the binary. One can set the `PGCOPYDB`
  environment variable to point to the binary to use for updating the
  documentation templates.
- Fix a casing issue in command descriptions. The first letter of the
  first word in the description of each command should be capitalized.

When registering the catalog setup, which includes the same-table
concurrency threshold, only consider that value when the table partitions
have been populated in the cache already. Otherwise the table partitions are
going to be computed again anyway.

* Improve SQLite write performance with journal_mode as WAL.

* Batch initial catalog setup write in a single transaction.

The SQLite patch series introduced a bug where we would skip extensions even
when the option --skip-extension is not used.

Also adjust the "pgcopydb copy extensions" to the recent changes where we
use a sub-process under a supervisor to simplify snapshot handling.

Make sure we don't try and allocate a zero-size memory area, which can lead
to "corrupted size vs. prev_size". Review all the calloc() call sites and
protect them better when they need to be.

Make it systematic to first count the lines and then allocate the right
amount of memory for the array of pointers to the start of each line, and
make sure we don't try and allocate a zero-size array, which can lead to
"corrupted size vs. prev_size" errors at run-time.

Unfortunately -Wshadow=compatible-local does not seem to be available when
using llvm as the compiler. Still, debian builds for Postgres community are
producing warnings that we can fix.

It turns out that when using just "exclude-index" in our filtering the code
would be confused as to how to implement that filtering option for listing
table and sequence objects from the Postgres catalogs.

* Use SQLite for LSN tracking.

* Ensure we open the catalogs in pgcopydb stream apply.

This makes it possible to see the timings while pgcopydb is operating,
including timing of top-level operations and cumulative timings.

Solution: Populate extension configuration objects along with their type
values. Based on the type value, perform either table copy or sequence copy.

At the moment we support only copying extension configuration tables & sequences.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Remove root_with_debug CommandLine that is identical to root

* Fix numerous issues with argument parsing

The following issues were fixed:
- some options were not recognized
- some nonexistent options were recognized
- some options with required arguments were not expecting an argument
- some commands did not print help messages when given invalid options

Solution: Use flush LSN from target to find the durable LSN.

To find a durable LSN with async commit, we maintain a list of LSN mapping
between source(sourceLSN) and target(insertLSN). New mapping will be
inserted into the list on COMMIT or KEEPALIVE message.

While reporting the replay_lsn to sentinel we have to get the flush lsn from
target and compare it against the insertLSN from the list to find the sourceLSN.

This commit introduces a shell script that runs all commands that accept
`--help` parameter and stores the result in template files that are
referenced in our documentation. You can use that script by running
`make update-docs` in top directory of the project.

Usage: `make DEBUG=1 install`

Introduce a new query to count the number of tables and indexes already
processed from the summary table, fixing the list progress output for items
"done".

* Use our SQLite catalogs for the pgcopydb sentinel table.

The main use-case for the sentinel table is to communicate progress made by
the three independent worker processes for stream, transform, and replay.
It is also needed to communicate new startpos, endpos and apply values with
those processes from another pgcopydb (interactive) command, a process
that's not in the process tree.

This was previously solved by creating and using a pgcopydb.sentinel table
on the source database. Now that we have a SQLite database internal to the
pgcopydb run, we can use that instead.

* More work on the sentinel table move to SQLite catalogs.

In particular, we need to share a common directory mount point in the test
and inject containers, because the inject container is setting the endpos
for the test container where the main tests are running.

For that, in my testing the best option is to use a host directory. This
also makes it very easy to then use sqlite3 and inspect what's going on from
the outside.

* Attempt to fix GitHub Action host workdir perms

* Fix follow-data-only tests: update sentinel at stream flush time.

* Assorted fixes, per self-review.

The copy-db command is deprecated and will get removed from pgcopydb on
next release, please upgrade your scripts and integrations.