Given ALTER TABLE ... DROP COLUMN ... we could well have a table without an
attribute with attnum == 1. Our code used to rely on that to happen in order
to allocate our SourceTableAttribute array in-memory, before filling it in.

Instead, use a window-function in the SQL query that we run against our
internal SQLite catalogs, using count(*) over(order by attnum) to map
whatever the Postgres attnum are into a proper gapless series that starts at
one.

This function does not exists anymore, but the header was still defined and
the call site was not removed yet.

Instead of storing the table size in the source database, store it in our internal catalogs in the SQLite database. This allows better compatibility with read-only source databases.

At the moment this removes the possibility for the user to provide their own table size data in order to avoid the `pg_table_size()` query which is known to be a long running query in some platforms. A later improvement could be done to enable external table size input again.

Missing xid triggers the following error during apply/replay.

"BUG: parseTxnMetadataFile is called with transaction xid: 0"

Solution: Use xid from commit message as a transaction xid.

Unlike wal2json, test_decoding don't have xid in the DML logical
messages. So we use the xid from the COMMIT message to update the
transaction xid.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

Solution: fflush stdout after printing the snapshot id. By default stdout is
buffered and flushed only when the buffer is full.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

Before embedding SQLite we could not depend on aggregates FILTER clause to
be available, and as a result we had to aggregate the attribute list in our
own C code. Turns out we had strange bugs around that, that hopefully the
very simple SQL query will not have.

Follow up of #595

This allows to ensure we're always using a known version of the SQLite lib
and allows pgcopydb code to use some "recent" features on all OS targets.

Such features include the RETURNING clause for DMLs, the FILTER clause for
aggregate function calls, and time formatting function or JSON functions.

See https://sqlite.org/amalgamation.html for details about that distribution
mode for SQLite.

Before this change we had 153 files on the main branch that did broke at
least one of our editorconfig rules. After this change, all of our files
honor the rules that are partially updated.

There are 2 places that are exempt from our rules:
- external libraries in src/bin/lib/
- debian directory in debian/
- test file from outside source in tests/extensions/

You can see the .editorconfig file in each of these directories for more
details.

I used a copy of Citus' CONTRIBUTING.md as a starting point.

I removed sections that are not relevant to pgcopydb, and added some
more sections about documentation and running tests.

I also added references to the CONTRIBUTING.md file in the README.md and
in documentation page about installing pgcopydb. While I was at it, I
also removed the outdated installation instructions from the README.md
file, and replaced them with a reference to the documentation page.

Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>

* Fix confusion around reporting operations in progress.

The SQLite refactor meant rewriting all the SourceTableArray loops to use a
query result iterator instead. Well there is a place for a SourceTableArray
loop still, and that's the pgcopydb list progress thing.

* Fix the table-in-copy SQLite query.

The query was meant to report actual duration and transmitted bytes for each
table in-progress, but the join and sum() operations are creating problems
in the SQLite query and the rest of the progress.c code was not prepared to
benefit from the extra information anyway.

The SQLite problem is that the sum(s.duration), sum(s.bytes) causes a single
row output when the dataset is empty, and the other columns are all set to
NULL. That might be because of the lack of a GROUP BY clause.

Simplify that aspect by removing the extra information for now.

Since the move of the summary and process information to the SQLite
catalogs, we don't have a progress JSON file on-disk with parts of the setup
anymore.

The code needed to be updated to behave correctly without having
--table-jobs and --index-jobs information.

* Use Boehm-Demers-Weiser Garbage Collector.

From their own documentation:

> The Boehm-Demers-Weiser conservative garbage collector can be used as a
> garbage collecting replacement for C malloc or C++ new. It allows you to
> allocate memory basically as you normally would, without explicitly
> deallocating memory that is no longer useful. The collector automatically
> recycles memory when it determines that it can no longer be otherwise
> accessed.

In pgcopydb C code base it's been hard to maintain proper malloc/free
concerns while also implementing proper error-handling, which is a common
problem when writing C code. In order to ease the maintenance of the code
and reduce production hazards, the best way to proceed looks like automating
the memory management altogether by using a Garbage Collector.

* Revert removal of regree() calls.

* Get rid of LinesBuffer.ownsBuffer.

Now that libgc is handling the memory for us, we don't need to keep track of
the ownership of the buffer anymore.

I observed frequent `[SQLite] PRAGMA journal_mode = WAL` when it should be
run only once as it's a sqlite database property, not something that has be
set for every connection. This might fix
https://github.com/dimitri/pgcopydb/actions/runs/7614380466/job/20736769276?pr=647

```
2024-01-22 16:45:01.923 44 ERROR  catalog.c:7129            [SQLite]: PRAGMA journal_mode = WAL failed: database is locked
```

- Adds CI to check that the documentation templates are up-to-date.
- Add a new `help` command to the documentation templates. This command
  prints the help message for all the sub-commands of `pgcopydb` in a
  single file. This is useful for users who want to see all the
  sub-commands in one place.
- Remove dependency on installing the binary before updating the
  documentation templates. This allows users to update the documentation
  templates without installing the binary. One can set the `PGCOPYDB`
  environment variable to point to the binary to use for updating the
  documentation templates.
- Fix a casing issue in command descriptions. The first letter of the
  first word in the description of each command should be capitalized.

When registering the catalog setup, which includes the same-table
concurrency threshold, only consider that value when the table partitions
have been populated in the cache already. Otherwise the table partitions are
going to be computed again anyway.

* Improve SQLite write performance with journal_mode as WAL.

* Batch initial catalog setup write in a single transaction.

The SQLite patch series introduced a bug where we would skip extensions even
when the option --skip-extension is not used.

Also adjust the "pgcopydb copy extensions" to the recent changes where we
use a sub-process under a supervisor to simplify snapshot handling.

Make sure we don't try and allocate a zero-size memory area, which can lead
to "corrupted size vs. prev_size". Review all the calloc() call sites and
protect them better when they need to be.

Make it systematic to first count the lines and then allocate the right
amount of memory for the array of pointers to the start of each line, and
make sure we don't try and allocate a zero-size array, which can lead to
"corrupted size vs. prev_size" errors at run-time.

Unfortunately -Wshadow=compatible-local does not seem to be available when
using llvm as the compiler. Still, debian builds for Postgres community are
producing warnings that we can fix.

It turns out that when using just "exclude-index" in our filtering the code
would be confused as to how to implement that filtering option for listing
table and sequence objects from the Postgres catalogs.

* Use SQLite for LSN tracking.

* Ensure we open the catalogs in pgcopydb stream apply.

This makes it possible to see the timings while pgcopydb is operating,
including timing of top-level operations and cumulative timings.

Solution: Populate extension configuration objects along with their type
values. Based on the type value, perform either table copy or sequence copy.

At the moment we support only copying extension configuration tables & sequences.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>