This makes it possible to see the timings while pgcopydb is operating,
including timing of top-level operations and cumulative timings.

Solution: Populate extension configuration objects along with their type
values. Based on the type value, perform either table copy or sequence copy.

At the moment we support only copying extension configuration tables & sequences.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Remove root_with_debug CommandLine that is identical to root

* Fix numerous issues with argument parsing

The following issues were fixed:
- some options were not recognized
- some nonexistent options were recognized
- some options with required arguments were not expecting an argument
- some commands did not print help messages when given invalid options

Solution: Use flush LSN from target to find the durable LSN.

To find a durable LSN with async commit, we maintain a list of LSN mapping
between source(sourceLSN) and target(insertLSN). New mapping will be
inserted into the list on COMMIT or KEEPALIVE message.

While reporting the replay_lsn to sentinel we have to get the flush lsn from
target and compare it against the insertLSN from the list to find the sourceLSN.

This commit introduces a shell script that runs all commands that accept
`--help` parameter and stores the result in template files that are
referenced in our documentation. You can use that script by running
`make update-docs` in top directory of the project.

Usage: `make DEBUG=1 install`

Introduce a new query to count the number of tables and indexes already
processed from the summary table, fixing the list progress output for items
"done".

* Use our SQLite catalogs for the pgcopydb sentinel table.

The main use-case for the sentinel table is to communicate progress made by
the three independent worker processes for stream, transform, and replay.
It is also needed to communicate new startpos, endpos and apply values with
those processes from another pgcopydb (interactive) command, a process
that's not in the process tree.

This was previously solved by creating and using a pgcopydb.sentinel table
on the source database. Now that we have a SQLite database internal to the
pgcopydb run, we can use that instead.

* More work on the sentinel table move to SQLite catalogs.

In particular, we need to share a common directory mount point in the test
and inject containers, because the inject container is setting the endpos
for the test container where the main tests are running.

For that, in my testing the best option is to use a host directory. This
also makes it very easy to then use sqlite3 and inspect what's going on from
the outside.

* Attempt to fix GitHub Action host workdir perms

* Fix follow-data-only tests: update sentinel at stream flush time.

* Assorted fixes, per self-review.

The copy-db command is deprecated and will get removed from pgcopydb on
next release, please upgrade your scripts and integrations.

The waitpid() API provides the process return code, which can still be zero
when the processed was terminated by a signal such as SIGSEGV. Improve our
sub-process supervision with taking that into account as an unexpected error
condition.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Remove summary files, use our SQLite database instead.

* Get rid of summary files for the indexing and constraints too.

This allows also to entirely remove the need for semaphores to handle
concurrency between worker processes, using SQLite insert-or-ignore for
concurrency control instead.

Also the idea of TablePaths and IndexPaths elements are gone, with the lock
files and the done files mechanisms.

* Protect SQLite concurrent write access with a semaphore.

SQLite has not been designed for write-write concurrency, so implement a
critical section in our code so that SQLite only sees a single writer at all
times. It may still see multiple readers though.

Because we have re-entrant queries (an SQLite iterator function may then
choose to run another SQLite query), this commit also adds support for
re-entrant semaphore in a way that doesn't call into semop(2) again when we
already are in the critical section.

* Assorted fixes.

When the COPY table-data workers fail before the iteration over the list of
tables is done, we have a race condition where the COPY supervisor would be
busy and is not using waitpid(), thus not reacting to worker process
termination.

To solve, limit the responsibilities of the COPY supervisor process even
more and create a new process that's responsible for iterating over the
table list and sending tables/parts to the queue, etc.

The code to call TRUNCATE ONLY was only active for tables setup with COPY
partitioning (--split-tables-larger-than), and was forced to "false" for the
other tables.

The code also evolved in a way that it would build a sub-SELECT query with
the table attribute list as the "srcQname" argument to the pgsql.c internal
bits for the copy, making it impossible to then re-use that bit in the sql
command for the TRUNCATE operation.

To fix, implement a new way to prepare the COPY query bits and pieces so
that we can re-use the qualified table name in the TRUNCATE command and also
build a full COPY (SELECT a, b, c FROM ONLY t WHERE ...) TO STDOUT; query.

In passing, because of the way we are changing the summary files to use our
internal SQLite database instead, also push the TRUNCATE command of COPY
partitionned table to the COPY supervisor process. This is a better way to
solve the concurrency issues and make sure that TRUNCATE is done only once,
and also finished before any of the copy-data worker processes get started.

* add quotes identifiers

* test changes

* Fix escaping of identifiers while transforming for wal2json plugin.

wal2json doesn't escape identifiers as test_decoding do. To fix this issue
we are using PQescapeIdentifier function. Even though this function doesn't
require PGconn to be connected, but creating an empty PGconn object was
unfeasible. Some fields of PGconn that are required by new versions of
libpq for PQescapeIdentifier are not available in older PGconn. So decided
to do this the right way, but creating PGconn with an actual server
connection. It turned out more simpler than managing a dummy PGconn object.

* Fix review comments.

* Fix style.

---------

Co-authored-by: VaibhaveS <vaibhavedavey@gmail.com>

test_decoding transform relies on source catalog to decode the update message.

Prior to this commit, update while using test_decoding would work only while doing `pgcopydb clone --follow`. This commit enables support for update message decoding
while using test_decoding plugin in `pgcopydb follow` too.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

We close the FILE which is used to log into file before calling
the `copydb_cleanup_sysv_resources` which still uses log functions leads
to crash due to invalid FILE* access.

This commit fixes the crash by closing the FILE after calling the
cleanup function.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

We track resources likes semaphore and queue into a global array and
deallocate the non freed objects using exit handler. The array tracking
these resources uses pointers, which could inadvertently track resource
handles allocated from the stack.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

The filter clause is not supported by my local SQLite for some reason,
so I needed to use a different approach to count the number of
constraints on a table.

This fixes the following error:

    ERROR  Failed to prepare SQLite statement: select count(1) as indexes, count(1) filter(where c.oid is not null) OVER () as constraints from s_index i left join s_constraint c on c.indexoid = i.oid where tableoid = $1
    ERROR  [SQLite] misuse of aggregate: count()
    ERROR  Failed to count indexes attached to table dist_tables.t1
    ERROR  Failed to copy data for table with oid 17226 and part number 0, see above for details

Relevant documentation at https://www.sqlite.org/lang_aggfunc.html says:

The count(X) function returns a count of the number of times that X is not NULL in a group

There were few issues with --skip-extensions flag

1) Excluded extensions were not part of the filter
2) Invalid memory access from pg_dump_db_extension_namespace_hook
3) Filter table populated without checking whether it is already done or
   not.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

Currently, pgcopydb dump & restore doesn't support skipping extensions or filtering because we don't populate the context with extension details & skip list information.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

Catalog queries may return a very large number of rows when the source
database contain a very large number of objects (tables, indexes, etc). To
best cope with that use Single Row Mode to fetch one row at a time in-memory
before inserting it to our internal SQLite catalogs.

This allows to implement an in-memory array with the capabilities to
spill-to-disk and also hash-table like lookup operations, and with good
multi-process concurrency handling.

Using SQLite internally will allow reducing the memory usage of pgcopydb and
make it possible to address databases with more than a million of tables
defined.

That allows C-c to work in more situations, which is always good.

We would see “WARNING: transaction already in progress” in the logs.

Postgres application_name is limited to NAMEDATALEN, sending more bytes
than that gives us a NOTICE and we can easily avoid it. In passing, avoid
redundancies in the application_name when the ps_buffer already starts with
"pgcopydb: ".

Search page shows 404, whereas genindex and modindex are empty pages.

* Add option to output numeric as string on wal2json

This change adds a new option `--wal2json-numeric-as-string` that
changes wal2json plugin output format to print numeric data types as
strings. This is accomplished by passing the
`--numeric-data-types-as-string` option to wal2json plugin.

This is useful to prevent precision loss when using wal2json
plugin to stream changes from a database that uses numeric data types.

wal2json plugin version that supports `--numeric-data-types-as-string`
option is required to use this pgcopydb option. As of today there is no
official wal2json release that supports this option, but it is available
on master branch of the project.

Relevant changes in wal2json plugin is at
https://github.com/eulerto/wal2json/pull/255

* Add env to output numeric as string on wal2json

PGCOPYDB_WAL2JSON_NUMERIC_AS_STRING can be set to a boolean value that
will be used to determine if pgcopydb should set the wal2json option
`--numeric-data-types-as-string`.

In passing, also add the PGCOPYDB_OUTPUT_PLUGIN env variable to all
relevant pages of our documentation.

* Run make indent target

This commit will allow developers to run `make indent` to indent their
code in their development branches by solving all the indentation issues
in the codebase.

* Add citus_indent to CI

Endpos can be set to 0/0 which is an invalid LSN.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>