This allows to implement an in-memory array with the capabilities to
spill-to-disk and also hash-table like lookup operations, and with good
multi-process concurrency handling.

Using SQLite internally will allow reducing the memory usage of pgcopydb and
make it possible to address databases with more than a million of tables
defined.

* Fix copy paste error on comment
* Fix fatal message content

Currently, when switching between modes, the follow process cleans up context files, which include details like WAL segment size and timeline. However, we haven't been resetting the in-memory state that decides whether the context should be read from these files. This can lead to the usage of outdated timeline information while retriving the context[1][2][3].

This commit addresses this issue by resetting the in-memory state during context cleanup. This ensures that the system will re-read the context from the files after any cleanup, leading to more accurate and reliable timeline information.

[1] https://github.com/dimitri/pgcopydb/blob/8d305a51f82a83bacac6a3fcce716c9450505e57/src/bin/pgcopydb/ld_transform.c#L201-L212
[2] https://github.com/dimitri/pgcopydb/blob/8d305a51f82a83bacac6a3fcce716c9450505e57/src/bin/pgcopydb/follow.c#L428-L438
[3] https://github.com/dimitri/pgcopydb/blob/8d305a51f82a83bacac6a3fcce716c9450505e57/src/bin/pgcopydb/ld_apply.c#L206-L216



Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

If `statement_timeout` is set on the target COPY may get interrupted,
hence this change.

* Fix readthedocs.org configuration.

* Add docs/requirements.txt

* Fix YAML syntax.

* Fix YAML syntax again.

Follow up of
https://github.com/dimitri/pgcopydb/pull/533#pullrequestreview-1720295497.
Context is sometimes msgrcv acts weird and corrupts the value of other
stack allocated variables. Using heap allocated queue message as the
workaround until its root cause is known.

Presently, each LOB worker creates a new connection for every object copy,
impacting migration performance, especially with a large number of objects.
The fix here is simple: use the same connection within each worker.

However, while doing so, we encountered an unexpected error:

`BUG: call to pgsql_commit() without holding an open multi statement
connection`

Upon investigation, we discovered that the `connectionStatementType` was
set to an unusual value of 32767. In GDB, it was
(PGSQL_CONNECTION_MULTI_STATEMENT | unknown: 32766). This happens only
after the call of `msgrcv` function. There are a few theories; one suggests
that because both `msg` and `dst` are on the stack, `msgrcv` is overwriting
`dst` data when it should only change `msg`. Despite having
`-fstack-protector` enabled, we did not observe a "stack smashing detected"
error, so maybe this theory is incorrect.

In any case, moving `msg` to the heap seemed to resolve the issue. However,
it's important to fully understand this problem, as it may have
implications in other areas of the code as well.

We actually only need the possibly new endpos value when a process just
terminated, in order to decide what to do. So only fetch the sentinel endpos
value when a process just terminated.

* feat(container): multi arch build
* fix(container): copy whole repo for the build step; add label for gh pkg

Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>

Previously, when a connection timeout occurred with the default connection
timeout set to pgconnect_timeout (10 seconds), there were no retries. This
was because the total retry timeout (maxT) passed to `pgsql_set_retry_policy`
in `pgsql_set_interactive_retry_policy` was also set to pgconnect_timeout,
which had already elapsed during the initial connection attempt.

To address this issue, the maxT value has been updated to
POSTGRES_PING_RETRY_TIMEOUT. This change also means that maxT cannot be
configured by altering PGCONNECT_TIMEOUT. If one wants to configure maxT
they can change the values in defaults.h or we can make it configurable
through some environment variable in future.

After all this change `pgconnect_timeout` seems redundant so removed it.

* Update pgcopydb sentinel in the main follow process.

To avoid infinite looping when endpos has been reached it's important to
update our endpos value to the sentinel's one even in the main follow
process.

In passing, review some error messages.

Also refrain from stopping early from reading data from a PIPE when a
terminating signal is received, we should finish reading as per the comments
in the code.

* Process ENDPOS in transform and replay processes.

When the ENDPOS internal message is received from the PIPEs in replay mode,
check the current sentinel.endpos value on the source database to make sure
the ENDPOS message matches with the current setting, and that being the case
stop processing.

This commit removes the usage of transaction metadata file. Initially, it was used by the apply process to bypass transactions that were already applied. However, this approach had its challenges. Specifically, in live replay mode, a transaction with numerous statements could fill the UNIX PIPE (an IPC primitive used in replay mode), leading to a potential deadlock. This is because the apply process would be waiting for the transaction metadata file.

By eliminating the transaction metadata file, the apply process lets the transaction proceed and decides whether to apply or skip it based on the commit LSN during the commit phase.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* remove PGPASSWORD environment processing from pgsql.c

instead use `pgsql->connectionString` containing the password
and adjust related function comments and removed unused PASSWORD_MASK

* fix mismatched safe{Source,Target}PGURI.password references

Apologies for such a small edit.
Always leave things better than you found them!

without this change `pgcopydb follow` would fail with e.g.:

```
Failed to parse decoding message for UPDATE on table "public"."tablename" which is not in our catalogs
```

As sprintf gets deprecated, update sources to use snprintf.

The for-loop is copying a string character by character and replacing `''` with `'`
by skipping a character if the current and next one are both `'`. When this
replacement doesn't occur then the target string will be filled completely
without a `\0' terminator at the end of the string.

Prior to this commit, the replay_lsn remained static when the source was idle
for an extended period, potentially leading to inaccurate replication lag
monitoring. This commit addresses this issue by ensuring the replay_lsn is
updated consistently, even when the source is idle, providing a more accurate
reflection of the system's state.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* add archive entry mapping for large objects
* add archive entry mapping for row security

* use LOG_WARN for warnings instead of LOG_ERROR

* use stronger matching for filtering warnings

---------

Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>

This patch switches the copydb_copy_database_properties() function to
use the target database instead of the source database name when constructing
the ALTER commands. Without this, the ALTER commands would break unless the
database names were identical.

Also introduce a change to use snprintf() instead of sprintf() in a couple
places, following gcc recommandations.

* Review logical decoding client tracking of LSNs.

When replaying from file, ensure that the last transaction of a file is
commited with synchronous_commit on, allowing better tracking of replication
progress.

Also, improve our way to keep track of the previousLSN and log when we reach
endpos (KEEPALIVE, COMMIT, ENDPOS) to help get a sense of what's happening.

By setting previousLSN to the proper value even when reaching endpos at a
KEEPALIVE message that we skip, we allow the overall system to exit properly
in case when there is no activity on the source server.

* Enhance clean_results() to avoid blocking calls to PQgetResult().

In a debug session with gdb, now enabled in our docker-compose setup, it was
apparent that a process would block on a PQgetResult() call done in the
context of clear_results() when an error would happen during the logical
replication streaming.

Fix both the clear_results() behavior and also refrain from calling it on
the error path, just finish the connection without trying to get possibly
pending results that we're not going to need anyway.

* Review process startup init and setting of endpos / --endpos.

When the command line option is used, we should not take the value that is
fetched from the source pgcopydb.sentinel table. Also protect some processes
against having already reached endpos at startup time.

In passing, try to test for Broken PIPE upon connection retries. Testing is
not conclusive on that parts, but it seems harmless.

* Properly reset stdIn and stdOut booleans to false when needed.

Now that we have pro-active actions implemented to check if the PIPE might
be broken we need to pay attention to these variables.

Also in passing have the catchup process between follow loops happen in
sub-processes too. That way the process model stays the same even in that
area of the code: the actual processing is done by sub-processes.

* Review Dockerfiles: replace psutils with psmisc.

psutils: PostScript document handling utilities
psmisc: utilities that use the proc file system (including pstree)

also add htop which is useful at displaying a process tree

* Fix working directory for pgcopydb stream cleanup.

* Add CHECK CONSTRAINT archive entry mapping.

* Fix success return of `copydb_target_finalize_schema` on failure.

* Add CHECK CONSTRAINT unit test

* Add LANGUAGE archive entry mapping

* Link issue in unit test

---------

Co-authored-by: Shubham Dhama <shubhamdhamaofficial@gmail.com>

* Fix typo: ther -> the

* Consistent casing in help messages

* Update versions in docs to latest

* Consistent forked process names in design docs

* Update package list in rockylinux

* Fix indentation in pgcopydb help message

* Fix typo: Create and exports -> Create and export

* Update some version outputs

* Fix typo

* Update version output using official docker image

The `read_from_stream` function in pgcopydb was encountering a data loss issue
when reading from a source system with transaction bodies containing more than
1024 inserts. The problem arose from the statically allocated array of size 1024
used to split lines within the function. When the buffer held more than 1024
lines, it caused subsequent lines to be ignored, resulting in data loss.

To resolve this issue, this commit introduces the following changes:
- We now count the number of lines in the buffer before splitting them.
- Dynamically allocate an array to hold the split lines, based on the line count.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

This change will make sure that pgcopydb can write to the database even
if the user has set the default_transaction_read_only GUC.

When an outside contributor submits a PR, the test runs fail because
they have the commits on a fork, and there is no easy way to get the
commits from the fork into the main repository. This change makes the
test runs work for outside contributors by always checking out the SHA
for the event.

We no longer need to have separate steps for PRs and non-PRs as
checkout@v3 has sane defaults for both. Below is an excerpt from the
docs of this action that explains the behavior of ref values:

  The branch, tag or SHA to checkout. When checking out the repository
  that triggered a workflow, this defaults to the reference or SHA for
  that event. Otherwise, uses the default branch.

* Switch to using libpq async API.

The easy to use API PQexec() and PQexecParams() are blocking, which means
those functions could fail to detect when a Postgres connection has been
lost, and now wait an indefinite amount of time.

Switch to using the async API PQsendQuery() or PQsendQueryParams() with the
PQgetResult() function, all with some lower-level socket API calls such as
select(2) and PQconsumeInput() and PQisBusy().

Note that when the server is no longer busy computing the result, we should
loop over all the available results... apparently.

* Code cleanup, add another check for interrupts.

* Remove extra check for interrupts.

The Postgres tooling for configuration variable settings that are specific
to a database is only activated when pg_restore --create is used, which
means that pg_restore is in charge of CREATE DATABASE.

In the context of pgcopydb, we do not want pg_restore to create the target
database for us, but we still want to copy over the database specific
settings that have been applied with either of the following commands:

    ALTER DATABASE SET
    ALTER ROLE IN DATABASE SET

To achieve that we need to copy some code from the pg_dump and pg_restore
implementation and along with it we also vendor-in some of the Postgres
common code (a part of string_utils.c) to avoid link-time issues on
different packaging systems.

* Improve error handling when COPY fails.

In the normal case we want the COPY process to continue with the rest of the
tables in the queue, and in the fail-fast case we want to bubble-up the
problem as fast as possible for a quick total failure.

The partial failure in the normal case can be resumed with the existing
option such as --resume, as long as the snapshot still is held, or when
using the option --not-consistent.

Also, ensure that when errors happen mid-flight in the COPY processes we
still send the STOP messages in the queues, or signal all the other
processes to terminate.

* Also review error handling for vacuum/create index queues.

Now that we grab the Postgres identifier names already quoted, we need to
expand our internal structures memory to handle more bytes: the quotes
around the names, and also any double-quote char within the name is going to
be doubled by the quoting rules.

This call should not be blocking in our implementation, because we want to
be able to react to network problems or server-side interruptions.

The test_decoding plugin escapes identifiers using double quotes for
column names. However, the pgcopydb transform process also escapes
identifiers in the same way, leading to double escaping and apply process
failures.

This commit addresses the issue by adding double quotes only when the
column name is not already quoted.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Add tests coverage for Postgres 16.

* Fix GitHub Actions workflow file.