* feat(container): multi arch build
* fix(container): copy whole repo for the build step; add label for gh pkg

Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>

Previously, when a connection timeout occurred with the default connection
timeout set to pgconnect_timeout (10 seconds), there were no retries. This
was because the total retry timeout (maxT) passed to `pgsql_set_retry_policy`
in `pgsql_set_interactive_retry_policy` was also set to pgconnect_timeout,
which had already elapsed during the initial connection attempt.

To address this issue, the maxT value has been updated to
POSTGRES_PING_RETRY_TIMEOUT. This change also means that maxT cannot be
configured by altering PGCONNECT_TIMEOUT. If one wants to configure maxT
they can change the values in defaults.h or we can make it configurable
through some environment variable in future.

After all this change `pgconnect_timeout` seems redundant so removed it.

* Update pgcopydb sentinel in the main follow process.

To avoid infinite looping when endpos has been reached it's important to
update our endpos value to the sentinel's one even in the main follow
process.

In passing, review some error messages.

Also refrain from stopping early from reading data from a PIPE when a
terminating signal is received, we should finish reading as per the comments
in the code.

* Process ENDPOS in transform and replay processes.

When the ENDPOS internal message is received from the PIPEs in replay mode,
check the current sentinel.endpos value on the source database to make sure
the ENDPOS message matches with the current setting, and that being the case
stop processing.

This commit removes the usage of transaction metadata file. Initially, it was used by the apply process to bypass transactions that were already applied. However, this approach had its challenges. Specifically, in live replay mode, a transaction with numerous statements could fill the UNIX PIPE (an IPC primitive used in replay mode), leading to a potential deadlock. This is because the apply process would be waiting for the transaction metadata file.

By eliminating the transaction metadata file, the apply process lets the transaction proceed and decides whether to apply or skip it based on the commit LSN during the commit phase.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* remove PGPASSWORD environment processing from pgsql.c

instead use `pgsql->connectionString` containing the password
and adjust related function comments and removed unused PASSWORD_MASK

* fix mismatched safe{Source,Target}PGURI.password references

Apologies for such a small edit.
Always leave things better than you found them!

without this change `pgcopydb follow` would fail with e.g.:

```
Failed to parse decoding message for UPDATE on table "public"."tablename" which is not in our catalogs
```

As sprintf gets deprecated, update sources to use snprintf.

The for-loop is copying a string character by character and replacing `''` with `'`
by skipping a character if the current and next one are both `'`. When this
replacement doesn't occur then the target string will be filled completely
without a `\0' terminator at the end of the string.

Prior to this commit, the replay_lsn remained static when the source was idle
for an extended period, potentially leading to inaccurate replication lag
monitoring. This commit addresses this issue by ensuring the replay_lsn is
updated consistently, even when the source is idle, providing a more accurate
reflection of the system's state.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* add archive entry mapping for large objects
* add archive entry mapping for row security

* use LOG_WARN for warnings instead of LOG_ERROR

* use stronger matching for filtering warnings

---------

Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>

This patch switches the copydb_copy_database_properties() function to
use the target database instead of the source database name when constructing
the ALTER commands. Without this, the ALTER commands would break unless the
database names were identical.

Also introduce a change to use snprintf() instead of sprintf() in a couple
places, following gcc recommandations.

* Review logical decoding client tracking of LSNs.

When replaying from file, ensure that the last transaction of a file is
commited with synchronous_commit on, allowing better tracking of replication
progress.

Also, improve our way to keep track of the previousLSN and log when we reach
endpos (KEEPALIVE, COMMIT, ENDPOS) to help get a sense of what's happening.

By setting previousLSN to the proper value even when reaching endpos at a
KEEPALIVE message that we skip, we allow the overall system to exit properly
in case when there is no activity on the source server.

* Enhance clean_results() to avoid blocking calls to PQgetResult().

In a debug session with gdb, now enabled in our docker-compose setup, it was
apparent that a process would block on a PQgetResult() call done in the
context of clear_results() when an error would happen during the logical
replication streaming.

Fix both the clear_results() behavior and also refrain from calling it on
the error path, just finish the connection without trying to get possibly
pending results that we're not going to need anyway.

* Review process startup init and setting of endpos / --endpos.

When the command line option is used, we should not take the value that is
fetched from the source pgcopydb.sentinel table. Also protect some processes
against having already reached endpos at startup time.

In passing, try to test for Broken PIPE upon connection retries. Testing is
not conclusive on that parts, but it seems harmless.

* Properly reset stdIn and stdOut booleans to false when needed.

Now that we have pro-active actions implemented to check if the PIPE might
be broken we need to pay attention to these variables.

Also in passing have the catchup process between follow loops happen in
sub-processes too. That way the process model stays the same even in that
area of the code: the actual processing is done by sub-processes.

* Review Dockerfiles: replace psutils with psmisc.

psutils: PostScript document handling utilities
psmisc: utilities that use the proc file system (including pstree)

also add htop which is useful at displaying a process tree

* Fix working directory for pgcopydb stream cleanup.

* Add CHECK CONSTRAINT archive entry mapping.

* Fix success return of `copydb_target_finalize_schema` on failure.

* Add CHECK CONSTRAINT unit test

* Add LANGUAGE archive entry mapping

* Link issue in unit test

---------

Co-authored-by: Shubham Dhama <shubhamdhamaofficial@gmail.com>

* Fix typo: ther -> the

* Consistent casing in help messages

* Update versions in docs to latest

* Consistent forked process names in design docs

* Update package list in rockylinux

* Fix indentation in pgcopydb help message

* Fix typo: Create and exports -> Create and export

* Update some version outputs

* Fix typo

* Update version output using official docker image

The `read_from_stream` function in pgcopydb was encountering a data loss issue
when reading from a source system with transaction bodies containing more than
1024 inserts. The problem arose from the statically allocated array of size 1024
used to split lines within the function. When the buffer held more than 1024
lines, it caused subsequent lines to be ignored, resulting in data loss.

To resolve this issue, this commit introduces the following changes:
- We now count the number of lines in the buffer before splitting them.
- Dynamically allocate an array to hold the split lines, based on the line count.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

This change will make sure that pgcopydb can write to the database even
if the user has set the default_transaction_read_only GUC.

When an outside contributor submits a PR, the test runs fail because
they have the commits on a fork, and there is no easy way to get the
commits from the fork into the main repository. This change makes the
test runs work for outside contributors by always checking out the SHA
for the event.

We no longer need to have separate steps for PRs and non-PRs as
checkout@v3 has sane defaults for both. Below is an excerpt from the
docs of this action that explains the behavior of ref values:

  The branch, tag or SHA to checkout. When checking out the repository
  that triggered a workflow, this defaults to the reference or SHA for
  that event. Otherwise, uses the default branch.

* Switch to using libpq async API.

The easy to use API PQexec() and PQexecParams() are blocking, which means
those functions could fail to detect when a Postgres connection has been
lost, and now wait an indefinite amount of time.

Switch to using the async API PQsendQuery() or PQsendQueryParams() with the
PQgetResult() function, all with some lower-level socket API calls such as
select(2) and PQconsumeInput() and PQisBusy().

Note that when the server is no longer busy computing the result, we should
loop over all the available results... apparently.

* Code cleanup, add another check for interrupts.

* Remove extra check for interrupts.

The Postgres tooling for configuration variable settings that are specific
to a database is only activated when pg_restore --create is used, which
means that pg_restore is in charge of CREATE DATABASE.

In the context of pgcopydb, we do not want pg_restore to create the target
database for us, but we still want to copy over the database specific
settings that have been applied with either of the following commands:

    ALTER DATABASE SET
    ALTER ROLE IN DATABASE SET

To achieve that we need to copy some code from the pg_dump and pg_restore
implementation and along with it we also vendor-in some of the Postgres
common code (a part of string_utils.c) to avoid link-time issues on
different packaging systems.

* Improve error handling when COPY fails.

In the normal case we want the COPY process to continue with the rest of the
tables in the queue, and in the fail-fast case we want to bubble-up the
problem as fast as possible for a quick total failure.

The partial failure in the normal case can be resumed with the existing
option such as --resume, as long as the snapshot still is held, or when
using the option --not-consistent.

Also, ensure that when errors happen mid-flight in the COPY processes we
still send the STOP messages in the queues, or signal all the other
processes to terminate.

* Also review error handling for vacuum/create index queues.

Now that we grab the Postgres identifier names already quoted, we need to
expand our internal structures memory to handle more bytes: the quotes
around the names, and also any double-quote char within the name is going to
be doubled by the quoting rules.

This call should not be blocking in our implementation, because we want to
be able to react to network problems or server-side interruptions.

The test_decoding plugin escapes identifiers using double quotes for
column names. However, the pgcopydb transform process also escapes
identifiers in the same way, leading to double escaping and apply process
failures.

This commit addresses the issue by adding double quotes only when the
column name is not already quoted.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Add tests coverage for Postgres 16.

* Fix GitHub Actions workflow file.

* Rewrite the ad-hoc parser for Postgres Archive TOC items.

This time use a tokenizer and hand-roll a grammar/parser on-top of it,
expecting tokens in the right order. We have a pretty simple grammar at the
moment, because we don't extend the support to the ACL and COMMENT composite
types. We might want to do that later.

* Reformat parsing of ACL and COMMENTS too.

* Free allocated memory for parsing pg_restore archive catalogs.

* Fix rebase.

* Protect against NULL pointer dereferencing.

* Fix string size compute bug (include 2 bytes for prefix).

Co-authored-by: lospejos <lospejos@github.com>

The following command will now parse the pg_restore --list output file and
output to standard out the parsed contents, allowing to check the parser.

    pgcopydb restore parse-list pre.list

This commit optimizes multi-value inserts by transferring insert values
from a new statement to an existing one if they target the same table.
This optimization significantly reduces network latency when multiple consecutive
insert statements are executed within a transaction.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

That way it's possible to review the pg_restore file separately from the
filtered file that we process in pgcopydb, and because both the files are
left around at the end of the command, it's also possible to diff them.

Using the Postgres function format with the %I formatter allows our code to
bypass any formatting internally when re-using the SQL object names, both in
our logs and also in the SQL queries we then emit.

Generalize that approach to attribute names, index names, index table names,
index constraint names, and sequences names.

When failing to start a sub-process (out-of-memory can be reported when
calling the fork() system call), properly handle the error rather than
continue trying the migration.

In order to cancel the other sub-processes that were succesfully started we
then send a TERM signal to all processes in our process group.

* Improve connection string TCP keepalive parameters handling.

Allow overriding at the command-line level, also set the parameters in the
connection string so that they're effective even when connecting.

* Fix computing of the safe URI: always do it now.

To work with shorter timeouts in network stacks (firewall, NAT settings,
etc), adjust to TCP keepalive settings to quite short values.

* Set Process Titles as seen in ps/top/htop etc.

This helps understand the process hierarchy better and could be useful to
debug or watch long running processes.

* Append current table/index information when we have it.