We would see “WARNING: transaction already in progress” in the logs.

Postgres application_name is limited to NAMEDATALEN, sending more bytes
than that gives us a NOTICE and we can easily avoid it. In passing, avoid
redundancies in the application_name when the ps_buffer already starts with
"pgcopydb: ".

Search page shows 404, whereas genindex and modindex are empty pages.

* Add option to output numeric as string on wal2json

This change adds a new option `--wal2json-numeric-as-string` that
changes wal2json plugin output format to print numeric data types as
strings. This is accomplished by passing the
`--numeric-data-types-as-string` option to wal2json plugin.

This is useful to prevent precision loss when using wal2json
plugin to stream changes from a database that uses numeric data types.

wal2json plugin version that supports `--numeric-data-types-as-string`
option is required to use this pgcopydb option. As of today there is no
official wal2json release that supports this option, but it is available
on master branch of the project.

Relevant changes in wal2json plugin is at
https://github.com/eulerto/wal2json/pull/255

* Add env to output numeric as string on wal2json

PGCOPYDB_WAL2JSON_NUMERIC_AS_STRING can be set to a boolean value that
will be used to determine if pgcopydb should set the wal2json option
`--numeric-data-types-as-string`.

In passing, also add the PGCOPYDB_OUTPUT_PLUGIN env variable to all
relevant pages of our documentation.

* Run make indent target

This commit will allow developers to run `make indent` to indent their
code in their development branches by solving all the indentation issues
in the codebase.

* Add citus_indent to CI

Endpos can be set to 0/0 which is an invalid LSN.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

This commit adds a new option --restore-jobs to pgcopydb that allows
specifying how many jobs can be used to run pg_restore operations in
parallel. This option can also be set using the PGCOPYDB_RESTORE_JOBS
environment variable.

When this option is set to 1, pgcopydb will run pg_restore with the
--single-transaction option, and pgcopydb will behave the same as it
used to. Otherwise, pg_restore will be run with the --jobs option set
to the number of jobs specified by the user.

If the user does not supply the --restore-jobs option, or set the
environment variable, pgcopydb will use the index-jobs value as default.

In some cases, the replay process doesn't reach the endpos due to a problem
in the stream process. The problem arises when stream process receives a
'k' message from wal sender and reaches the endpos. But instead of writing
the newly received LSN, we write previous LSN with the KEEPALIVE message.
The following filtered logs from the streaming process illustrate this
issue.

```
endpos is now set to 3/AF01C408
Reported write_lsn 3/AF01C408, flush_lsn 3/AF01BF38, replay_lsn 3/AF01BF38
streamFlush: 3/AF01C408 3/AF01C0A0
Inserted action KEEPALIVE for lsn 3/AF01C0A0 @2023-11-27 11:43:55.946327+0000
Flushed up to 3/AF01C0A0 in file "cdc/0000000100000003000000AF.json.partial"
pgsql_stream_logical: endpos reached on keepalive: 3/AF01C5A0
streamFlush: 3/AF01C5A0 3/AF01C0A0
Inserted action KEEPALIVE for lsn 3/AF01C0A0 @2023-11-27 11:43:59.754050+0000
```

This commit addresses and resolves the issue of duplicate key errors when resuming partially executed transactions (continuedTxn) in pgcopydb. We have reintroduced the transaction metadata file, which is essential for identifying the commitLSN of a partial transaction.

Unlike our previous approach, which led to a deadlock between the transform and apply phases, this update brings a more efficient process. Now, the apply phase creates metadata for any partial (continued) transactions during the commit. This metadata is then used to accurately skip the already applied partial transaction if a resume is needed.

This fix is crucial, particularly for tables with unique constraints, where executing the same continued transaction twice previously resulted in duplicate key errors. With this update, pgcopydb ensures smooth and error-free handling of transaction resumes.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

In the follow process, we create two Unix pipes for bidirectional communication between child processes (prefetch, transform, apply) during replay mode. Each child process closes its unused pipe descriptors upon startup. For example, the prefetch process, which only writes to the transform process, closes the read end of its pipe descriptors immediately.

However, the follow process (parent) itself, which initializes these pipes, does not utilize them for communication. Leaving these pipe descriptors open in the follow process leads to an issue where EOF is not sent, even after the child processes close their ends upon completion. This situation can result in a deadlock, especially when an interruption occurs through Ctrl+C or a SIGTERM signal.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

In .rst files, the first line of a code block must be empty.  This
was causing a malformed code block that is hard to read.

* Update options for pgcopydb snapshot command

* Update options for pgcopydb compare

* Update options for pgcopydb dump subcommands

* Rename create_snapshot_command to snapshot_command

* Capitalize first letters of print and version command descriptions.

This change makes it easier to identify the connections of pgcopydb in
pg_stat_activity. The process title contains information on the current
operation, and the pid of the process is also included in the
application name. This way we can easily identify the connections of
pgcopydb in pg_stat_activity even if the process title is truncated.

Currently for each table we are creating and closing a new connection to
the target. Using a same connection would save us some time and resource,
both at pgcopydb and target side.

Also, seting GUCs only once for a given target connection session would
again save us resources. This would also reduce 8 extra log lines that are
currently produced for each table which otherwise would produce noise if we
have say, 10K tables.

The issue was that we weren't setting the snapshot for the query that
fetches the list of large object ids. This inconsistency led to errors when
trying to access large objects that weren't restored on target.

Error:
```
15:30:11 108 ERROR  [SOURCE 85] ERROR:  large object 16390 does not exist
15:30:11 108 ERROR  [SOURCE 85] Context: Failed to open large object 16390
15:30:11 107 ERROR  [SOURCE 83] ERROR:  large object 16392 does not exist
15:30:11 107 ERROR  [SOURCE 83] Context: Failed to open large object 16392
```

Here is a snippet of man page for msgsnd & msgrcv,

```
      int msgsnd(int msqid, const void *msgp, size_t msgsz, int msgflg);

       ssize_t msgrcv(int msqid, void *msgp, size_t msgsz, long msgtyp,
                      int msgflg);

DESCRIPTION
       The  msgsnd() and msgrcv() system calls are used to send messages to, and receive messages from, a System V message queue.  The calling process must have write
       permission on the message queue in order to send a message, and read permission to receive a message.

       The msgp argument is a pointer to a caller-defined structure of the following general form:

           struct msgbuf {
               long mtype;       /* message type, must be > 0 */
               char mtext[1];    /* message data */
           };

       The mtext field is an array (or other structure) whose size is specified by msgsz, a nonnegative integer value.  Messages of zero length (i.e., no mtext field)
       are  permitted.   The  mtype field must have a strictly positive integer value.  This value can be used by the receiving process for message selection (see the
       description of msgrcv() below).
```

According to it, the 3rd arg of both msgsnd and msgrcv must pass the
size of mtext i.e. excluding the mtype.

If we apply this logic into our context, we must pass the sizeof(QMessage->data), not sizeof(QMessage).

We can also find the corruption using valgrind like below,

```
valgrind --trace-children=yes --tool=memcheck --leak-check=no pgcopydb
copy table-data
```

It gives the following summary,

```
==640986== Syscall param msgsnd(msgp->mtext) points to uninitialised byte(s)
==640986==    at 0x4AC92AA: msgsnd (msgsnd.c:25)
==640986==    by 0x14E178: queue_send (queue_utils.c:100)
==640986==    by 0x162643: UnknownInlinedFun (table-data.c:403)
==640986==    by 0x162643: UnknownInlinedFun (table-data.c:295)
==640986==    by 0x162643: UnknownInlinedFun (table-data.c:232)
==640986==    by 0x162643: UnknownInlinedFun (table-data.c:209)
==640986==    by 0x162643: UnknownInlinedFun (table-data.c:117)
==640986==    by 0x162643: UnknownInlinedFun (table-data.c:89)
==640986==    by 0x162643: copydb_copy_all_table_data (table-data.c:57)
==640986==    by 0x11406E: cli_copy_table_data (cli_copy.c:395)
==640986==    by 0x1118B2: UnknownInlinedFun (commandline.c:71)
==640986==    by 0x1118B2: main (main.c:142)
==640986==  Address 0x1ffeeec5f0 is on thread 1's stack
==640986==  in frame #2, created by copydb_copy_all_table_data (table-data.c:38)
==640986==
==640986== Syscall param msgsnd(msgp->mtext) points to uninitialised byte(s)
==640986==    at 0x4AC92AA: msgsnd (msgsnd.c:25)
==640986==    by 0x14E178: queue_send (queue_utils.c:100)
==640986==    by 0x162AD8: UnknownInlinedFun (table-data.c:380)
==640986==    by 0x162AD8: UnknownInlinedFun (table-data.c:307)
==640986==    by 0x162AD8: UnknownInlinedFun (table-data.c:232)
==640986==    by 0x162AD8: UnknownInlinedFun (table-data.c:209)
==640986==    by 0x162AD8: UnknownInlinedFun (table-data.c:117)
==640986==    by 0x162AD8: UnknownInlinedFun (table-data.c:89)
==640986==    by 0x162AD8: copydb_copy_all_table_data (table-data.c:57)
==640986==    by 0x11406E: cli_copy_table_data (cli_copy.c:395)
==640986==    by 0x1118B2: UnknownInlinedFun (commandline.c:71)
==640986==    by 0x1118B2: main (main.c:142)
==640986==  Address 0x1ffeeec5f0 is on thread 1's stack
==640986==  in frame #2, created by copydb_copy_all_table_data (table-data.c:38)
==640986==

...
```

The above error disappers with the fix.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Add byte level details to list progress command

This patch allows reporting the total number of bytes transferred during
a copy operation. However, it has a limitation that the reported value
is only updated after each table is copied. This is because we do not
update table summary files while the copy is in progress.

Changes include:
- 3 new fields in table summary structures:
  - network.bytes: total number of bytes transmitted
  - network.bytes-pretty: pretty printed form of network.bytes
  - network.transmit-rate: pretty printed bytes transmitted per second
- Some new fields on the `pgcopydb list progress --json --summary`
  command:
  - steps[].network is a new field that exists for COPY step and has
    the following fields:
	- bytes: total number of bytes transmitted for all tables
	- bytes-pretty: pretty printed form of bytes
  - tables[].network is a new json value with the following fields:
	- bytes: total number of bytes transmitted for table
	- bytes-pretty: pretty printed form of bytes
	- transmit-rate: pretty printed bytes transmitted per second
- A new column on top level summary that shows total number of bytes
  copied.

* Address reviews

This commit addresses 2 issues raised in the review:
1. Using a SI standards when reporting data-rate units
2. Updating the relevant documentation with reproducible examples

The documentation update is done in a way that allowed me to run the
commands in a docker environment. Once I polish my changes in this
docker environment, I will update all the documentation with it and
share it in a separate PR.

* Use bit/s instead of byte/s for data-rate units

Before this update, exiting the process (either reaching ENDPOS or user-initiated abort like Ctrl-C) could leave us in the midst of an unfinished transaction. Upon resumption, PostgreSQL begins streaming from a consistent point, which might be the start of this incomplete transaction or from an earlier one. Our previous method tried to bypass all earlier messages until hitting the final message of the incomplete transaction, but this approach had its shortcomings, especially when the messages originated from several transactions back.

This commit introduces a more straightforward solution. Now, if the process is aborted and the last transaction is not complete, we'll issue a rollback command for that transaction. This ensures a clean and consistent state for a graceful exit and subsequent resumption.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* Fix copy paste error on comment
* Fix fatal message content

Currently, when switching between modes, the follow process cleans up context files, which include details like WAL segment size and timeline. However, we haven't been resetting the in-memory state that decides whether the context should be read from these files. This can lead to the usage of outdated timeline information while retriving the context[1][2][3].

This commit addresses this issue by resetting the in-memory state during context cleanup. This ensures that the system will re-read the context from the files after any cleanup, leading to more accurate and reliable timeline information.

[1] https://github.com/dimitri/pgcopydb/blob/8d305a51f82a83bacac6a3fcce716c9450505e57/src/bin/pgcopydb/ld_transform.c#L201-L212
[2] https://github.com/dimitri/pgcopydb/blob/8d305a51f82a83bacac6a3fcce716c9450505e57/src/bin/pgcopydb/follow.c#L428-L438
[3] https://github.com/dimitri/pgcopydb/blob/8d305a51f82a83bacac6a3fcce716c9450505e57/src/bin/pgcopydb/ld_apply.c#L206-L216



Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

If `statement_timeout` is set on the target COPY may get interrupted,
hence this change.

* Fix readthedocs.org configuration.

* Add docs/requirements.txt

* Fix YAML syntax.

* Fix YAML syntax again.

Follow up of
https://github.com/dimitri/pgcopydb/pull/533#pullrequestreview-1720295497.
Context is sometimes msgrcv acts weird and corrupts the value of other
stack allocated variables. Using heap allocated queue message as the
workaround until its root cause is known.

Presently, each LOB worker creates a new connection for every object copy,
impacting migration performance, especially with a large number of objects.
The fix here is simple: use the same connection within each worker.

However, while doing so, we encountered an unexpected error:

`BUG: call to pgsql_commit() without holding an open multi statement
connection`

Upon investigation, we discovered that the `connectionStatementType` was
set to an unusual value of 32767. In GDB, it was
(PGSQL_CONNECTION_MULTI_STATEMENT | unknown: 32766). This happens only
after the call of `msgrcv` function. There are a few theories; one suggests
that because both `msg` and `dst` are on the stack, `msgrcv` is overwriting
`dst` data when it should only change `msg`. Despite having
`-fstack-protector` enabled, we did not observe a "stack smashing detected"
error, so maybe this theory is incorrect.

In any case, moving `msg` to the heap seemed to resolve the issue. However,
it's important to fully understand this problem, as it may have
implications in other areas of the code as well.

We actually only need the possibly new endpos value when a process just
terminated, in order to decide what to do. So only fetch the sentinel endpos
value when a process just terminated.

* feat(container): multi arch build
* fix(container): copy whole repo for the build step; add label for gh pkg

Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>

Previously, when a connection timeout occurred with the default connection
timeout set to pgconnect_timeout (10 seconds), there were no retries. This
was because the total retry timeout (maxT) passed to `pgsql_set_retry_policy`
in `pgsql_set_interactive_retry_policy` was also set to pgconnect_timeout,
which had already elapsed during the initial connection attempt.

To address this issue, the maxT value has been updated to
POSTGRES_PING_RETRY_TIMEOUT. This change also means that maxT cannot be
configured by altering PGCONNECT_TIMEOUT. If one wants to configure maxT
they can change the values in defaults.h or we can make it configurable
through some environment variable in future.

After all this change `pgconnect_timeout` seems redundant so removed it.

* Update pgcopydb sentinel in the main follow process.

To avoid infinite looping when endpos has been reached it's important to
update our endpos value to the sentinel's one even in the main follow
process.

In passing, review some error messages.

Also refrain from stopping early from reading data from a PIPE when a
terminating signal is received, we should finish reading as per the comments
in the code.

* Process ENDPOS in transform and replay processes.

When the ENDPOS internal message is received from the PIPEs in replay mode,
check the current sentinel.endpos value on the source database to make sure
the ENDPOS message matches with the current setting, and that being the case
stop processing.

This commit removes the usage of transaction metadata file. Initially, it was used by the apply process to bypass transactions that were already applied. However, this approach had its challenges. Specifically, in live replay mode, a transaction with numerous statements could fill the UNIX PIPE (an IPC primitive used in replay mode), leading to a potential deadlock. This is because the apply process would be waiting for the transaction metadata file.

By eliminating the transaction metadata file, the apply process lets the transaction proceed and decides whether to apply or skip it based on the commit LSN during the commit phase.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

* remove PGPASSWORD environment processing from pgsql.c

instead use `pgsql->connectionString` containing the password
and adjust related function comments and removed unused PASSWORD_MASK

* fix mismatched safe{Source,Target}PGURI.password references

Apologies for such a small edit.
Always leave things better than you found them!

without this change `pgcopydb follow` would fail with e.g.:

```
Failed to parse decoding message for UPDATE on table "public"."tablename" which is not in our catalogs
```

As sprintf gets deprecated, update sources to use snprintf.

The for-loop is copying a string character by character and replacing `''` with `'`
by skipping a character if the current and next one are both `'`. When this
replacement doesn't occur then the target string will be filled completely
without a `\0' terminator at the end of the string.