It could be that we have several messages with the same LSN: I, L, E, for
instance. Take that into account in streamCheckResumePosition.

It might happen that several JSON messages have the same LSN in our stream,
usually that concerns KEEPALIVE and BEGIN and COMMIT message. Unless paying
attention to the latest message action, we could skip a BEGIN message that
has the same LSN as the latest message even if that message was a KEEPALIVE.

Fix this by keeping around the latest message action. It should be all we
need to distinguish a new message from the latest one.

The return value should be about the success of the operation, not about the
fact that the apply mode is enabled or disable: that information is tracked
properly in the StreamApplyContext instance given as a parameter.

We probably should parse the JSON file as soon as it exists rather than only
when a SQL file was already produced in the previous round.

On Azure Flexible Server for Postgres the COMMENT ON EXTENSION command is
not supported as a non-superuser and could cause migrations to fail. Waiting
for a fix in Azure, allow skipping that part of the pg_dump and restore
activity.

When resuming operations from pre-existing files on-disk, we implement some
cache invalidation by transforming the JSON file again from scratch, and
tracking progress in our internal data structures.

That's needed in case the previous version of the file terminates in the
middle of a transaction rather than at a clean transaction boundary.

The previous approach missed that the internal tracking state kept in the
StreamContext instance was not shared. Now it is.

The current debian Postgres version string is longer than we anticipated:

    "16beta1 (Debian 16~beta1-2.pgdg+~20230605.2256.g3f1aaaa)"

In passing, also make sure that `pgcopydb ping` sets the right GUCs even
when using Postgres 9.5, and allow more GUCs for that version too.

* Release v0.12.

* Fix a warning found by debian C compiler.

When dropping the replication slot on the source database make sure to also
remove the snapshot and slot files.

This time we add unit testing to make sure we get it right and avoid
regressions.

When the streaming is interrupted in the middle of a JSON/SQL file and then
resumes in replay mode, the transform process needs to read and parse the
latest JSON/SQL file in case it's being restarted in the middle of a
transaction.

The transform process rewrites the SQL file even when it was done
previously, as a way of implementing "cache invalidation" where the source
of truth is the JSON file. This PR implements the same approach to the
replay mode start-up sequence.

At flushAndSendFeedback time make sure that the current LSN position that we
have written is updated as our next startpos. This also makes the startpos
available to the other sub-modules (such as transform) at their start-time
and when switching between operation modes.

* Have --skip-extensions also skip pre-existing schemas on the target.

When skipping extensions we assume that another process did take care of
creating the needed extensions on the target instance, such as e.g. the
`pgcopydb copy extensions` command.

The "CREATE EXTENSION" command may create a schema, and we need to skip
schemas that such a command would have created on the target already. It is
not always possible to track schemas created by extensions, some extension
scripts are going out of their way to make sure of that:

    DO $$
      CREATE SCHEMA tiger_data;
    $$;

So we track all the schemas that already exist on the target database and
skip creating them when the --skip-extensions command is used.

* Fix copydb_schema_already_exists to follow API, assorted fixes.

* Update documentation, improve error reporting.

* We don't have support for pg_partman and postgis_tiger_geocoder yet.

These extensions require superuser for dump&restore.

Fix pgsql.c for the case when we fail to execute a SQL command without
parameters, such as e.g. the very simple "COMMIT;" command. First, we might
not be given a SQLSTATE, and second, we need to pay attention about that
lack of debugParameters.

Also improve `pgcopydb clone` command to avoid importing a snapshot in the
top-level process. This process might need to export a snapshot and hold-on
to it while sub-processes are running, but never executes SQL itself so
won't need to import a snapshot in its own transaction.

- Fix BYTEA handling: refrain from using C-Style Escapes for BYTEA values.
- Fix LogicalMessageValueEq for INT8OID (typo).

Escape string constants in our SQL files by following the Postgres
documentation about String Constants With C-Style Escapes, see:

  https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-STRINGS

This format helps guarantee that a single SQL statement in our SQL files are
always found on one-line only, which we use in the apply process to avoid
having to actually parse the SQL files.

The logical decoding messages both from test_decoding and wal2json might
contain the same column in both WHERE and SET clauses with the same value.
When that's the case, it's best to reduce the UPDATE statement and refrain
from adding the same column-value pair in both clauses.

Also that's not just an optimisation now, as without this the following
could happen:

    ERROR:  column "id" can only be updated to DEFAULT
    DETAIL:  Column "id" is an identity column defined as GENERATED ALWAYS.

The test_decoding logical decoding output module sometimes issues UPDATE
statements with a single list of columns and values, and the client side is
then supposed to know what columns are in the WHERE clause and which in the
SET clause.

To know that, we need to have a version of the catalogs in-memory and
implement lookups. This PR does that.

We want to grab the password and remove it from the connection string so
that logs don't contain it, and we set the password in the environment of
the child processes when running pg_dump etc commands.

Fix the code so that's what actually happens. See #323.

* Fix cli_list environment variable parsing.

Arrange to share more code from cli_common, and in passing introduce a new
structure to handle the bytes and bytesPretty within a single unit.

* Review Postgres URI handling and parsing APIs.

Make the internal parsing APIs use dynamic memory, and then keep a single
instance of the connections strings, safe versions of them (without the
password), and the password separated in-memory.

* Only compute safeURI when logging the connection string.

* Use pg_strdup and removed NULL checks

---------

Co-authored-by: Chanderprabh Jain <chajain@microsoft.com>
Co-authored-by: cpj2195 <chanderprabhjain95@gmail.com>

* Refactor code to prepare for test_decoding improvements.

To finish implementing test_decoding support, we need to be able to look-up
attribute names for them being part of a primary-key definition, in order to
process UPDATE statements without the old-key: and new-key: parts.

In order to do that we prepare internal data structures to make it easy to
provide our internal catalogs representation to the transform sub-processes,
both when using the catchup mode API (from disk) or the replay mode
API (from a stream).

In passing, share more code in the ld_transform.c module in order to
simplify maintenance. It also looks like we're going to use much less memory
to implement the same job, that's a nice side effect of this refactoring.

* Introduce a new pgcopydb internal message: ENDPOS.

When reaching endpos in the middle of a transaction we should stop
processing the stream, transform what we have, and apply up to the last full
transaction, ignoring the partial one at the end.

Because of the streaming approach taken by pgcopydb, the next best thing we
can do is to ROLLBACK the last partial transaction while still being able to
recognise that --endpos has been reached.

Previously to this patch, pgcopydb would use the endpos LSN to forge a
keepalive message, meaning that we would then update our replay_lsn and
replication origin tracking to a position in the middle of a transaction we
did not replay.

In case when the user then restart pgcopydb with a new endpos then we would
skip one transaction which was written off as already replayed. This patch
fixes this situation.

* Fix resuming CDC after chaging endpos again.

It should be possible to reset endpos to a later point in time and resume
replaying changes up to the new point in time, even when the previously set
endpos did fell in the middle of a transaction.

* Fix comparing with startpos from replication slot.

When comparing startpos with a previously written JSON file we need to avoid
writing the same LSN twice, but that computation/guard doesn't apply when
starting CDC the first time from the replication slot: then we want to write
starting at the startpos, not skip it.

Currently, we use txnCommitLSN to determine if we have reached the
startpos. If the txnCommitLSN is invalid, we read the transaction metadata
file from disk or wait until it exists. However, there is a bug that we
continue to check the validity of txnCommitLSN and read metadata file even
after reaching startpos. This additional work is unnecessary.

Furthermore, this commit partially address an issue where a transaction is
split by KEEPALIVE message during the replay phase. Previously, we would
wait for transaction metadata file to be created, which would never happen
and eventually lead to migration failure.

We have the information now, and we need a way to share that with the
transform process for primary-key lookup and type information for parsing in
some cases too.

Schema queries for dependencies and sequences for exclusion filters where
both having bugs, impacting the selection of the objects to be filtered-out
when using exclusion filters.

Fixes #280.

Postgres GUC session_replication_role should be used and set to 'replica'
when applying changes from the source database system, to avoid triggers
firing again.

- When the filtering results in an empty list of selected tables, make
    sure we don't enter infinite loop waiting for events that won't happen.

  - Make sure to prepare the pgcopydb_table_size (temporary) table when
    doing the pgcopydb copy ... commands that need it.

  - Ensure proper memory allocation and calloc() error detection when
    printing the summary tables.

  - Work around what looks like a compiler bug when int errors = 0; actually
    sets the errors counter to 32759 on macOs. For that the COPY workers are
	now always in "fail-fast" mode, we can't count the errors anymore.

* Fix file rotation issue during streaming.

Fixes a bug that occurs when multiple transactions have BEGIN and other
statements in one WAL file, while their COMMITs are in the next file. The
issue arises when pgcopydb receives the COMMIT of the first transaction,
causing it to move to the next file. However, upon receiving the BEGIN of
the next transaction, it incorrectly goes back to the previous file,
resulting in an inconsistent file rotation pattern.

To address this issue, this commit introduces tracking the LSN of the last
message written to the file. With this, file rotation is skipped if the
incoming message has an LSN less than the lastWrittenLSN, ensuring that
writes consistently occur in the current or next file.

* Rename lastWrite to lastWriteTime.

* Fix missing commit_lsn for continued transaction.

When a transaction is split into multiple JSON WAL files, the transformed
BEGIN statement does not include the commit_lsn since the COMMIT statement
appears in the next file. This poses a challenge during the apply process,
as the commit_lsn is necessary to determine when to start applying the
current and subsequent changes.

To resolve this issue, this fix parse the commit_lsn during the apply phase
from a transaction metadata file. This file is created during the transform
phase as soon as the COMMIT message is processed. However, this also
implies that the apply process must wait for the creation of this file
before it can begin applying the changes.

Fixes: #299

This ensures that Postgres object names are properly handled even when they
may contain multiple lines, which is accepted, possible... and then found in
the field.

In passing, ensure that we VACUUM ANALYZE tables without indexes. It's
almost relevant to that patch because the failure mode of multiple-lines
table names would break our vacuum module.

Using semaphores for work-sharing isn't ideal and brings some complexity and
inefficiencies (each worker process loops over the whole table array). Now
that we have introduced Sys V queues, use that mechanism instead to share
the workload between table-data COPY workers.

We still need to use a semaphore to create the lockfile and donefile so that
we can track progress, resume from interrupted operations, and discover if
all the partitions of a single table have been processed already. That said,
we don't have to deal with same-table concurrency using the semaphore and
files, the queue solves that for us.

This is currently a moving target. Version 3.3.3 adds topology_id_seq as an
Extension Configuration Table, and pgcopydb is not ready to handle a
sequence there.

This fixes the issues where we skip applying I/U/D/T/C leading to the
warning "there is already a transaction in progress" when a txn continues
to next file and first statement of that file is not a BEGIN or KEEPALIVE
statement. In that case, `reachedStartPos` is never set to true and apply
never starts.

This happens because `reachedStartPos` is set to `false` everytime a new
file is processed.

During prefetching, when a transaction continues to the next file and the
first line of the next file (which is also the only remaining statement of
the transaction) is a COMMIT message, it is incorrectly interpreted as an
empty transaction. This fix addresses that issue.

This fixes the missing `xid` and `timestamp` for continued transactions,
```sql
COMMIT; -- {"xid":0,"lsn":"6/A50001C8","timestamp":""}
```

This timestamp is used in `pgsql_replication_origin_xact_setup` which we
update along with `COMMIT;` statement. So it makes more sense to put
COMMIT's `timestamp` as origin timestamp.

Retaining the current update with `BEGIN` action as well. This avoids
`timestamp` being empty for `BEGIN` SQL statement in the file when COMMIT
spills to next SQL file.

Check the number of messages in the queue before entering a message receive
loop that won't exit before reading in the STOP message. When the queue is
empty, this turns into an infinite loop.