Also, for test_decoding plugin we would require the catalog information so it's better to let it initialise first and then we start the prefetch.

When the replication connection is interrupted due to network or server issues, although there is a retry logic in place, it fails to work effectively. This is because the interruption may leave the last transaction in a partial state, leading to a failure when the ld_transform module attempts to transform the message.

Solution: Add a ROLLBACK message when the last transaction is partially written and the stream is in retry mode.

Use the following query on psql to test the network interruption behaviour while doing DML on source.

```.sql
 SELECT pg_cancel_backend(pid) FROM pg_stat_activity where application_name like '%pgcopydb%' and query ilike '%START_REPL%';

\watch 5
```

The following error message will be loged on console when it is interrupted,

> 11:40:54.061 94512 ERROR  unexpected termination of replication stream:
11:40:54.061 94512 ERROR  ERROR:  canceling statement due to user request
11:40:54.061 94512 ERROR  CONTEXT:  slot "pgcopydb", output plugin "test_decoding", in the change callback, associated LSN 7B/704E6A18
11:40:54.061 94512 WARN   Streaming got interrupted at 7B/704E6A18, reconnecting in 1s

Without this fix, ld_transform would fail with the following console error logs,

>2024-02-28 18:44:44.785 37 ERROR  ld_transform.c:1099       Failed to parse BEGIN: transaction already in progress
2024-02-28 18:44:44.785 37 ERROR  ld_transform.c:924        Failed to parse JSON message: {"action":"B","xid":"3560764890","lsn":"DD68/8ADC210","timestamp":"2024-02-28 17:59:31.756879+0000","message":{"action":"B","xid":3560764890}}
2024-02-28 18:44:44.789 37 ERROR  ld_transform.c:686        Stream transform worker encountered 1 errors, see above for details
2024-02-28 18:44:44.789 37 INFO   follow.c:758              Transform process has terminated
2024-02-28 18:44:44.855 19 ERROR  follow.c:993              Subprocess transform with pid 37 has exited with error code 12
2024-02-28 18:44:45.016 19 ERROR  follow.c:471              Failed to transform 2 messages from the queue, see above for details

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>

This fixes the case where copydb_table_data_worker receives a
`QMSG_TYPE_STOP` and we return within the switch statement without calling
catalog_delete_process outside the swtich body.

There were several issues with the CFLAGS in the pgcopydb Makefile. This patch fixes them.

1. Removal of conditional logic for USE_SECURITY_FLAGS

Historically, we needed to support some linux distros such as RHEL6 that
did not support the subset of security flags that are required for
Microsoft Compliance. However, we are now at a point where we can remove
this conditional logic and always use those security flags. This patch
removes the conditional logic and always uses the security flags.

2. Fix typo in JENKINS_CFLAGS

The JENKINS_CFLAGS variable was misspelled as CFAGS. This patch fixes
the typo.

3. Remove unnecessary override of CFLAGS

The override of CFLAGS was unnecessary and was causing the security
flags to be overridden. This patch removes the override. If we do not
have -Wformat option, -Wformat-security which is a required compiler
flag for compliance is ignored.

Relevant error message:
cc1: error: -Wformat-security ignored without -Wformat [-Werror=format-security]

Extension configuration tables can have foreign key dependencies between
them. If the configuration tables are restored in the incorrect order,
it can cause foreign key constraint violations.

This change ensures that the configuration tables are ordered according
to the dependency order in `pg_constraint`.

We always call `summary_start_timing` regardless of whether the section has been already done or not., which causing insertion(insert or replace) of default value to `done_time_epoch` because `summary_start_timing` won't be called for already done section.

Solution: Check whether the section is already completed using `runState` and exit early without calling `summary_start_timing` for already done section.

Related to https://github.com/dimitri/pgcopydb/issues/692



Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

pgcopydb recently switched to Boehm-Demers-Weiser Garbage Collector[1] and replaced
malloc family with GC_malloc and friends. However, strdup is not replaced
to GC_strdup.

Solution: Use macros to replace strdup/strndup with it's GC variant.

[1] https://github.com/dimitri/pgcopydb/pull/647



Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

Given ALTER TABLE ... DROP COLUMN ... we could well have a table without an
attribute with attnum == 1. Our code used to rely on that to happen in order
to allocate our SourceTableAttribute array in-memory, before filling it in.

Instead, use a window-function in the SQL query that we run against our
internal SQLite catalogs, using count(*) over(order by attnum) to map
whatever the Postgres attnum are into a proper gapless series that starts at
one.

This function does not exists anymore, but the header was still defined and
the call site was not removed yet.

Instead of storing the table size in the source database, store it in our internal catalogs in the SQLite database. This allows better compatibility with read-only source databases.

At the moment this removes the possibility for the user to provide their own table size data in order to avoid the `pg_table_size()` query which is known to be a long running query in some platforms. A later improvement could be done to enable external table size input again.

Missing xid triggers the following error during apply/replay.

"BUG: parseTxnMetadataFile is called with transaction xid: 0"

Solution: Use xid from commit message as a transaction xid.

Unlike wal2json, test_decoding don't have xid in the DML logical
messages. So we use the xid from the COMMIT message to update the
transaction xid.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

Solution: fflush stdout after printing the snapshot id. By default stdout is
buffered and flushed only when the buffer is full.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>

Before embedding SQLite we could not depend on aggregates FILTER clause to
be available, and as a result we had to aggregate the attribute list in our
own C code. Turns out we had strange bugs around that, that hopefully the
very simple SQL query will not have.

Follow up of #595

This allows to ensure we're always using a known version of the SQLite lib
and allows pgcopydb code to use some "recent" features on all OS targets.

Such features include the RETURNING clause for DMLs, the FILTER clause for
aggregate function calls, and time formatting function or JSON functions.

See https://sqlite.org/amalgamation.html for details about that distribution
mode for SQLite.

Before this change we had 153 files on the main branch that did broke at
least one of our editorconfig rules. After this change, all of our files
honor the rules that are partially updated.

There are 2 places that are exempt from our rules:
- external libraries in src/bin/lib/
- debian directory in debian/
- test file from outside source in tests/extensions/

You can see the .editorconfig file in each of these directories for more
details.

I used a copy of Citus' CONTRIBUTING.md as a starting point.

I removed sections that are not relevant to pgcopydb, and added some
more sections about documentation and running tests.

I also added references to the CONTRIBUTING.md file in the README.md and
in documentation page about installing pgcopydb. While I was at it, I
also removed the outdated installation instructions from the README.md
file, and replaced them with a reference to the documentation page.

Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>

* Fix confusion around reporting operations in progress.

The SQLite refactor meant rewriting all the SourceTableArray loops to use a
query result iterator instead. Well there is a place for a SourceTableArray
loop still, and that's the pgcopydb list progress thing.

* Fix the table-in-copy SQLite query.

The query was meant to report actual duration and transmitted bytes for each
table in-progress, but the join and sum() operations are creating problems
in the SQLite query and the rest of the progress.c code was not prepared to
benefit from the extra information anyway.

The SQLite problem is that the sum(s.duration), sum(s.bytes) causes a single
row output when the dataset is empty, and the other columns are all set to
NULL. That might be because of the lack of a GROUP BY clause.

Simplify that aspect by removing the extra information for now.

Since the move of the summary and process information to the SQLite
catalogs, we don't have a progress JSON file on-disk with parts of the setup
anymore.

The code needed to be updated to behave correctly without having
--table-jobs and --index-jobs information.

* Use Boehm-Demers-Weiser Garbage Collector.

From their own documentation:

> The Boehm-Demers-Weiser conservative garbage collector can be used as a
> garbage collecting replacement for C malloc or C++ new. It allows you to
> allocate memory basically as you normally would, without explicitly
> deallocating memory that is no longer useful. The collector automatically
> recycles memory when it determines that it can no longer be otherwise
> accessed.

In pgcopydb C code base it's been hard to maintain proper malloc/free
concerns while also implementing proper error-handling, which is a common
problem when writing C code. In order to ease the maintenance of the code
and reduce production hazards, the best way to proceed looks like automating
the memory management altogether by using a Garbage Collector.

* Revert removal of regree() calls.

* Get rid of LinesBuffer.ownsBuffer.

Now that libgc is handling the memory for us, we don't need to keep track of
the ownership of the buffer anymore.

I observed frequent `[SQLite] PRAGMA journal_mode = WAL` when it should be
run only once as it's a sqlite database property, not something that has be
set for every connection. This might fix
https://github.com/dimitri/pgcopydb/actions/runs/7614380466/job/20736769276?pr=647

```
2024-01-22 16:45:01.923 44 ERROR  catalog.c:7129            [SQLite]: PRAGMA journal_mode = WAL failed: database is locked
```

- Adds CI to check that the documentation templates are up-to-date.
- Add a new `help` command to the documentation templates. This command
  prints the help message for all the sub-commands of `pgcopydb` in a
  single file. This is useful for users who want to see all the
  sub-commands in one place.
- Remove dependency on installing the binary before updating the
  documentation templates. This allows users to update the documentation
  templates without installing the binary. One can set the `PGCOPYDB`
  environment variable to point to the binary to use for updating the
  documentation templates.
- Fix a casing issue in command descriptions. The first letter of the
  first word in the description of each command should be capitalized.

When registering the catalog setup, which includes the same-table
concurrency threshold, only consider that value when the table partitions
have been populated in the cache already. Otherwise the table partitions are
going to be computed again anyway.

* Improve SQLite write performance with journal_mode as WAL.

* Batch initial catalog setup write in a single transaction.

The SQLite patch series introduced a bug where we would skip extensions even
when the option --skip-extension is not used.

Also adjust the "pgcopydb copy extensions" to the recent changes where we
use a sub-process under a supervisor to simplify snapshot handling.

Make sure we don't try and allocate a zero-size memory area, which can lead
to "corrupted size vs. prev_size". Review all the calloc() call sites and
protect them better when they need to be.

Make it systematic to first count the lines and then allocate the right
amount of memory for the array of pointers to the start of each line, and
make sure we don't try and allocate a zero-size array, which can lead to
"corrupted size vs. prev_size" errors at run-time.