Unverified Commit 7e5fbb80 authored by Dimitri Fontaine's avatar Dimitri Fontaine Committed by GitHub
Browse files

Implement "live" streaming of changes from source to target. (#185)

* Implement "live" streaming of changes from source to target.

This patch introduces support for Unix Pipes and an internal streaming API
where changes flow through the system rather than being batched in files.
The following command line can be used to replay changes in a "live" fashion
from the source to the target database.

  $ pgcopydb stream receive --to-stdout
    | pgcopydb stream transform - -
    | pgcopydb stream apply -

The command `pgcopydb stream replay` is scheduled to glue together those
internal parts, and then the `pgcopydb clone --follow` will be extended to
automatically switch from catchup mode to replay mode when suitable.

* Code refactoring to share more bits.

Share the code that reads through a unix pipe (stream) line by line,
introducing a user callback facility with a (void *) context.

* First pass at adding tests for our low-level API/CLI.

* Switch producers to line buffered output mode.

This avoids partial reads of a part of a line, now we read a line at a time,
unless it doesn't fit in our 128 MB read buffer, a case that is handled in
the code already (albeit quite poorly at the time being).

Also improve logging, install set -o pipefail in the test suite.

* Add the new test case to GitHub Actions CI.

* Fix internal API to the new refactored code.
parent 5752a98e
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment