Unverified Commit 0595dda1 authored by Dimitri Fontaine's avatar Dimitri Fontaine Committed by GitHub
Browse files

Fix/index jobs (#106)

* Add OS details about fork failures (%m).

* Review some DEBUG messages to make them NOTICE instead.

* Fix preparing copy data specs for constraints.

* Review the process model for pgcopydb.

In order to handle database schemas with a very large number of objects,
change the pgcopydb process model to use a stable number of processes during
all operations.

To be able to do that, we need a communication system between processes so
that we can start the CREATE INDEX commands for a given table only when that
table data are known to have been COPYd entirely already. For that we
introduce a support layer for System V message queues (see msgget(2) etc).

In passing, also improve the schema querying done by pgcopydb: at startup
catalog queries are used to fetch the list of all tables to handle, and all
indexex to handle. Previous to this work, pgcopydb would still run a
separate query for each table to list its indexes. Instead, pgcopydb now
links the indexes to their table in its in-memory data structures for the
source schema, in a way that we don't have to run more queries later.

Finally, prior to this work VACUUM ANALYZE jobs would be created as soon as
a table COPY would complete, creating as many sub-processes as we have
tables total. Now a static set of VACUUM ANALYZE sub-processes are created
and a message queue is used to start working on a table when it is ready.

* Update docs for the new concurrency handling of pgcopydb.

* Fix list indentation.

* Fix race condition when indexes are all built.

It is necessary that all indexes have been built already to trigger the
creation of the constraints, and also that only one process is assigned with
this task.
parent f7a27c82
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment