Fix deadlock when pthread_atfork handler calls pthread_atfork or dlclose

In multi-threaded programs, registering via pthread_atfork,
de-registering implicitly via dlclose, or running pthread_atfork
handlers during fork was protected by an internal lock.  This meant
that a pthread_atfork handler attempting to register another handler or
dlclose a dynamically loaded library would lead to a deadlock.

This commit fixes the deadlock in the following way:

During the execution of handlers at fork time, the atfork lock is
released prior to the execution of each handler and taken again upon its
return.  Any handler registrations or de-registrations that occurred
during the execution of the handler are accounted for before proceeding
with further handler execution.

If a handler that hasn't been executed yet gets de-registered by another
handler during fork, it will not be executed.   If a handler gets
registered by another handler during fork, it will not be executed
during that particular fork.

The possibility that handlers may now be registered or deregistered
during handler execution means that identifying the next handler to be
run after a given handler may register/de-register others requires some
bookkeeping.  The fork_handler struct has an additional field, 'id',
which is assigned sequentially during registration.  Thus, handlers are
executed in ascending order of 'id' during 'prepare', and descending
order of 'id' during parent/child handler execution after the fork.

Two tests are included:

* tst-atfork3: Adhemerval Zanella <adhemerval.zanella@linaro.org>
  This test exercises calling dlclose from prepare, parent, and child
  handlers.

* tst-atfork4: This test exercises calling pthread_atfork and dlclose
  from the prepare handler.

[BZ #24595, BZ #27054]

Co-authored-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Adhemerval Zanella  <adhemerval.zanella@linaro.org>
(cherry picked from commit 52a103e237329b9f88a28513fe7506ffc3bd8ced)

1 files changed, 13 insertions, 13 deletions

diff --git a/include/register-atfork.h b/include/register-atfork.h
index be63113..5ebe5a0 100644
--- a/include/register-atfork.h
+++ b/include/register-atfork.h
@@ -26,6 +26,7 @@ struct fork_handler
   void (*parent_handler) (void);
   void (*child_handler) (void);
   void *dso_handle;
+  uint64_t id;
 };
 
 /* Function to call to unregister fork handlers.  */
@@ -39,19 +40,18 @@ enum __run_fork_handler_type
   atfork_run_parent
 };
 
-/* Run the atfork handlers and lock/unlock the internal lock depending
-   of the WHO argument:
-
-   - atfork_run_prepare: run all the PREPARE_HANDLER in reverse order of
-			 insertion and locks the internal lock.
-   - atfork_run_child: run all the CHILD_HANDLER and unlocks the internal
-		       lock.
-   - atfork_run_parent: run all the PARENT_HANDLER and unlocks the internal
-			lock.
-
-   Perform locking only if DO_LOCKING.  */
-extern void __run_fork_handlers (enum __run_fork_handler_type who,
-				 _Bool do_locking) attribute_hidden;
+/* Run the atfork prepare handlers in the reverse order of registration and
+   return the ID of the last registered handler.  If DO_LOCKING is true, the
+   internal lock is held locked upon return.  */
+extern uint64_t __run_prefork_handlers (_Bool do_locking) attribute_hidden;
+
+/* Given a handler type (parent or child), run all the atfork handlers in
+   the order of registration up to and including the handler with id equal
+   to LASTRUN.  If DO_LOCKING is true, the internal lock is unlocked prior
+   to return.  */
+extern void __run_postfork_handlers (enum __run_fork_handler_type who,
+                                     _Bool do_locking,
+                                     uint64_t lastrun) attribute_hidden;
 
 /* C library side function to register new fork handlers.  */
 extern int __register_atfork (void (*__prepare) (void),