Move malloc hooks into a compat DSO

Remove all malloc hook uses from core malloc functions and move it
into a new library libc_malloc_debug.so.  With this, the hooks now no
longer have any effect on the core library.

libc_malloc_debug.so is a malloc interposer that needs to be preloaded
to get hooks functionality back so that the debugging features that
depend on the hooks, i.e. malloc-check, mcheck and mtrace work again.
Without the preloaded DSO these debugging features will be nops.
These features will be ported away from hooks in subsequent patches.

Similarly, legacy applications that need hooks functionality need to
preload libc_malloc_debug.so.

The symbols exported by libc_malloc_debug.so are maintained at exactly
the same version as libc.so.

Finally, static binaries will no longer be able to use malloc
debugging features since they cannot preload the debugging DSO.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>

2 files changed, 16 insertions, 4 deletions

diff --git a/manual/memory.texi b/manual/memory.texi
index 31ee36b..93305f2 100644
--- a/manual/memory.texi
+++ b/manual/memory.texi
@@ -1256,8 +1256,9 @@ environment variable @env{MALLOC_ARENA_MAX} to the desired value.
 @cindex consistency checking, of heap
 
 You can ask @code{malloc} to check the consistency of dynamic memory by
-using the @code{mcheck} function.  This function is a GNU extension,
-declared in @file{mcheck.h}.
+using the @code{mcheck} function and preloading the malloc debug library
+@file{libc_malloc_debug.so} using the @var{LD_PRELOAD} environment variable.
+This function is a GNU extension, declared in @file{mcheck.h}.
 @pindex mcheck.h
 
 @deftypefun int mcheck (void (*@var{abortfn}) (enum mcheck_status @var{status}))
@@ -1368,7 +1369,10 @@ non-zero value less than 4, a special (less efficient) implementation is
 used which is designed to be tolerant against simple errors, such as
 double calls of @code{free} with the same argument, or overruns of a
 single byte (off-by-one bugs).  Not all such errors can be protected
-against, however, and memory leaks can result.
+against, however, and memory leaks can result.  Like in the case of
+@code{mcheck}, one would need to preload the @file{libc_malloc_debug.so}
+library to enable @code{MALLOC_CHECK_} functionality.  Without this
+preloaded library, setting @code{MALLOC_CHECK_} will have no effect.
 
 Any detected heap corruption results in immediate termination of the
 process.
@@ -1747,6 +1751,12 @@ penalties for the program if the debugging mode is not enabled.
 @c  fprintf dup (on newly-created stream) @aculock
 @c  __cxa_atexit (once) dup @asulock @aculock @acsmem
 @c  free dup @ascuheap @acsmem
+The @code{mtrace} function provides a way to trace memory allocation
+events in the program that calls it.  It is disabled by default in the
+library and can be enabled by preloading the debugging library
+@file{libc_malloc_debug.so} using the @code{LD_PRELOAD} environment
+variable.
+
 When the @code{mtrace} function is called it looks for an environment
 variable named @code{MALLOC_TRACE}.  This variable is supposed to
 contain a valid file name.  The user must have write access.  If the
diff --git a/manual/tunables.texi b/manual/tunables.texi
index ebdb562..e264e91 100644
--- a/manual/tunables.texi
+++ b/manual/tunables.texi
@@ -113,7 +113,9 @@ following tunables in the @code{malloc} namespace:
 
 @deftp Tunable glibc.malloc.check
 This tunable supersedes the @env{MALLOC_CHECK_} environment variable and is
-identical in features.
+identical in features. This tunable has no effect by default and needs the
+debug library @file{libc_malloc_debug.so} to be preloaded using the
+@code{LD_PRELOAD} environment variable.
 
 Setting this tunable to a non-zero value less than 4 enables a special (less
 efficient) memory allocator for the @code{malloc} family of functions that is