[libc++] Fix the behavior of throwing `operator new` under -fno-exceptions (#69498)

In D144319, Clang tried to land a change that would cause some functions
that are not supposed to return nullptr to optimize better. As reported
in https://reviews.llvm.org/D144319#4203982, libc++ started seeing
failures in its CI shortly after this change was landed.

As explained in D146379, the reason for these failures is that libc++'s
throwing `operator new` can in fact return nullptr when compiled with
exceptions disabled. However, this contradicts the Standard, which
clearly says that the throwing version of `operator new(size_t)` should
never return nullptr. This is actually a long standing issue. I've
previously seen a case where LTO would optimize incorrectly based on the
assumption that `operator new` doesn't return nullptr, an assumption
that was violated in that case because libc++.dylib was compiled with
-fno-exceptions.

Unfortunately, fixing this is kind of tricky. The Standard has a few
requirements for the allocation functions, some of which are impossible
to satisfy under -fno-exceptions:
1. `operator new(size_t)` must never return nullptr
2. `operator new(size_t, nothrow_t)` must call the throwing version and
return nullptr on failure to allocate
3. We can't throw exceptions when compiled with -fno-exceptions

In the case where exceptions are enabled, things work nicely.
`new(size_t)` throws and `new(size_t, nothrow_t)` uses a try-catch to
return nullptr. However, when compiling the library with
-fno-exceptions, we can't throw an exception from `new(size_t)`, and we
can't catch anything from `new(size_t, nothrow_t)`. The only thing we
can do from `new(size_t)` is actually abort the program, which does not
make it possible for `new(size_t, nothrow_t)` to catch something and
return nullptr.

This patch makes the following changes:
1. When compiled with -fno-exceptions, the throwing version of `operator
new` will now abort on failure instead of returning nullptr on failure.
This resolves the issue that the compiler could mis-compile based on the
assumption that nullptr is never returned. This constitutes an API and
ABI breaking change for folks compiling the library with -fno-exceptions
(which is not the general public, who merely uses libc++ headers but use
a shared library that has already been compiled). This should mostly
impact vendors and other folks who compile libc++.dylib themselves.

2. When the library is compiled with -fexceptions, the nothrow version
of `operator new` has no change. When the library is compiled with
-fno-exceptions, the nothrow version of `operator new` will now check
whether the throwing version of `operator new` has been overridden. If
it has not been overridden, then it will use an implementation
equivalent to that of the throwing `operator new`, except it will return
nullptr on failure to allocate (instead of terminating). However, if the
throwing `operator new` has been overridden, it is now an error NOT to
also override the nothrow `operator new`. Indeed, there is no way for us
to implement a valid nothrow `operator new` without knowing the exact
implementation of the throwing version.

In summary, this change will impact people who fall into the following
intersection of conditions:
- They use the libc++ shared/static library built with `-fno-exceptions`
- They do not override `operator new(..., std::nothrow_t)`
- They override `operator new(...)` (the throwing version)
- They use `operator new(..., std::nothrow_t)`

We believe this represents a small number of people.

Fixes #60129
rdar://103958777

Differential Revision: https://reviews.llvm.org/D150610

1 files changed, 110 insertions, 62 deletions

diff --git a/libcxxabi/src/stdlib_new_delete.cpp b/libcxxabi/src/stdlib_new_delete.cpp
index f8a00ec..b802559 100644
--- a/libcxxabi/src/stdlib_new_delete.cpp
+++ b/libcxxabi/src/stdlib_new_delete.cpp
@@ -7,7 +7,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "__cxxabi_config.h"
+#include "abort_message.h"
+#include "include/overridable_function.h" // from libc++
 #include <__memory/aligned_alloc.h>
+#include <cstddef>
 #include <cstdlib>
 #include <new>
 
@@ -25,6 +28,20 @@
 #  error libc++ and libc++abi seem to disagree on whether exceptions are enabled
 #endif
 
+inline void __throw_bad_alloc_shim() {
+#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
+  throw std::bad_alloc();
+#else
+  abort_message("bad_alloc was thrown in -fno-exceptions mode");
+#endif
+}
+
+#define _LIBCPP_ASSERT_SHIM(expr, str)                                                                                 \
+  do {                                                                                                                 \
+    if (!expr)                                                                                                         \
+      abort_message(str);                                                                                              \
+  } while (false)
+
 // ------------------ BEGIN COPY ------------------
 // Implement all new and delete operators as weak definitions
 // in this shared library, so that they can be overridden by programs
@@ -46,64 +63,76 @@ static void* operator_new_impl(std::size_t size) {
   return p;
 }
 
-_LIBCPP_WEAK
-void* operator new(std::size_t size) _THROW_BAD_ALLOC {
+_LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE _LIBCPP_WEAK void* operator new(std::size_t size) _THROW_BAD_ALLOC {
   void* p = operator_new_impl(size);
-#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   if (p == nullptr)
-    throw std::bad_alloc();
-#endif
+    __throw_bad_alloc_shim();
   return p;
 }
 
-_LIBCPP_WEAK
-void* operator new(size_t size, const std::nothrow_t&) noexcept {
+_LIBCPP_WEAK void* operator new(size_t size, const std::nothrow_t&) noexcept {
+#ifdef _LIBCPP_HAS_NO_EXCEPTIONS
+#  if _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION
+  _LIBCPP_ASSERT_SHIM(
+      !std::__is_function_overridden(static_cast<void* (*)(std::size_t)>(&operator new)),
+      "libc++ was configured with exceptions disabled and `operator new(size_t)` has been overridden, "
+      "but `operator new(size_t, nothrow_t)` has not been overridden. This is problematic because "
+      "`operator new(size_t, nothrow_t)` must call `operator new(size_t)`, which will terminate in case "
+      "it fails to allocate, making it impossible for `operator new(size_t, nothrow_t)` to fulfill its "
+      "contract (since it should return nullptr upon failure). Please make sure you override "
+      "`operator new(size_t, nothrow_t)` as well.");
+#  endif
+
+  return operator_new_impl(size);
+#else
   void* p = nullptr;
-#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   try {
-#endif // _LIBCPP_HAS_NO_EXCEPTIONS
     p = ::operator new(size);
-#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   } catch (...) {
   }
-#endif // _LIBCPP_HAS_NO_EXCEPTIONS
   return p;
+#endif
 }
 
-_LIBCPP_WEAK
-void* operator new[](size_t size) _THROW_BAD_ALLOC { return ::operator new(size); }
+_LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE _LIBCPP_WEAK void* operator new[](size_t size) _THROW_BAD_ALLOC {
+  return ::operator new(size);
+}
 
-_LIBCPP_WEAK
-void* operator new[](size_t size, const std::nothrow_t&) noexcept {
+_LIBCPP_WEAK void* operator new[](size_t size, const std::nothrow_t&) noexcept {
+#ifdef _LIBCPP_HAS_NO_EXCEPTIONS
+#  if _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION
+  _LIBCPP_ASSERT_SHIM(
+      !std::__is_function_overridden(static_cast<void* (*)(std::size_t)>(&operator new[])),
+      "libc++ was configured with exceptions disabled and `operator new[](size_t)` has been overridden, "
+      "but `operator new[](size_t, nothrow_t)` has not been overridden. This is problematic because "
+      "`operator new[](size_t, nothrow_t)` must call `operator new[](size_t)`, which will terminate in case "
+      "it fails to allocate, making it impossible for `operator new[](size_t, nothrow_t)` to fulfill its "
+      "contract (since it should return nullptr upon failure). Please make sure you override "
+      "`operator new[](size_t, nothrow_t)` as well.");
+#  endif
+
+  return operator_new_impl(size);
+#else
   void* p = nullptr;
-#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   try {
-#endif // _LIBCPP_HAS_NO_EXCEPTIONS
     p = ::operator new[](size);
-#ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   } catch (...) {
   }
-#endif // _LIBCPP_HAS_NO_EXCEPTIONS
   return p;
+#endif
 }
 
-_LIBCPP_WEAK
-void operator delete(void* ptr) noexcept { std::free(ptr); }
+_LIBCPP_WEAK void operator delete(void* ptr) noexcept { std::free(ptr); }
 
-_LIBCPP_WEAK
-void operator delete(void* ptr, const std::nothrow_t&) noexcept { ::operator delete(ptr); }
+_LIBCPP_WEAK void operator delete(void* ptr, const std::nothrow_t&) noexcept { ::operator delete(ptr); }
 
-_LIBCPP_WEAK
-void operator delete(void* ptr, size_t) noexcept { ::operator delete(ptr); }
+_LIBCPP_WEAK void operator delete(void* ptr, size_t) noexcept { ::operator delete(ptr); }
 
-_LIBCPP_WEAK
-void operator delete[](void* ptr) noexcept { ::operator delete(ptr); }
+_LIBCPP_WEAK void operator delete[](void* ptr) noexcept { ::operator delete(ptr); }
 
-_LIBCPP_WEAK
-void operator delete[](void* ptr, const std::nothrow_t&) noexcept { ::operator delete[](ptr); }
+_LIBCPP_WEAK void operator delete[](void* ptr, const std::nothrow_t&) noexcept { ::operator delete[](ptr); }
 
-_LIBCPP_WEAK
-void operator delete[](void* ptr, size_t) noexcept { ::operator delete[](ptr); }
+_LIBCPP_WEAK void operator delete[](void* ptr, size_t) noexcept { ::operator delete[](ptr); }
 
 #if !defined(_LIBCPP_HAS_NO_LIBRARY_ALIGNED_ALLOCATION)
 
@@ -127,70 +156,89 @@ static void* operator_new_aligned_impl(std::size_t size, std::align_val_t alignm
   return p;
 }
 
-_LIBCPP_WEAK
-void* operator new(std::size_t size, std::align_val_t alignment) _THROW_BAD_ALLOC {
+_LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE _LIBCPP_WEAK void*
+operator new(std::size_t size, std::align_val_t alignment) _THROW_BAD_ALLOC {
   void* p = operator_new_aligned_impl(size, alignment);
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   if (p == nullptr)
-    throw std::bad_alloc();
-#  endif
+    __throw_bad_alloc_shim();
   return p;
 }
 
-_LIBCPP_WEAK
-void* operator new(size_t size, std::align_val_t alignment, const std::nothrow_t&) noexcept {
+_LIBCPP_WEAK void* operator new(size_t size, std::align_val_t alignment, const std::nothrow_t&) noexcept {
+#  ifdef _LIBCPP_HAS_NO_EXCEPTIONS
+#    if _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION
+  _LIBCPP_ASSERT_SHIM(
+      !std::__is_function_overridden(static_cast<void* (*)(std::size_t, std::align_val_t)>(&operator new)),
+      "libc++ was configured with exceptions disabled and `operator new(size_t, align_val_t)` has been overridden, "
+      "but `operator new(size_t, align_val_t, nothrow_t)` has not been overridden. This is problematic because "
+      "`operator new(size_t, align_val_t, nothrow_t)` must call `operator new(size_t, align_val_t)`, which will "
+      "terminate in case it fails to allocate, making it impossible for `operator new(size_t, align_val_t, nothrow_t)` "
+      "to fulfill its contract (since it should return nullptr upon failure). Please make sure you override "
+      "`operator new(size_t, align_val_t, nothrow_t)` as well.");
+#    endif
+
+  return operator_new_aligned_impl(size, alignment);
+#  else
   void* p = nullptr;
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   try {
-#  endif // _LIBCPP_HAS_NO_EXCEPTIONS
     p = ::operator new(size, alignment);
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   } catch (...) {
   }
-#  endif // _LIBCPP_HAS_NO_EXCEPTIONS
   return p;
+#  endif
 }
 
-_LIBCPP_WEAK
-void* operator new[](size_t size, std::align_val_t alignment) _THROW_BAD_ALLOC {
+_LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE _LIBCPP_WEAK void*
+operator new[](size_t size, std::align_val_t alignment) _THROW_BAD_ALLOC {
   return ::operator new(size, alignment);
 }
 
-_LIBCPP_WEAK
-void* operator new[](size_t size, std::align_val_t alignment, const std::nothrow_t&) noexcept {
+_LIBCPP_WEAK void* operator new[](size_t size, std::align_val_t alignment, const std::nothrow_t&) noexcept {
+#  ifdef _LIBCPP_HAS_NO_EXCEPTIONS
+#    if _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION
+  _LIBCPP_ASSERT_SHIM(
+      !std::__is_function_overridden(static_cast<void* (*)(std::size_t, std::align_val_t)>(&operator new[])),
+      "libc++ was configured with exceptions disabled and `operator new[](size_t, align_val_t)` has been overridden, "
+      "but `operator new[](size_t, align_val_t, nothrow_t)` has not been overridden. This is problematic because "
+      "`operator new[](size_t, align_val_t, nothrow_t)` must call `operator new[](size_t, align_val_t)`, which will "
+      "terminate in case it fails to allocate, making it impossible for `operator new[](size_t, align_val_t, "
+      "nothrow_t)` to fulfill its contract (since it should return nullptr upon failure). Please make sure you "
+      "override "
+      "`operator new[](size_t, align_val_t, nothrow_t)` as well.");
+#    endif
+
+  return operator_new_aligned_impl(size, alignment);
+#  else
   void* p = nullptr;
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   try {
-#  endif // _LIBCPP_HAS_NO_EXCEPTIONS
     p = ::operator new[](size, alignment);
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   } catch (...) {
   }
-#  endif // _LIBCPP_HAS_NO_EXCEPTIONS
   return p;
+#  endif
 }
 
-_LIBCPP_WEAK
-void operator delete(void* ptr, std::align_val_t) noexcept { std::__libcpp_aligned_free(ptr); }
+_LIBCPP_WEAK void operator delete(void* ptr, std::align_val_t) noexcept { std::__libcpp_aligned_free(ptr); }
 
-_LIBCPP_WEAK
-void operator delete(void* ptr, std::align_val_t alignment, const std::nothrow_t&) noexcept {
+_LIBCPP_WEAK void operator delete(void* ptr, std::align_val_t alignment, const std::nothrow_t&) noexcept {
   ::operator delete(ptr, alignment);
 }
 
-_LIBCPP_WEAK
-void operator delete(void* ptr, size_t, std::align_val_t alignment) noexcept { ::operator delete(ptr, alignment); }
+_LIBCPP_WEAK void operator delete(void* ptr, size_t, std::align_val_t alignment) noexcept {
+  ::operator delete(ptr, alignment);
+}
 
-_LIBCPP_WEAK
-void operator delete[](void* ptr, std::align_val_t alignment) noexcept { ::operator delete(ptr, alignment); }
+_LIBCPP_WEAK void operator delete[](void* ptr, std::align_val_t alignment) noexcept {
+  ::operator delete(ptr, alignment);
+}
 
-_LIBCPP_WEAK
-void operator delete[](void* ptr, std::align_val_t alignment, const std::nothrow_t&) noexcept {
+_LIBCPP_WEAK void operator delete[](void* ptr, std::align_val_t alignment, const std::nothrow_t&) noexcept {
   ::operator delete[](ptr, alignment);
 }
 
-_LIBCPP_WEAK
-void operator delete[](void* ptr, size_t, std::align_val_t alignment) noexcept { ::operator delete[](ptr, alignment); }
+_LIBCPP_WEAK void operator delete[](void* ptr, size_t, std::align_val_t alignment) noexcept {
+  ::operator delete[](ptr, alignment);
+}
 
 #endif // !_LIBCPP_HAS_NO_LIBRARY_ALIGNED_ALLOCATION
 // ------------------ END COPY ------------------