ubsan: default to trap on unreachable at -O0 and -Og [PR104642]

When not optimizing, we can't do anything useful with unreachability in
terms of code performance, so we might as well improve debugging by turning
__builtin_unreachable into a trap.  I think it also makes sense to do this
when we're explicitly optimizing for the debugging experience.

In the PR richi suggested introducing an -funreachable-traps flag for this.
This functionality is already implemented as -fsanitize=unreachable
-fsanitize-trap=unreachable, and we want to share the implementation, but it
does seem useful to have a separate flag that isn't affected by the various
sanitization controls.  -fsanitize=unreachable takes priority over
-funreachable-traps if both are enabled.

Jakub observed that this would slow down -O0 by default from running the
sanopt pass, so this revision avoids the need for sanopt by rewriting calls
introduced by the compiler immediately, and calls written by the user at
fold time.  Many of the calls introduced by the compiler are also rewritten
immediately to ubsan calls when not trapping, which fixes ubsan-8b.C;
previously the call to f() was optimized away before sanopt.  But this early
rewriting isn't practical for uses of __builtin_unreachable in
devirtualization and such, so sanopt rewriting is still done for
non-trapping sanitize.

	PR c++/104642

gcc/ChangeLog:

	* common.opt: Add -funreachable-traps.
	* doc/invoke.texi (-funreachable-traps): Document it.
	* opts.cc (finish_options): Enable at -O0 or -Og.
	* tree.cc (build_common_builtin_nodes): Add __builtin_trap.
	(builtin_decl_unreachable, build_builtin_unreachable): New.
	* tree.h: Declare them.
	* ubsan.cc (sanitize_unreachable_fn): Factor out.
	(ubsan_instrument_unreachable): Use
	gimple_build_builtin_unreachable.
	* ubsan.h (sanitize_unreachable_fn): Declare.
	* gimple.cc (gimple_build_builtin_unreachable): New.
	* gimple.h: Declare it.
	* builtins.cc (expand_builtin_unreachable): Add assert.
	(fold_builtin_0): Call build_builtin_unreachable.
	* sanopt.cc: Don't run for just SANITIZE_RETURN
	or SANITIZE_UNREACHABLE when trapping.
	* cgraphunit.cc (walk_polymorphic_call_targets): Use new
	unreachable functions.
	* gimple-fold.cc (gimple_fold_call)
	(gimple_get_virt_method_for_vtable)
	* ipa-fnsummary.cc (redirect_to_unreachable)
	* ipa-prop.cc (ipa_make_edge_direct_to_target)
	(ipa_impossible_devirt_target)
	* ipa.cc (walk_polymorphic_call_targets)
	* tree-cfg.cc (pass_warn_function_return::execute)
	(execute_fixup_cfg)
	* tree-ssa-loop-ivcanon.cc (remove_exits_and_undefined_stmts)
	(unloop_loops)
	* tree-ssa-sccvn.cc (eliminate_dom_walker::eliminate_stmt):
	Likewise.

gcc/cp/ChangeLog:

	* constexpr.cc (cxx_eval_builtin_function_call): Handle
	unreachable/trap earlier.
	* cp-gimplify.cc (cp_maybe_instrument_return): Use
	build_builtin_unreachable.

gcc/testsuite/ChangeLog:

	* g++.dg/ubsan/return-8a.C: New test.
	* g++.dg/ubsan/return-8b.C: New test.
	* g++.dg/ubsan/return-8d.C: New test.
	* g++.dg/ubsan/return-8e.C: New test.

1 files changed, 40 insertions, 0 deletions

diff --git a/gcc/tree.cc b/gcc/tree.cc
index 2bfb674..84000dd 100644
--- a/gcc/tree.cc
+++ b/gcc/tree.cc
@@ -71,6 +71,8 @@ along with GCC; see the file COPYING3.  If not see
 #include "gimple-range.h"
 #include "gomp-constants.h"
 #include "dfp.h"
+#include "asan.h"
+#include "ubsan.h"
 
 /* Tree code classes.  */
 
@@ -9649,6 +9651,7 @@ build_common_builtin_nodes (void)
     }
 
   if (!builtin_decl_explicit_p (BUILT_IN_UNREACHABLE)
+      || !builtin_decl_explicit_p (BUILT_IN_TRAP)
       || !builtin_decl_explicit_p (BUILT_IN_ABORT))
     {
       ftype = build_function_type (void_type_node, void_list_node);
@@ -9662,6 +9665,10 @@ build_common_builtin_nodes (void)
 	local_define_builtin ("__builtin_abort", ftype, BUILT_IN_ABORT,
 			      "abort",
 			      ECF_LEAF | ECF_NORETURN | ECF_CONST | ECF_COLD);
+      if (!builtin_decl_explicit_p (BUILT_IN_TRAP))
+	local_define_builtin ("__builtin_trap", ftype, BUILT_IN_TRAP,
+			      "__builtin_trap",
+			      ECF_NORETURN | ECF_NOTHROW | ECF_LEAF | ECF_COLD);
     }
 
   if (!builtin_decl_explicit_p (BUILT_IN_MEMCPY)
@@ -10779,6 +10786,39 @@ build_alloca_call_expr (tree size, unsigned int align, HOST_WIDE_INT max_size)
     }
 }
 
+/* The built-in decl to use to mark code points believed to be unreachable.
+   Typically __builtin_unreachable, but __builtin_trap if
+   -fsanitize=unreachable -fsanitize-trap=unreachable.  If only
+   -fsanitize=unreachable, we rely on sanopt to replace calls with the
+   appropriate ubsan function.  When building a call directly, use
+   {gimple_},build_builtin_unreachable instead.  */
+
+tree
+builtin_decl_unreachable ()
+{
+  enum built_in_function fncode = BUILT_IN_UNREACHABLE;
+
+  if (sanitize_flags_p (SANITIZE_UNREACHABLE)
+      ? (flag_sanitize_trap & SANITIZE_UNREACHABLE)
+      : flag_unreachable_traps)
+    fncode = BUILT_IN_TRAP;
+  /* For non-trapping sanitize, we will rewrite __builtin_unreachable () later,
+     in the sanopt pass.  */
+
+  return builtin_decl_explicit (fncode);
+}
+
+/* Build a call to __builtin_unreachable, possibly rewritten by
+   -fsanitize=unreachable.  Use this rather than the above when practical.  */
+
+tree
+build_builtin_unreachable (location_t loc)
+{
+  tree data = NULL_TREE;
+  tree fn = sanitize_unreachable_fn (&data, loc);
+  return build_call_expr_loc (loc, fn, data != NULL_TREE, data);
+}
+
 /* Create a new constant string literal of type ELTYPE[SIZE] (or LEN
    if SIZE == -1) and return a tree node representing char* pointer to
    it as an ADDR_EXPR (ARRAY_REF (ELTYPE, ...)).  When STR is nonnull