Add type-generic clz/ctz/clrsb/ffs/parity/popcount builtins [PR111309]

The following patch adds 6 new type-generic builtins,
__builtin_clzg
__builtin_ctzg
__builtin_clrsbg
__builtin_ffsg
__builtin_parityg
__builtin_popcountg
The g at the end stands for generic because the unsuffixed variant
of the builtins already have unsigned int or int arguments.

The main reason to add these is to support arbitrary unsigned (for
clrsb/ffs signed) bit-precise integer types and also __int128 which
wasn't supported by the existing builtins, so that e.g. <stdbit.h>
type-generic functions could then support not just bit-precise unsigned
integer type whose width matches a standard or extended integer type,
but others too.

None of these new builtins promote their first argument, so the argument
can be e.g. unsigned char or unsigned short or unsigned __int20 etc.
The first 2 support either 1 or 2 arguments, if only 1 argument is supplied,
the behavior is undefined for argument 0 like for other __builtin_c[lt]z*
builtins, if 2 arguments are supplied, the second argument should be int
that will be returned if the argument is 0.  All other builtins have
just one argument.  For __builtin_clrsbg and __builtin_ffsg the argument
shall be any signed standard/extended or bit-precise integer, for the others
any unsigned standard/extended or bit-precise integer (bool not allowed).

One possibility would be to also allow signed integer types for
the clz/ctz/parity/popcount ones (and just cast the argument to
unsigned_type_for during folding) and similarly unsigned integer types
for the clrsb/ffs ones, dunno what is better; for stdbit.h the current
version is sufficient and diagnoses use of the inappropriate sign,
though on the other side I wonder if users won't be confused by
__builtin_clzg (1) being an error and having to write __builtin_clzg (1U).

The new builtins are lowered to corresponding builtins with other suffixes
or internal calls (plus casts and adjustments where needed) during FE
folding or during gimplification at latest, the non-suffixed builtins
handling precisions up to precision of int, l up to precision of long,
ll up to precision of long long, up to __int128 precision lowered to
double-word expansion early and the rest (which must be _BitInt) lowered
to internal fn calls - those are then lowered during bitint lowering pass.

The patch also changes representation of IFN_CLZ and IFN_CTZ calls,
previously they were in the IL only if they are directly supported optab
and depending on C[LT]Z_DEFINED_VALUE_AT_ZERO (...) == 2 they had or didn't
have defined behavior at 0, now they are in the IL either if directly
supported optab, or for the large/huge BITINT_TYPEs and they have either
1 or 2 arguments.  If one, the behavior is undefined at zero, if 2, the
second argument is an int constant that should be returned for 0.
As there is no extra support during expansion, for directly supported optab
the second argument if present should still match the
C[LT]Z_DEFINED_VALUE_AT_ZERO (...) == 2 value, but for BITINT_TYPE arguments
it can be arbitrary int INTEGER_CST.

The indended uses in stdbit.h are e.g.
 #ifdef __has_builtin
 #if __has_builtin(__builtin_clzg) && __has_builtin(__builtin_ctzg) && __has_builtin(__builtin_popcountg)
 #define stdc_leading_zeros(value) \
 ((unsigned int) __builtin_clzg (value, __builtin_popcountg ((__typeof (value)) ~(__typeof (value)) 0)))
 #define stdc_leading_ones(value) \
 ((unsigned int) __builtin_clzg ((__typeof (value)) ~(value), __builtin_popcountg ((__typeof (value)) ~(__typeof (value)) 0)))
 #define stdc_first_trailing_one(value) \
 ((unsigned int) (__builtin_ctzg (value, -1) + 1))
 #define stdc_trailing_zeros(value) \
 ((unsigned int) __builtin_ctzg (value, __builtin_popcountg ((__typeof (value)) ~(__typeof (value)) 0)))
 #endif
 #endif
where __builtin_popcountg ((__typeof (x)) -1) computes the bit precision
of x's type (kind of _Bitwidthof (x) alternative).

They also allow casting of arbitrary unsigned _BitInt other than
unsigned _BitInt(1) to corresponding signed _BitInt by using
signed _BitInt(__builtin_popcountg ((__typeof (a)) -1))
and of arbitrary signed _BitInt to corresponding unsigned _BitInt
using unsigned _BitInt(__builtin_clrsbg ((__typeof (a)) -1) + 1).

2023-11-14  Jakub Jelinek  <jakub@redhat.com>

	PR c/111309
gcc/
	* builtins.def (BUILT_IN_CLZG, BUILT_IN_CTZG, BUILT_IN_CLRSBG,
	BUILT_IN_FFSG, BUILT_IN_PARITYG, BUILT_IN_POPCOUNTG): New
	builtins.
	* builtins.cc (fold_builtin_bit_query): New function.
	(fold_builtin_1): Use it for
	BUILT_IN_{CLZ,CTZ,CLRSB,FFS,PARITY,POPCOUNT}G.
	(fold_builtin_2): Use it for BUILT_IN_{CLZ,CTZ}G.
	* fold-const-call.cc: Fix comment typo on tm.h inclusion.
	(fold_const_call_ss): Handle
	CFN_BUILT_IN_{CLZ,CTZ,CLRSB,FFS,PARITY,POPCOUNT}G.
	(fold_const_call_sss): New function.
	(fold_const_call_1): Call it for 2 argument functions returning
	scalar when passed 2 INTEGER_CSTs.
	* genmatch.cc (cmp_operand): For function calls also compare
	number of arguments.
	(fns_cmp): New function.
	(dt_node::gen_kids): Sort fns and generic_fns.
	(dt_node::gen_kids_1): Handle fns with the same id but different
	number of arguments.
	* match.pd (CLZ simplifications): Drop checks for defined behavior
	at zero.  Add variant of simplifications for IFN_CLZ with 2 arguments.
	(CTZ simplifications): Drop checks for defined behavior at zero,
	don't optimize precisions above MAX_FIXED_MODE_SIZE.  Add variant of
	simplifications for IFN_CTZ with 2 arguments.
	(a != 0 ? CLZ(a) : CST -> .CLZ(a)): Use TREE_TYPE (@3) instead of
	type, add BITINT_TYPE handling, create 2 argument IFN_CLZ rather than
	one argument.  Add variant for matching CLZ with 2 arguments.
	(a != 0 ? CTZ(a) : CST -> .CTZ(a)): Similarly.
	* gimple-lower-bitint.cc (bitint_large_huge::lower_bit_query): New
	method.
	(bitint_large_huge::lower_call): Use it for IFN_{CLZ,CTZ,CLRSB,FFS}
	and IFN_{PARITY,POPCOUNT} calls.
	* gimple-range-op.cc (cfn_clz::fold_range): Don't check
	CLZ_DEFINED_VALUE_AT_ZERO for m_gimple_call_internal_p, instead
	assume defined value at zero if the call has 2 arguments and use
	second argument value for that case.
	(cfn_ctz::fold_range): Similarly.
	(gimple_range_op_handler::maybe_builtin_call): Use op_cfn_clz_internal
	or op_cfn_ctz_internal only if internal fn call has 2 arguments and
	set m_op2 in that case.
	* tree-vect-patterns.cc (vect_recog_ctz_ffs_pattern,
	vect_recog_popcount_clz_ctz_ffs_pattern): For value defined at zero
	use second argument of calls if present, otherwise assume UB at zero,
	create 2 argument .CLZ/.CTZ calls if needed.
	* tree-vect-stmts.cc (vectorizable_call): Handle 2 argument .CLZ/.CTZ
	calls.
	* tree-ssa-loop-niter.cc (build_cltz_expr): Create 2 argument
	.CLZ/.CTZ calls if needed.
	* tree-ssa-forwprop.cc (simplify_count_trailing_zeroes): Create 2
	argument .CTZ calls if needed.
	* tree-ssa-phiopt.cc (cond_removal_in_builtin_zero_pattern): Handle
	2 argument .CLZ/.CTZ calls, handle BITINT_TYPE, create 2 argument
	.CLZ/.CTZ calls.
	* doc/extend.texi (__builtin_clzg, __builtin_ctzg, __builtin_clrsbg,
	__builtin_ffsg, __builtin_parityg, __builtin_popcountg): Document.
gcc/c-family/
	* c-common.cc (check_builtin_function_arguments): Handle
	BUILT_IN_{CLZ,CTZ,CLRSB,FFS,PARITY,POPCOUNT}G.
	* c-gimplify.cc (c_gimplify_expr): If __builtin_c[lt]zg second
	argument hasn't been folded into constant yet, transform it to one
	argument call inside of a COND_EXPR which for first argument 0
	returns the second argument.
gcc/c/
	* c-typeck.cc (convert_arguments): Don't promote first argument
	of BUILT_IN_{CLZ,CTZ,CLRSB,FFS,PARITY,POPCOUNT}G.
gcc/cp/
	* call.cc (magic_varargs_p): Return 4 for
	BUILT_IN_{CLZ,CTZ,CLRSB,FFS,PARITY,POPCOUNT}G.
	(build_over_call): Don't promote first argument of
	BUILT_IN_{CLZ,CTZ,CLRSB,FFS,PARITY,POPCOUNT}G.
	* cp-gimplify.cc (cp_gimplify_expr): For BUILT_IN_C{L,T}ZG use
	c_gimplify_expr.
gcc/testsuite/
	* c-c++-common/pr111309-1.c: New test.
	* c-c++-common/pr111309-2.c: New test.
	* gcc.dg/torture/bitint-43.c: New test.
	* gcc.dg/torture/bitint-44.c: New test.

22 files changed, 2576 insertions, 143 deletions

diff --git a/gcc/builtins.cc b/gcc/builtins.cc
index cb90bd0..5ece0d2 100644
--- a/gcc/builtins.cc
+++ b/gcc/builtins.cc
@@ -9573,6 +9573,271 @@ fold_builtin_arith_overflow (location_t loc, enum built_in_function fcode,
   return build2_loc (loc, COMPOUND_EXPR, boolean_type_node, store, ovfres);
 }
 
+/* Fold __builtin_{clz,ctz,clrsb,ffs,parity,popcount}g into corresponding
+   internal function.  */
+
+static tree
+fold_builtin_bit_query (location_t loc, enum built_in_function fcode,
+			tree arg0, tree arg1)
+{
+  enum internal_fn ifn;
+  enum built_in_function fcodei, fcodel, fcodell;
+  tree arg0_type = TREE_TYPE (arg0);
+  tree cast_type = NULL_TREE;
+  int addend = 0;
+
+  switch (fcode)
+    {
+    case BUILT_IN_CLZG:
+      if (arg1 && TREE_CODE (arg1) != INTEGER_CST)
+	return NULL_TREE;
+      ifn = IFN_CLZ;
+      fcodei = BUILT_IN_CLZ;
+      fcodel = BUILT_IN_CLZL;
+      fcodell = BUILT_IN_CLZLL;
+      break;
+    case BUILT_IN_CTZG:
+      if (arg1 && TREE_CODE (arg1) != INTEGER_CST)
+	return NULL_TREE;
+      ifn = IFN_CTZ;
+      fcodei = BUILT_IN_CTZ;
+      fcodel = BUILT_IN_CTZL;
+      fcodell = BUILT_IN_CTZLL;
+      break;
+    case BUILT_IN_CLRSBG:
+      ifn = IFN_CLRSB;
+      fcodei = BUILT_IN_CLRSB;
+      fcodel = BUILT_IN_CLRSBL;
+      fcodell = BUILT_IN_CLRSBLL;
+      break;
+    case BUILT_IN_FFSG:
+      ifn = IFN_FFS;
+      fcodei = BUILT_IN_FFS;
+      fcodel = BUILT_IN_FFSL;
+      fcodell = BUILT_IN_FFSLL;
+      break;
+    case BUILT_IN_PARITYG:
+      ifn = IFN_PARITY;
+      fcodei = BUILT_IN_PARITY;
+      fcodel = BUILT_IN_PARITYL;
+      fcodell = BUILT_IN_PARITYLL;
+      break;
+    case BUILT_IN_POPCOUNTG:
+      ifn = IFN_POPCOUNT;
+      fcodei = BUILT_IN_POPCOUNT;
+      fcodel = BUILT_IN_POPCOUNTL;
+      fcodell = BUILT_IN_POPCOUNTLL;
+      break;
+    default:
+      gcc_unreachable ();
+    }
+
+  if (TYPE_PRECISION (arg0_type)
+      <= TYPE_PRECISION (long_long_unsigned_type_node))
+    {
+      if (TYPE_PRECISION (arg0_type) <= TYPE_PRECISION (unsigned_type_node))
+
+	cast_type = (TYPE_UNSIGNED (arg0_type)
+		     ? unsigned_type_node : integer_type_node);
+      else if (TYPE_PRECISION (arg0_type)
+	       <= TYPE_PRECISION (long_unsigned_type_node))
+	{
+	  cast_type = (TYPE_UNSIGNED (arg0_type)
+		       ? long_unsigned_type_node : long_integer_type_node);
+	  fcodei = fcodel;
+	}
+      else
+	{
+	  cast_type = (TYPE_UNSIGNED (arg0_type)
+		       ? long_long_unsigned_type_node
+		       : long_long_integer_type_node);
+	  fcodei = fcodell;
+	}
+    }
+  else if (TYPE_PRECISION (arg0_type) <= MAX_FIXED_MODE_SIZE)
+    {
+      cast_type
+	= build_nonstandard_integer_type (MAX_FIXED_MODE_SIZE,
+					  TYPE_UNSIGNED (arg0_type));
+      gcc_assert (TYPE_PRECISION (cast_type)
+		  == 2 * TYPE_PRECISION (long_long_unsigned_type_node));
+      fcodei = END_BUILTINS;
+    }
+  else
+    fcodei = END_BUILTINS;
+  if (cast_type)
+    {
+      switch (fcode)
+	{
+	case BUILT_IN_CLZG:
+	case BUILT_IN_CLRSBG:
+	  addend = TYPE_PRECISION (arg0_type) - TYPE_PRECISION (cast_type);
+	  break;
+	default:
+	  break;
+	}
+      arg0 = fold_convert (cast_type, arg0);
+      arg0_type = cast_type;
+    }
+
+  if (arg1)
+    arg1 = fold_convert (integer_type_node, arg1);
+
+  tree arg2 = arg1;
+  if (fcode == BUILT_IN_CLZG && addend)
+    {
+      if (arg1)
+	arg0 = save_expr (arg0);
+      arg2 = NULL_TREE;
+    }
+  tree call = NULL_TREE, tem;
+  if (TYPE_PRECISION (arg0_type) == MAX_FIXED_MODE_SIZE
+      && (TYPE_PRECISION (arg0_type)
+	  == 2 * TYPE_PRECISION (long_long_unsigned_type_node)))
+    {
+      /* __int128 expansions using up to 2 long long builtins.  */
+      arg0 = save_expr (arg0);
+      tree type = (TYPE_UNSIGNED (arg0_type)
+		   ? long_long_unsigned_type_node
+		   : long_long_integer_type_node);
+      tree hi = fold_build2 (RSHIFT_EXPR, arg0_type, arg0,
+			     build_int_cst (integer_type_node,
+					    MAX_FIXED_MODE_SIZE / 2));
+      hi = fold_convert (type, hi);
+      tree lo = fold_convert (type, arg0);
+      switch (fcode)
+	{
+	case BUILT_IN_CLZG:
+	  call = fold_builtin_bit_query (loc, fcode, lo, NULL_TREE);
+	  call = fold_build2 (PLUS_EXPR, integer_type_node, call,
+			      build_int_cst (integer_type_node,
+					     MAX_FIXED_MODE_SIZE / 2));
+	  if (arg2)
+	    call = fold_build3 (COND_EXPR, integer_type_node,
+				fold_build2 (NE_EXPR, boolean_type_node,
+					     lo, build_zero_cst (type)),
+				call, arg2);
+	  call = fold_build3 (COND_EXPR, integer_type_node,
+			      fold_build2 (NE_EXPR, boolean_type_node,
+					   hi, build_zero_cst (type)),
+			      fold_builtin_bit_query (loc, fcode, hi,
+						      NULL_TREE),
+			      call);
+	  break;
+	case BUILT_IN_CTZG:
+	  call = fold_builtin_bit_query (loc, fcode, hi, NULL_TREE);
+	  call = fold_build2 (PLUS_EXPR, integer_type_node, call,
+			      build_int_cst (integer_type_node,
+					     MAX_FIXED_MODE_SIZE / 2));
+	  if (arg2)
+	    call = fold_build3 (COND_EXPR, integer_type_node,
+				fold_build2 (NE_EXPR, boolean_type_node,
+					     hi, build_zero_cst (type)),
+				call, arg2);
+	  call = fold_build3 (COND_EXPR, integer_type_node,
+			      fold_build2 (NE_EXPR, boolean_type_node,
+					   lo, build_zero_cst (type)),
+			      fold_builtin_bit_query (loc, fcode, lo,
+						      NULL_TREE),
+			      call);
+	  break;
+	case BUILT_IN_CLRSBG:
+	  tem = fold_builtin_bit_query (loc, fcode, lo, NULL_TREE);
+	  tem = fold_build2 (PLUS_EXPR, integer_type_node, tem,
+			     build_int_cst (integer_type_node,
+					    MAX_FIXED_MODE_SIZE / 2));
+	  tem = fold_build3 (COND_EXPR, integer_type_node,
+			     fold_build2 (LT_EXPR, boolean_type_node,
+					  fold_build2 (BIT_XOR_EXPR, type,
+						       lo, hi),
+					  build_zero_cst (type)),
+			     build_int_cst (integer_type_node,
+					    MAX_FIXED_MODE_SIZE / 2 - 1),
+			     tem);
+	  call = fold_builtin_bit_query (loc, fcode, hi, NULL_TREE);
+	  call = save_expr (call);
+	  call = fold_build3 (COND_EXPR, integer_type_node,
+			      fold_build2 (NE_EXPR, boolean_type_node,
+					   call,
+					   build_int_cst (integer_type_node,
+							  MAX_FIXED_MODE_SIZE
+							  / 2 - 1)),
+			      call, tem);
+	  break;
+	case BUILT_IN_FFSG:
+	  call = fold_builtin_bit_query (loc, fcode, hi, NULL_TREE);
+	  call = fold_build2 (PLUS_EXPR, integer_type_node, call,
+			      build_int_cst (integer_type_node,
+					     MAX_FIXED_MODE_SIZE / 2));
+	  call = fold_build3 (COND_EXPR, integer_type_node,
+			      fold_build2 (NE_EXPR, boolean_type_node,
+					   hi, build_zero_cst (type)),
+			      call, integer_zero_node);
+	  call = fold_build3 (COND_EXPR, integer_type_node,
+			      fold_build2 (NE_EXPR, boolean_type_node,
+					   lo, build_zero_cst (type)),
+			      fold_builtin_bit_query (loc, fcode, lo,
+						      NULL_TREE),
+			      call);
+	  break;
+	case BUILT_IN_PARITYG:
+	  call = fold_builtin_bit_query (loc, fcode,
+					 fold_build2 (BIT_XOR_EXPR, type,
+						      lo, hi), NULL_TREE);
+	  break;
+	case BUILT_IN_POPCOUNTG:
+	  call = fold_build2 (PLUS_EXPR, integer_type_node,
+			      fold_builtin_bit_query (loc, fcode, hi,
+						      NULL_TREE),
+			      fold_builtin_bit_query (loc, fcode, lo,
+						      NULL_TREE));
+	  break;
+	default:
+	  gcc_unreachable ();
+	}
+    }
+  else
+    {
+      /* Only keep second argument to IFN_CLZ/IFN_CTZ if it is the
+	 value defined at zero during GIMPLE, or for large/huge _BitInt
+	 (which are then lowered during bitint lowering).  */
+      if (arg2 && TREE_CODE (TREE_TYPE (arg0)) != BITINT_TYPE)
+	{
+	  int val;
+	  if (fcode == BUILT_IN_CLZG)
+	    {
+	      if (CLZ_DEFINED_VALUE_AT_ZERO (SCALAR_TYPE_MODE (arg0_type),
+					     val) != 2
+		  || wi::to_widest (arg2) != val)
+		arg2 = NULL_TREE;
+	    }
+	  else if (CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_TYPE_MODE (arg0_type),
+					      val) != 2
+		   || wi::to_widest (arg2) != val)
+	    arg2 = NULL_TREE;
+	  if (!direct_internal_fn_supported_p (ifn, arg0_type,
+					       OPTIMIZE_FOR_BOTH))
+	    arg2 = NULL_TREE;
+	}
+      if (fcodei == END_BUILTINS || arg2)
+	call = build_call_expr_internal_loc (loc, ifn, integer_type_node,
+					     arg2 ? 2 : 1, arg0, arg2);
+      else
+	call = build_call_expr_loc (loc, builtin_decl_explicit (fcodei), 1,
+				    arg0);
+    }
+  if (addend)
+    call = fold_build2 (PLUS_EXPR, integer_type_node, call,
+			build_int_cst (integer_type_node, addend));
+  if (arg1 && arg2 == NULL_TREE)
+    call = fold_build3 (COND_EXPR, integer_type_node,
+			fold_build2 (NE_EXPR, boolean_type_node,
+				     arg0, build_zero_cst (arg0_type)),
+			call, arg1);
+
+  return call;
+}
+
 /* Fold __builtin_{add,sub}c{,l,ll} into pair of internal functions
    that return both result of arithmetics and overflowed boolean
    flag in a complex integer result.  */
@@ -9824,6 +10089,14 @@ fold_builtin_1 (location_t loc, tree expr, tree fndecl, tree arg0)
 	return build_empty_stmt (loc);
       break;
 
+    case BUILT_IN_CLZG:
+    case BUILT_IN_CTZG:
+    case BUILT_IN_CLRSBG:
+    case BUILT_IN_FFSG:
+    case BUILT_IN_PARITYG:
+    case BUILT_IN_POPCOUNTG:
+      return fold_builtin_bit_query (loc, fcode, arg0, NULL_TREE);
+
     default:
       break;
     }
@@ -9913,6 +10186,10 @@ fold_builtin_2 (location_t loc, tree expr, tree fndecl, tree arg0, tree arg1)
     case BUILT_IN_ATOMIC_IS_LOCK_FREE:
       return fold_builtin_atomic_is_lock_free (arg0, arg1);
 
+    case BUILT_IN_CLZG:
+    case BUILT_IN_CTZG:
+      return fold_builtin_bit_query (loc, fcode, arg0, arg1);
+
     default:
       break;
     }
diff --git a/gcc/builtins.def b/gcc/builtins.def
index a6fdb22..33e6cad 100644
--- a/gcc/builtins.def
+++ b/gcc/builtins.def
@@ -962,15 +962,18 @@ DEF_GCC_BUILTIN        (BUILT_IN_CLZ, "clz", BT_FN_INT_UINT, ATTR_CONST_NOTHROW_
 DEF_GCC_BUILTIN        (BUILT_IN_CLZIMAX, "clzimax", BT_FN_INT_UINTMAX, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_CLZL, "clzl", BT_FN_INT_ULONG, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_CLZLL, "clzll", BT_FN_INT_ULONGLONG, ATTR_CONST_NOTHROW_LEAF_LIST)
+DEF_GCC_BUILTIN        (BUILT_IN_CLZG, "clzg", BT_FN_INT_VAR, ATTR_CONST_NOTHROW_TYPEGENERIC_LEAF)
 DEF_GCC_BUILTIN        (BUILT_IN_CONSTANT_P, "constant_p", BT_FN_INT_VAR, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_CTZ, "ctz", BT_FN_INT_UINT, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_CTZIMAX, "ctzimax", BT_FN_INT_UINTMAX, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_CTZL, "ctzl", BT_FN_INT_ULONG, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_CTZLL, "ctzll", BT_FN_INT_ULONGLONG, ATTR_CONST_NOTHROW_LEAF_LIST)
+DEF_GCC_BUILTIN        (BUILT_IN_CTZG, "ctzg", BT_FN_INT_VAR, ATTR_CONST_NOTHROW_TYPEGENERIC_LEAF)
 DEF_GCC_BUILTIN        (BUILT_IN_CLRSB, "clrsb", BT_FN_INT_INT, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_CLRSBIMAX, "clrsbimax", BT_FN_INT_INTMAX, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_CLRSBL, "clrsbl", BT_FN_INT_LONG, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_CLRSBLL, "clrsbll", BT_FN_INT_LONGLONG, ATTR_CONST_NOTHROW_LEAF_LIST)
+DEF_GCC_BUILTIN        (BUILT_IN_CLRSBG, "clrsbg", BT_FN_INT_VAR, ATTR_CONST_NOTHROW_TYPEGENERIC_LEAF)
 DEF_EXT_LIB_BUILTIN    (BUILT_IN_DCGETTEXT, "dcgettext", BT_FN_STRING_CONST_STRING_CONST_STRING_INT, ATTR_FORMAT_ARG_2)
 DEF_EXT_LIB_BUILTIN    (BUILT_IN_DGETTEXT, "dgettext", BT_FN_STRING_CONST_STRING_CONST_STRING, ATTR_FORMAT_ARG_2)
 DEF_GCC_BUILTIN        (BUILT_IN_DWARF_CFA, "dwarf_cfa", BT_FN_PTR, ATTR_NULL)
@@ -993,6 +996,7 @@ DEF_EXT_LIB_BUILTIN    (BUILT_IN_FFS, "ffs", BT_FN_INT_INT, ATTR_CONST_NOTHROW_L
 DEF_EXT_LIB_BUILTIN    (BUILT_IN_FFSIMAX, "ffsimax", BT_FN_INT_INTMAX, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_EXT_LIB_BUILTIN    (BUILT_IN_FFSL, "ffsl", BT_FN_INT_LONG, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_EXT_LIB_BUILTIN    (BUILT_IN_FFSLL, "ffsll", BT_FN_INT_LONGLONG, ATTR_CONST_NOTHROW_LEAF_LIST)
+DEF_GCC_BUILTIN        (BUILT_IN_FFSG, "ffsg", BT_FN_INT_VAR, ATTR_CONST_NOTHROW_TYPEGENERIC_LEAF)
 DEF_EXT_LIB_BUILTIN        (BUILT_IN_FORK, "fork", BT_FN_PID, ATTR_NOTHROW_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_FRAME_ADDRESS, "frame_address", BT_FN_PTR_UINT, ATTR_NULL)
 /* [trans-mem]: Adjust BUILT_IN_TM_FREE if BUILT_IN_FREE is changed.  */
@@ -1041,10 +1045,12 @@ DEF_GCC_BUILTIN        (BUILT_IN_PARITY, "parity", BT_FN_INT_UINT, ATTR_CONST_NO
 DEF_GCC_BUILTIN        (BUILT_IN_PARITYIMAX, "parityimax", BT_FN_INT_UINTMAX, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_PARITYL, "parityl", BT_FN_INT_ULONG, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_PARITYLL, "parityll", BT_FN_INT_ULONGLONG, ATTR_CONST_NOTHROW_LEAF_LIST)
+DEF_GCC_BUILTIN        (BUILT_IN_PARITYG, "parityg", BT_FN_INT_VAR, ATTR_CONST_NOTHROW_TYPEGENERIC_LEAF)
 DEF_GCC_BUILTIN        (BUILT_IN_POPCOUNT, "popcount", BT_FN_INT_UINT, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_POPCOUNTIMAX, "popcountimax", BT_FN_INT_UINTMAX, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_POPCOUNTL, "popcountl", BT_FN_INT_ULONG, ATTR_CONST_NOTHROW_LEAF_LIST)
 DEF_GCC_BUILTIN        (BUILT_IN_POPCOUNTLL, "popcountll", BT_FN_INT_ULONGLONG, ATTR_CONST_NOTHROW_LEAF_LIST)
+DEF_GCC_BUILTIN        (BUILT_IN_POPCOUNTG, "popcountg", BT_FN_INT_VAR, ATTR_CONST_NOTHROW_TYPEGENERIC_LEAF)
 DEF_EXT_LIB_BUILTIN    (BUILT_IN_POSIX_MEMALIGN, "posix_memalign", BT_FN_INT_PTRPTR_SIZE_SIZE, ATTR_NOTHROW_NONNULL_LEAF)
 DEF_GCC_BUILTIN        (BUILT_IN_PREFETCH, "prefetch", BT_FN_VOID_CONST_PTR_VAR, ATTR_NOVOPS_LEAF_LIST)
 DEF_LIB_BUILTIN        (BUILT_IN_REALLOC, "realloc", BT_FN_PTR_PTR_SIZE, ATTR_ALLOC_WARN_UNUSED_RESULT_SIZE_2_NOTHROW_LEAF_LIST)
diff --git a/gcc/c-family/c-common.cc b/gcc/c-family/c-common.cc
index 77faf17..a619429 100644
--- a/gcc/c-family/c-common.cc
+++ b/gcc/c-family/c-common.cc
@@ -6475,14 +6475,14 @@ check_builtin_function_arguments (location_t loc, vec<location_t> arg_loc,
 	      }
 	  if (TREE_CODE (TREE_TYPE (args[2])) == ENUMERAL_TYPE)
 	    {
-	      error_at (ARG_LOCATION (2), "argument 3 in call to function "
-			"%qE has enumerated type", fndecl);
+	      error_at (ARG_LOCATION (2), "argument %u in call to function "
+			"%qE has enumerated type", 3, fndecl);
 	      return false;
 	    }
 	  else if (TREE_CODE (TREE_TYPE (args[2])) == BOOLEAN_TYPE)
 	    {
-	      error_at (ARG_LOCATION (2), "argument 3 in call to function "
-			"%qE has boolean type", fndecl);
+	      error_at (ARG_LOCATION (2), "argument %u in call to function "
+			"%qE has boolean type", 3, fndecl);
 	      return false;
 	    }
 	  return true;
@@ -6522,6 +6522,72 @@ check_builtin_function_arguments (location_t loc, vec<location_t> arg_loc,
 	}
       return false;
 
+    case BUILT_IN_CLZG:
+    case BUILT_IN_CTZG:
+    case BUILT_IN_CLRSBG:
+    case BUILT_IN_FFSG:
+    case BUILT_IN_PARITYG:
+    case BUILT_IN_POPCOUNTG:
+      if (nargs == 2
+	  && (DECL_FUNCTION_CODE (fndecl) == BUILT_IN_CLZG
+	      || DECL_FUNCTION_CODE (fndecl) == BUILT_IN_CTZG))
+	{
+	  if (!INTEGRAL_TYPE_P (TREE_TYPE (args[1])))
+	    {
+	      error_at (ARG_LOCATION (1), "argument %u in call to function "
+			"%qE does not have integral type", 2, fndecl);
+	      return false;
+	    }
+	  if ((TYPE_PRECISION (TREE_TYPE (args[1]))
+	       > TYPE_PRECISION (integer_type_node))
+	      || (TYPE_PRECISION (TREE_TYPE (args[1]))
+		  == TYPE_PRECISION (integer_type_node)
+		  && TYPE_UNSIGNED (TREE_TYPE (args[1]))))
+	    {
+	      error_at (ARG_LOCATION (1), "argument %u in call to function "
+			"%qE does not have %<int%> type", 2, fndecl);
+	      return false;
+	    }
+	}
+      else if (!builtin_function_validate_nargs (loc, fndecl, nargs, 1))
+	return false;
+
+      if (!INTEGRAL_TYPE_P (TREE_TYPE (args[0])))
+	{
+	  error_at (ARG_LOCATION (0), "argument %u in call to function "
+		    "%qE does not have integral type", 1, fndecl);
+	  return false;
+	}
+      if (TREE_CODE (TREE_TYPE (args[0])) == ENUMERAL_TYPE)
+	{
+	  error_at (ARG_LOCATION (0), "argument %u in call to function "
+		    "%qE has enumerated type", 1, fndecl);
+	  return false;
+	}
+      if (TREE_CODE (TREE_TYPE (args[0])) == BOOLEAN_TYPE)
+	{
+	  error_at (ARG_LOCATION (0), "argument %u in call to function "
+		    "%qE has boolean type", 1, fndecl);
+	  return false;
+	}
+      if (DECL_FUNCTION_CODE (fndecl) == BUILT_IN_FFSG
+	  || DECL_FUNCTION_CODE (fndecl) == BUILT_IN_CLRSBG)
+	{
+	  if (TYPE_UNSIGNED (TREE_TYPE (args[0])))
+	    {
+	      error_at (ARG_LOCATION (0), "argument 1 in call to function "
+			"%qE has unsigned type", fndecl);
+	      return false;
+	    }
+	}
+      else if (!TYPE_UNSIGNED (TREE_TYPE (args[0])))
+	{
+	  error_at (ARG_LOCATION (0), "argument 1 in call to function "
+		    "%qE has signed type", fndecl);
+	  return false;
+	}
+      return true;
+
     default:
       return true;
     }
diff --git a/gcc/c-family/c-gimplify.cc b/gcc/c-family/c-gimplify.cc
index 17b0610..2f44e12 100644
--- a/gcc/c-family/c-gimplify.cc
+++ b/gcc/c-family/c-gimplify.cc
@@ -818,6 +818,28 @@ c_gimplify_expr (tree *expr_p, gimple_seq *pre_p ATTRIBUTE_UNUSED,
 	break;
       }
 
+    case CALL_EXPR:
+      {
+	tree fndecl = get_callee_fndecl (*expr_p);
+	if (fndecl
+	    && fndecl_built_in_p (fndecl, BUILT_IN_CLZG, BUILT_IN_CTZG)
+	    && call_expr_nargs (*expr_p) == 2
+	    && TREE_CODE (CALL_EXPR_ARG (*expr_p, 1)) != INTEGER_CST)
+	  {
+	    tree a = save_expr (CALL_EXPR_ARG (*expr_p, 0));
+	    tree c = build_call_expr_loc (EXPR_LOCATION (*expr_p),
+					  fndecl, 1, a);
+	    *expr_p = build3_loc (EXPR_LOCATION (*expr_p), COND_EXPR,
+				  integer_type_node,
+				  build2_loc (EXPR_LOCATION (*expr_p),
+					      NE_EXPR, boolean_type_node, a,
+					      build_zero_cst (TREE_TYPE (a))),
+				  c, CALL_EXPR_ARG (*expr_p, 1));
+	    return GS_OK;
+	  }
+	break;
+      }
+
     default:;
     }
 
diff --git a/gcc/c/c-typeck.cc b/gcc/c/c-typeck.cc
index 366ca88..1dbb447 100644
--- a/gcc/c/c-typeck.cc
+++ b/gcc/c/c-typeck.cc
@@ -3416,6 +3416,7 @@ convert_arguments (location_t loc, vec<location_t> arg_loc, tree typelist,
     && lookup_attribute ("type generic", TYPE_ATTRIBUTES (TREE_TYPE (fundecl)));
   bool type_generic_remove_excess_precision = false;
   bool type_generic_overflow_p = false;
+  bool type_generic_bit_query = false;
   tree selector;
 
   /* Change pointer to function to the function itself for
@@ -3471,6 +3472,17 @@ convert_arguments (location_t loc, vec<location_t> arg_loc, tree typelist,
 	    type_generic_overflow_p = true;
 	    break;
 
+	  case BUILT_IN_CLZG:
+	  case BUILT_IN_CTZG:
+	  case BUILT_IN_CLRSBG:
+	  case BUILT_IN_FFSG:
+	  case BUILT_IN_PARITYG:
+	  case BUILT_IN_POPCOUNTG:
+	    /* The first argument of these type-generic builtins
+	       should not be promoted.  */
+	    type_generic_bit_query = true;
+	    break;
+
 	  default:
 	    break;
 	  }
@@ -3606,11 +3618,13 @@ convert_arguments (location_t loc, vec<location_t> arg_loc, tree typelist,
 	    }
 	}
       else if ((excess_precision && !type_generic)
-	       || (type_generic_overflow_p && parmnum == 2))
+	       || (type_generic_overflow_p && parmnum == 2)
+	       || (type_generic_bit_query && parmnum == 0))
 	/* A "double" argument with excess precision being passed
 	   without a prototype or in variable arguments.
 	   The last argument of __builtin_*_overflow_p should not be
-	   promoted.  */
+	   promoted, similarly the first argument of
+	   __builtin_{clz,ctz,clrsb,ffs,parity,popcount}g.  */
 	parmval = convert (valtype, val);
       else if ((invalid_func_diag =
 		targetm.calls.invalid_arg_for_unprototyped_fn (typelist, fundecl, val)))
diff --git a/gcc/cp/call.cc b/gcc/cp/call.cc
index 4516677..709fd74 100644
--- a/gcc/cp/call.cc
+++ b/gcc/cp/call.cc
@@ -9290,7 +9290,9 @@ convert_for_arg_passing (tree type, tree val, tsubst_flags_t complain)
    This is true for some builtins which don't act like normal functions.
    Return 2 if just decay_conversion and removal of excess precision should
    be done, 1 if just decay_conversion.  Return 3 for special treatment of
-   the 3rd argument for __builtin_*_overflow_p.  */
+   the 3rd argument for __builtin_*_overflow_p.  Return 4 for special
+   treatment of the 1st argument for
+   __builtin_{clz,ctz,clrsb,ffs,parity,popcount}g.  */
 
 int
 magic_varargs_p (tree fn)
@@ -9317,6 +9319,14 @@ magic_varargs_p (tree fn)
       case BUILT_IN_FPCLASSIFY:
 	return 2;
 
+      case BUILT_IN_CLZG:
+      case BUILT_IN_CTZG:
+      case BUILT_IN_CLRSBG:
+      case BUILT_IN_FFSG:
+      case BUILT_IN_PARITYG:
+      case BUILT_IN_POPCOUNTG:
+	return 4;
+
       default:
 	return lookup_attribute ("type generic",
 				 TYPE_ATTRIBUTES (TREE_TYPE (fn))) != 0;
@@ -10122,7 +10132,7 @@ build_over_call (struct z_candidate *cand, int flags, tsubst_flags_t complain)
   for (; arg_index < vec_safe_length (args); ++arg_index)
     {
       tree a = (*args)[arg_index];
-      if (magic == 3 && arg_index == 2)
+      if ((magic == 3 && arg_index == 2) || (magic == 4 && arg_index == 0))
 	{
 	  /* Do no conversions for certain magic varargs.  */
 	  a = mark_type_use (a);
diff --git a/gcc/cp/cp-gimplify.cc b/gcc/cp/cp-gimplify.cc
index 9375a11..795c811 100644
--- a/gcc/cp/cp-gimplify.cc
+++ b/gcc/cp/cp-gimplify.cc
@@ -771,6 +771,10 @@ cp_gimplify_expr (tree *expr_p, gimple_seq *pre_p, gimple_seq *post_p)
 	      default:
 		break;
 	      }
+	  else if (decl
+		   && fndecl_built_in_p (decl, BUILT_IN_CLZG, BUILT_IN_CTZG))
+	    ret = (enum gimplify_status) c_gimplify_expr (expr_p, pre_p,
+							  post_p);
 	}
       break;
 
diff --git a/gcc/doc/extend.texi b/gcc/doc/extend.texi
index 7cdfdf8..406ccc9 100644
--- a/gcc/doc/extend.texi
+++ b/gcc/doc/extend.texi
@@ -14960,6 +14960,48 @@ Similar to @code{__builtin_parity}, except the argument type is
 @code{unsigned long long}.
 @enddefbuiltin
 
+@defbuiltin{int __builtin_ffsg (...)}
+Similar to @code{__builtin_ffs}, except the argument is type-generic
+signed integer (standard, extended or bit-precise).  No integral argument
+promotions are performed on the argument.
+@enddefbuiltin
+
+@defbuiltin{int __builtin_clzg (...)}
+Similar to @code{__builtin_clz}, except the argument is type-generic
+unsigned integer (standard, extended or bit-precise) and there is
+optional second argument with int type.  No integral argument promotions
+are performed on the first argument.  If two arguments are specified,
+and first argument is 0, the result is the second argument.  If only
+one argument is specified and it is 0, the result is undefined.
+@enddefbuiltin
+
+@defbuiltin{int __builtin_ctzg (...)}
+Similar to @code{__builtin_ctz}, except the argument is type-generic
+unsigned integer (standard, extended or bit-precise) and there is
+optional second argument with int type.  No integral argument promotions
+are performed on the first argument.  If two arguments are specified,
+and first argument is 0, the result is the second argument.  If only
+one argument is specified and it is 0, the result is undefined.
+@enddefbuiltin
+
+@defbuiltin{int __builtin_clrsbg (...)}
+Similar to @code{__builtin_clrsb}, except the argument is type-generic
+signed integer (standard, extended or bit-precise).  No integral argument
+promotions are performed on the argument.
+@enddefbuiltin
+
+@defbuiltin{int __builtin_popcountg (...)}
+Similar to @code{__builtin_popcount}, except the argument is type-generic
+unsigned integer (standard, extended or bit-precise).  No integral argument
+promotions are performed on the argument.
+@enddefbuiltin
+
+@defbuiltin{int __builtin_parityg (...)}
+Similar to @code{__builtin_parity}, except the argument is type-generic
+unsigned integer (standard, extended or bit-precise).  No integral argument
+promotions are performed on the argument.
+@enddefbuiltin
+
 @defbuiltin{double __builtin_powi (double, int)}
 @defbuiltinx{float __builtin_powif (float, int)}
 @defbuiltinx{{long double} __builtin_powil (long double, int)}
diff --git a/gcc/fold-const-call.cc b/gcc/fold-const-call.cc
index 04be3d2..36f4ecc 100644
--- a/gcc/fold-const-call.cc
+++ b/gcc/fold-const-call.cc
@@ -27,7 +27,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "fold-const.h"
 #include "fold-const-call.h"
 #include "case-cfn-macros.h"
-#include "tm.h" /* For C[LT]Z_DEFINED_AT_ZERO.  */
+#include "tm.h" /* For C[LT]Z_DEFINED_VALUE_AT_ZERO.  */
 #include "builtins.h"
 #include "gimple-expr.h"
 #include "tree-vector-builder.h"
@@ -1017,14 +1017,18 @@ fold_const_call_ss (wide_int *result, combined_fn fn, const wide_int_ref &arg,
   switch (fn)
     {
     CASE_CFN_FFS:
+    case CFN_BUILT_IN_FFSG:
       *result = wi::shwi (wi::ffs (arg), precision);
       return true;
 
     CASE_CFN_CLZ:
+    case CFN_BUILT_IN_CLZG:
       {
 	int tmp;
 	if (wi::ne_p (arg, 0))
 	  tmp = wi::clz (arg);
+	else if (TREE_CODE (arg_type) == BITINT_TYPE)
+	  tmp = TYPE_PRECISION (arg_type);
 	else if (!CLZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (arg_type),
 					     tmp))
 	  tmp = TYPE_PRECISION (arg_type);
@@ -1033,10 +1037,13 @@ fold_const_call_ss (wide_int *result, combined_fn fn, const wide_int_ref &arg,
       }
 
     CASE_CFN_CTZ:
+    case CFN_BUILT_IN_CTZG:
       {
 	int tmp;
 	if (wi::ne_p (arg, 0))
 	  tmp = wi::ctz (arg);
+	else if (TREE_CODE (arg_type) == BITINT_TYPE)
+	  tmp = TYPE_PRECISION (arg_type);
 	else if (!CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (arg_type),
 					     tmp))
 	  tmp = TYPE_PRECISION (arg_type);
@@ -1045,14 +1052,17 @@ fold_const_call_ss (wide_int *result, combined_fn fn, const wide_int_ref &arg,
       }
 
     CASE_CFN_CLRSB:
+    case CFN_BUILT_IN_CLRSBG:
       *result = wi::shwi (wi::clrsb (arg), precision);
       return true;
 
     CASE_CFN_POPCOUNT:
+    case CFN_BUILT_IN_POPCOUNTG:
       *result = wi::shwi (wi::popcount (arg), precision);
       return true;
 
     CASE_CFN_PARITY:
+    case CFN_BUILT_IN_PARITYG:
       *result = wi::shwi (wi::parity (arg), precision);
       return true;
 
@@ -1531,6 +1541,49 @@ fold_const_call_sss (real_value *result, combined_fn fn,
 
 /* Try to evaluate:
 
+      *RESULT = FN (ARG0, ARG1)
+
+   where ARG_TYPE is the type of ARG0 and PRECISION is the number of bits in
+   the result.  Return true on success.  */
+
+static bool
+fold_const_call_sss (wide_int *result, combined_fn fn,
+		     const wide_int_ref &arg0, const wide_int_ref &arg1,
+		     unsigned int precision, tree arg_type ATTRIBUTE_UNUSED)
+{
+  switch (fn)
+    {
+    case CFN_CLZ:
+    case CFN_BUILT_IN_CLZG:
+      {
+	int tmp;
+	if (wi::ne_p (arg0, 0))
+	  tmp = wi::clz (arg0);
+	else
+	  tmp = arg1.to_shwi ();
+	*result = wi::shwi (tmp, precision);
+	return true;
+      }
+
+    case CFN_CTZ:
+    case CFN_BUILT_IN_CTZG:
+      {
+	int tmp;
+	if (wi::ne_p (arg0, 0))
+	  tmp = wi::ctz (arg0);
+	else
+	  tmp = arg1.to_shwi ();
+	*result = wi::shwi (tmp, precision);
+	return true;
+      }
+
+    default:
+      return false;
+    }
+}
+
+/* Try to evaluate:
+
       RESULT = fn (ARG0, ARG1)
 
    where FORMAT is the format of the real and imaginary parts of RESULT
@@ -1565,6 +1618,19 @@ fold_const_call_1 (combined_fn fn, tree type, tree arg0, tree arg1)
   machine_mode arg0_mode = TYPE_MODE (TREE_TYPE (arg0));
   machine_mode arg1_mode = TYPE_MODE (TREE_TYPE (arg1));
 
+  if (integer_cst_p (arg0) && integer_cst_p (arg1))
+    {
+      if (SCALAR_INT_MODE_P (mode))
+	{
+	  wide_int result;
+	  if (fold_const_call_sss (&result, fn, wi::to_wide (arg0),
+				   wi::to_wide (arg1), TYPE_PRECISION (type),
+				   TREE_TYPE (arg0)))
+	    return wide_int_to_tree (type, result);
+	}
+      return NULL_TREE;
+    }
+
   if (mode == arg0_mode
       && real_cst_p (arg0)
       && real_cst_p (arg1))
diff --git a/gcc/genmatch.cc b/gcc/genmatch.cc
index 4572342..3488764 100644
--- a/gcc/genmatch.cc
+++ b/gcc/genmatch.cc
@@ -1896,8 +1896,14 @@ cmp_operand (operand *o1, operand *o2)
     {
       expr *e1 = static_cast<expr *>(o1);
       expr *e2 = static_cast<expr *>(o2);
-      return (e1->operation == e2->operation
-	      && e1->is_generic == e2->is_generic);
+      if (e1->operation != e2->operation
+	  || e1->is_generic != e2->is_generic)
+	return false;
+      if (e1->operation->kind == id_base::FN
+	  /* For function calls also compare number of arguments.  */
+	  && e1->ops.length () != e2->ops.length ())
+	return false;
+      return true;
     }
   else
     return false;
@@ -3071,6 +3077,26 @@ dt_operand::gen_generic_expr (FILE *f, int indent, const char *opname)
   return 0;
 }
 
+/* Compare 2 fns or generic_fns vector entries for vector sorting.
+   Same operation entries with different number of arguments should
+   be adjacent.  */
+
+static int
+fns_cmp (const void *p1, const void *p2)
+{
+  dt_operand *op1 = *(dt_operand *const *) p1;
+  dt_operand *op2 = *(dt_operand *const *) p2;
+  expr *e1 = as_a <expr *> (op1->op);
+  expr *e2 = as_a <expr *> (op2->op);
+  id_base *b1 = e1->operation;
+  id_base *b2 = e2->operation;
+  if (b1->hashval < b2->hashval)
+    return -1;
+  if (b1->hashval > b2->hashval)
+    return 1;
+  return strcmp (b1->id, b2->id);
+}
+
 /* Generate matching code for the children of the decision tree node.  */
 
 void
@@ -3144,6 +3170,8 @@ dt_node::gen_kids (FILE *f, int indent, bool gimple, int depth)
 	     Like DT_TRUE, DT_MATCH serves as a barrier as it can cause
 	     dependent matches to get out-of-order.  Generate code now
 	     for what we have collected sofar.  */
+	  fns.qsort (fns_cmp);
+	  generic_fns.qsort (fns_cmp);
 	  gen_kids_1 (f, indent, gimple, depth, gimple_exprs, generic_exprs,
 		      fns, generic_fns, preds, others);
 	  /* And output the true operand itself.  */
@@ -3160,6 +3188,8 @@ dt_node::gen_kids (FILE *f, int indent, bool gimple, int depth)
     }
 
   /* Generate code for the remains.  */
+  fns.qsort (fns_cmp);
+  generic_fns.qsort (fns_cmp);
   gen_kids_1 (f, indent, gimple, depth, gimple_exprs, generic_exprs,
 	      fns, generic_fns, preds, others);
 }
@@ -3257,14 +3287,21 @@ dt_node::gen_kids_1 (FILE *f, int indent, bool gimple, int depth,
 
 	  indent += 4;
 	  fprintf_indent (f, indent, "{\n");
+	  id_base *last_op = NULL;
 	  for (unsigned i = 0; i < fns_len; ++i)
 	    {
 	      expr *e = as_a <expr *>(fns[i]->op);
-	      if (user_id *u = dyn_cast <user_id *> (e->operation))
-		for (auto id : u->substitutes)
-		  fprintf_indent (f, indent, "case %s:\n", id->id);
-	      else
-		fprintf_indent (f, indent, "case %s:\n", e->operation->id);
+	      if (e->operation != last_op)
+		{
+		  if (i)
+		    fprintf_indent (f, indent, "  break;\n");
+		  if (user_id *u = dyn_cast <user_id *> (e->operation))
+		    for (auto id : u->substitutes)
+		      fprintf_indent (f, indent, "case %s:\n", id->id);
+		  else
+		    fprintf_indent (f, indent, "case %s:\n", e->operation->id);
+		}
+	      last_op = e->operation;
 	      /* We need to be defensive against bogus prototypes allowing
 		 calls with not enough arguments.  */
 	      fprintf_indent (f, indent,
@@ -3273,9 +3310,9 @@ dt_node::gen_kids_1 (FILE *f, int indent, bool gimple, int depth,
 	      fprintf_indent (f, indent, "    {\n");
 	      fns[i]->gen (f, indent + 6, true, depth);
 	      fprintf_indent (f, indent, "    }\n");
-	      fprintf_indent (f, indent, "  break;\n");
 	    }
 
+	  fprintf_indent (f, indent, "  break;\n");
 	  fprintf_indent (f, indent, "default:;\n");
 	  fprintf_indent (f, indent, "}\n");
 	  indent -= 4;
@@ -3335,18 +3372,25 @@ dt_node::gen_kids_1 (FILE *f, int indent, bool gimple, int depth,
 		      "    {\n");
       indent += 4;
 
+      id_base *last_op = NULL;
       for (unsigned j = 0; j < generic_fns.length (); ++j)
 	{
 	  expr *e = as_a <expr *>(generic_fns[j]->op);
 	  gcc_assert (e->operation->kind == id_base::FN);
 
-	  fprintf_indent (f, indent, "case %s:\n", e->operation->id);
+	  if (e->operation != last_op)
+	    {
+	      if (j)
+		fprintf_indent (f, indent, "  break;\n");
+	      fprintf_indent (f, indent, "case %s:\n", e->operation->id);
+	    }
+	  last_op = e->operation;
 	  fprintf_indent (f, indent, "  if (call_expr_nargs (%s) == %d)\n"
 				     "    {\n", kid_opname, e->ops.length ());
 	  generic_fns[j]->gen (f, indent + 6, false, depth);
-	  fprintf_indent (f, indent, "    }\n"
-				     "  break;\n");
+	  fprintf_indent (f, indent, "    }\n");
 	}
+      fprintf_indent (f, indent, "  break;\n");
       fprintf_indent (f, indent, "default:;\n");
 
       indent -= 4;
diff --git a/gcc/gimple-lower-bitint.cc b/gcc/gimple-lower-bitint.cc
index 6655859..c429cb2 100644
--- a/gcc/gimple-lower-bitint.cc
+++ b/gcc/gimple-lower-bitint.cc
@@ -427,6 +427,7 @@ struct bitint_large_huge
   void lower_mul_overflow (tree, gimple *);
   void lower_cplxpart_stmt (tree, gimple *);
   void lower_complexexpr_stmt (gimple *);
+  void lower_bit_query (gimple *);
   void lower_call (tree, gimple *);
   void lower_asm (gimple *);
   void lower_stmt (gimple *);
@@ -4455,6 +4456,524 @@ bitint_large_huge::lower_complexexpr_stmt (gimple *stmt)
   insert_before (g);
 }
 
+/* Lower a .{CLZ,CTZ,CLRSB,FFS,PARITY,POPCOUNT} call with one large/huge _BitInt
+   argument.  */
+
+void
+bitint_large_huge::lower_bit_query (gimple *stmt)
+{
+  tree arg0 = gimple_call_arg (stmt, 0);
+  tree arg1 = (gimple_call_num_args (stmt) == 2
+	       ? gimple_call_arg (stmt, 1) : NULL_TREE);
+  tree lhs = gimple_call_lhs (stmt);
+  gimple *g;
+
+  if (!lhs)
+    {
+      gimple_stmt_iterator gsi = gsi_for_stmt (stmt);
+      gsi_remove (&gsi, true);
+      return;
+    }
+  tree type = TREE_TYPE (arg0);
+  gcc_assert (TREE_CODE (type) == BITINT_TYPE);
+  bitint_prec_kind kind = bitint_precision_kind (type);
+  gcc_assert (kind >= bitint_prec_large);
+  enum internal_fn ifn = gimple_call_internal_fn (stmt);
+  enum built_in_function fcode = END_BUILTINS;
+  gcc_assert (TYPE_PRECISION (unsigned_type_node) == limb_prec
+	      || TYPE_PRECISION (long_unsigned_type_node) == limb_prec
+	      || TYPE_PRECISION (long_long_unsigned_type_node) == limb_prec);
+  switch (ifn)
+    {
+    case IFN_CLZ:
+      if (TYPE_PRECISION (unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_CLZ;
+      else if (TYPE_PRECISION (long_unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_CLZL;
+      else
+	fcode = BUILT_IN_CLZLL;
+      break;
+    case IFN_FFS:
+      /* .FFS (X) is .CTZ (X, -1) + 1, though under the hood
+	 we don't add the addend at the end.  */
+      arg1 = integer_zero_node;
+      /* FALLTHRU */
+    case IFN_CTZ:
+      if (TYPE_PRECISION (unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_CTZ;
+      else if (TYPE_PRECISION (long_unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_CTZL;
+      else
+	fcode = BUILT_IN_CTZLL;
+      m_upwards = true;
+      break;
+    case IFN_CLRSB:
+      if (TYPE_PRECISION (unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_CLRSB;
+      else if (TYPE_PRECISION (long_unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_CLRSBL;
+      else
+	fcode = BUILT_IN_CLRSBLL;
+      break;
+    case IFN_PARITY:
+      if (TYPE_PRECISION (unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_PARITY;
+      else if (TYPE_PRECISION (long_unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_PARITYL;
+      else
+	fcode = BUILT_IN_PARITYLL;
+      m_upwards = true;
+      break;
+    case IFN_POPCOUNT:
+      if (TYPE_PRECISION (unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_POPCOUNT;
+      else if (TYPE_PRECISION (long_unsigned_type_node) == limb_prec)
+	fcode = BUILT_IN_POPCOUNTL;
+      else
+	fcode = BUILT_IN_POPCOUNTLL;
+      m_upwards = true;
+      break;
+    default:
+      gcc_unreachable ();
+    }
+  tree fndecl = builtin_decl_explicit (fcode), res = NULL_TREE;
+  unsigned cnt = 0, rem = 0, end = 0, prec = TYPE_PRECISION (type);
+  struct bq_details { edge e; tree val, addend; } *bqp = NULL;
+  basic_block edge_bb = NULL;
+  if (m_upwards)
+    {
+      tree idx = NULL_TREE, idx_first = NULL_TREE, idx_next = NULL_TREE;
+      if (kind == bitint_prec_large)
+	cnt = CEIL (prec, limb_prec);
+      else
+	{
+	  rem = (prec % (2 * limb_prec));
+	  end = (prec - rem) / limb_prec;
+	  cnt = 2 + CEIL (rem, limb_prec);
+	  idx = idx_first = create_loop (size_zero_node, &idx_next);
+	}
+
+      if (ifn == IFN_CTZ || ifn == IFN_FFS)
+	{
+	  gimple_stmt_iterator gsi = gsi_for_stmt (stmt);
+	  gsi_prev (&gsi);
+	  edge e = split_block (gsi_bb (gsi), gsi_stmt (gsi));
+	  edge_bb = e->src;
+	  if (kind == bitint_prec_large)
+	    {
+	      m_gsi = gsi_last_bb (edge_bb);
+	      if (!gsi_end_p (m_gsi))
+		gsi_next (&m_gsi);
+	    }
+	  bqp = XALLOCAVEC (struct bq_details, cnt);
+	}
+      else
+	m_after_stmt = stmt;
+      if (kind != bitint_prec_large)
+	m_upwards_2limb = end;
+
+      for (unsigned i = 0; i < cnt; i++)
+	{
+	  m_data_cnt = 0;
+	  if (kind == bitint_prec_large)
+	    idx = size_int (i);
+	  else if (i >= 2)
+	    idx = size_int (end + (i > 2));
+
+	  tree rhs1 = handle_operand (arg0, idx);
+	  if (!useless_type_conversion_p (m_limb_type, TREE_TYPE (rhs1)))
+	    {
+	      if (!TYPE_UNSIGNED (TREE_TYPE (rhs1)))
+		rhs1 = add_cast (unsigned_type_for (TREE_TYPE (rhs1)), rhs1);
+	      rhs1 = add_cast (m_limb_type, rhs1);
+	    }
+
+	  tree in, out, tem;
+	  if (ifn == IFN_PARITY)
+	    in = prepare_data_in_out (build_zero_cst (m_limb_type), idx, &out);
+	  else if (ifn == IFN_FFS)
+	    in = prepare_data_in_out (integer_one_node, idx, &out);
+	  else
+	    in = prepare_data_in_out (integer_zero_node, idx, &out);
+
+	  switch (ifn)
+	    {
+	    case IFN_CTZ:
+	    case IFN_FFS:
+	      g = gimple_build_cond (NE_EXPR, rhs1,
+				     build_zero_cst (m_limb_type),
+				     NULL_TREE, NULL_TREE);
+	      insert_before (g);
+	      edge e1, e2;
+	      e1 = split_block (gsi_bb (m_gsi), g);
+	      e1->flags = EDGE_FALSE_VALUE;
+	      e2 = make_edge (e1->src, gimple_bb (stmt), EDGE_TRUE_VALUE);
+	      e1->probability = profile_probability::unlikely ();
+	      e2->probability = e1->probability.invert ();
+	      if (i == 0)
+		set_immediate_dominator (CDI_DOMINATORS, e2->dest, e2->src);
+	      m_gsi = gsi_after_labels (e1->dest);
+	      bqp[i].e = e2;
+	      bqp[i].val = rhs1;
+	      if (tree_fits_uhwi_p (idx))
+		bqp[i].addend
+		  = build_int_cst (integer_type_node,
+				   tree_to_uhwi (idx) * limb_prec
+				   + (ifn == IFN_FFS));
+	      else
+		{
+		  bqp[i].addend = in;
+		  if (i == 1)
+		    res = out;
+		  else
+		    res = make_ssa_name (integer_type_node);
+		  g = gimple_build_assign (res, PLUS_EXPR, in,
+					   build_int_cst (integer_type_node,
+							  limb_prec));
+		  insert_before (g);
+		  m_data[m_data_cnt] = res;
+		}
+	      break;
+	    case IFN_PARITY:
+	      if (!integer_zerop (in))
+		{
+		  if (kind == bitint_prec_huge && i == 1)
+		    res = out;
+		  else
+		    res = make_ssa_name (m_limb_type);
+		  g = gimple_build_assign (res, BIT_XOR_EXPR, in, rhs1);
+		  insert_before (g);
+		}
+	      else
+		res = rhs1;
+	      m_data[m_data_cnt] = res;
+	      break;
+	    case IFN_POPCOUNT:
+	      g = gimple_build_call (fndecl, 1, rhs1);
+	      tem = make_ssa_name (integer_type_node);
+	      gimple_call_set_lhs (g, tem);
+	      insert_before (g);
+	      if (!integer_zerop (in))
+		{
+		  if (kind == bitint_prec_huge && i == 1)
+		    res = out;
+		  else
+		    res = make_ssa_name (integer_type_node);
+		  g = gimple_build_assign (res, PLUS_EXPR, in, tem);
+		  insert_before (g);
+		}
+	      else
+		res = tem;
+	      m_data[m_data_cnt] = res;
+	      break;
+	    default:
+	      gcc_unreachable ();
+	    }
+
+	  m_first = false;
+	  if (kind == bitint_prec_huge && i <= 1)
+	    {
+	      if (i == 0)
+		{
+		  idx = make_ssa_name (sizetype);
+		  g = gimple_build_assign (idx, PLUS_EXPR, idx_first,
+					   size_one_node);
+		  insert_before (g);
+		}
+	      else
+		{
+		  g = gimple_build_assign (idx_next, PLUS_EXPR, idx_first,
+					   size_int (2));
+		  insert_before (g);
+		  g = gimple_build_cond (NE_EXPR, idx_next, size_int (end),
+					 NULL_TREE, NULL_TREE);
+		  insert_before (g);
+		  if (ifn == IFN_CTZ || ifn == IFN_FFS)
+		    m_gsi = gsi_after_labels (edge_bb);
+		  else
+		    m_gsi = gsi_for_stmt (stmt);
+		}
+	    }
+	}
+    }
+  else
+    {
+      tree idx = NULL_TREE, idx_next = NULL_TREE, first = NULL_TREE;
+      int sub_one = 0;
+      if (kind == bitint_prec_large)
+	cnt = CEIL (prec, limb_prec);
+      else
+	{
+	  rem = prec % limb_prec;
+	  if (rem == 0 && (!TYPE_UNSIGNED (type) || ifn == IFN_CLRSB))
+	    rem = limb_prec;
+	  end = (prec - rem) / limb_prec;
+	  cnt = 1 + (rem != 0);
+	  if (ifn == IFN_CLRSB)
+	    sub_one = 1;
+	}
+
+      gimple_stmt_iterator gsi = gsi_for_stmt (stmt);
+      gsi_prev (&gsi);
+      edge e = split_block (gsi_bb (gsi), gsi_stmt (gsi));
+      edge_bb = e->src;
+      m_gsi = gsi_last_bb (edge_bb);
+      if (!gsi_end_p (m_gsi))
+	gsi_next (&m_gsi);
+
+      if (ifn == IFN_CLZ)
+	bqp = XALLOCAVEC (struct bq_details, cnt);
+      else
+	{
+	  gsi = gsi_for_stmt (stmt);
+	  gsi_prev (&gsi);
+	  e = split_block (gsi_bb (gsi), gsi_stmt (gsi));
+	  edge_bb = e->src;
+	  bqp = XALLOCAVEC (struct bq_details, 2 * cnt);
+	}
+
+      for (unsigned i = 0; i < cnt; i++)
+	{
+	  m_data_cnt = 0;
+	  if (kind == bitint_prec_large)
+	    idx = size_int (cnt - i - 1);
+	  else if (i == cnt - 1)
+	    idx = create_loop (size_int (end - 1), &idx_next);
+	  else
+	    idx = size_int (end);
+
+	  tree rhs1 = handle_operand (arg0, idx);
+	  if (!useless_type_conversion_p (m_limb_type, TREE_TYPE (rhs1)))
+	    {
+	      if (ifn == IFN_CLZ && !TYPE_UNSIGNED (TREE_TYPE (rhs1)))
+		rhs1 = add_cast (unsigned_type_for (TREE_TYPE (rhs1)), rhs1);
+	      else if (ifn == IFN_CLRSB && TYPE_UNSIGNED (TREE_TYPE (rhs1)))
+		rhs1 = add_cast (signed_type_for (TREE_TYPE (rhs1)), rhs1);
+	      rhs1 = add_cast (m_limb_type, rhs1);
+	    }
+
+	  if (ifn == IFN_CLZ)
+	    {
+	      g = gimple_build_cond (NE_EXPR, rhs1,
+				     build_zero_cst (m_limb_type),
+				     NULL_TREE, NULL_TREE);
+	      insert_before (g);
+	      edge e1 = split_block (gsi_bb (m_gsi), g);
+	      e1->flags = EDGE_FALSE_VALUE;
+	      edge e2 = make_edge (e1->src, gimple_bb (stmt), EDGE_TRUE_VALUE);
+	      e1->probability = profile_probability::unlikely ();
+	      e2->probability = e1->probability.invert ();
+	      if (i == 0)
+		set_immediate_dominator (CDI_DOMINATORS, e2->dest, e2->src);
+	      m_gsi = gsi_after_labels (e1->dest);
+	      bqp[i].e = e2;
+	      bqp[i].val = rhs1;
+	    }
+	  else
+	    {
+	      if (i == 0)
+		{
+		  first = rhs1;
+		  g = gimple_build_assign (make_ssa_name (m_limb_type),
+					   PLUS_EXPR, rhs1,
+					   build_int_cst (m_limb_type, 1));
+		  insert_before (g);
+		  g = gimple_build_cond (GT_EXPR, gimple_assign_lhs (g),
+					 build_int_cst (m_limb_type, 1),
+					 NULL_TREE, NULL_TREE);
+		  insert_before (g);
+		}
+	      else
+		{
+		  g = gimple_build_assign (make_ssa_name (m_limb_type),
+					   BIT_XOR_EXPR, rhs1, first);
+		  insert_before (g);
+		  tree stype = signed_type_for (m_limb_type);
+		  g = gimple_build_cond (LT_EXPR,
+					 add_cast (stype,
+						   gimple_assign_lhs (g)),
+					 build_zero_cst (stype),
+					 NULL_TREE, NULL_TREE);
+		  insert_before (g);
+		  edge e1 = split_block (gsi_bb (m_gsi), g);
+		  e1->flags = EDGE_FALSE_VALUE;
+		  edge e2 = make_edge (e1->src, gimple_bb (stmt),
+				       EDGE_TRUE_VALUE);
+		  e1->probability = profile_probability::unlikely ();
+		  e2->probability = e1->probability.invert ();
+		  if (i == 1)
+		    set_immediate_dominator (CDI_DOMINATORS, e2->dest,
+					     e2->src);
+		  m_gsi = gsi_after_labels (e1->dest);
+		  bqp[2 * i].e = e2;
+		  g = gimple_build_cond (NE_EXPR, rhs1, first,
+					 NULL_TREE, NULL_TREE);
+		  insert_before (g);
+		}
+	      edge e1 = split_block (gsi_bb (m_gsi), g);
+	      e1->flags = EDGE_FALSE_VALUE;
+	      edge e2 = make_edge (e1->src, edge_bb, EDGE_TRUE_VALUE);
+	      e1->probability = profile_probability::unlikely ();
+	      e2->probability = e1->probability.invert ();
+	      if (i == 0)
+		set_immediate_dominator (CDI_DOMINATORS, e2->dest, e2->src);
+	      m_gsi = gsi_after_labels (e1->dest);
+	      bqp[2 * i + 1].e = e2;
+	      bqp[i].val = rhs1;
+	    }
+	  if (tree_fits_uhwi_p (idx))
+	    bqp[i].addend
+	      = build_int_cst (integer_type_node,
+			       (int) prec
+			       - (((int) tree_to_uhwi (idx) + 1)
+				  * limb_prec) - sub_one);
+	  else
+	    {
+	      tree in, out;
+	      in = build_int_cst (integer_type_node, rem - sub_one);
+	      m_first = true;
+	      in = prepare_data_in_out (in, idx, &out);
+	      out = m_data[m_data_cnt + 1];
+	      bqp[i].addend = in;
+	      g = gimple_build_assign (out, PLUS_EXPR, in,
+				       build_int_cst (integer_type_node,
+						      limb_prec));
+	      insert_before (g);
+	      m_data[m_data_cnt] = out;
+	    }
+
+	  m_first = false;
+	  if (kind == bitint_prec_huge && i == cnt - 1)
+	    {
+	      g = gimple_build_assign (idx_next, PLUS_EXPR, idx,
+				       size_int (-1));
+	      insert_before (g);
+	      g = gimple_build_cond (NE_EXPR, idx, size_zero_node,
+				     NULL_TREE, NULL_TREE);
+	      insert_before (g);
+	      edge true_edge, false_edge;
+	      extract_true_false_edges_from_block (gsi_bb (m_gsi),
+						   &true_edge, &false_edge);
+	      m_gsi = gsi_after_labels (false_edge->dest);
+	    }
+	}
+    }
+  switch (ifn)
+    {
+    case IFN_CLZ:
+    case IFN_CTZ:
+    case IFN_FFS:
+      gphi *phi1, *phi2, *phi3;
+      basic_block bb;
+      bb = gsi_bb (m_gsi);
+      remove_edge (find_edge (bb, gimple_bb (stmt)));
+      phi1 = create_phi_node (make_ssa_name (m_limb_type),
+			      gimple_bb (stmt));
+      phi2 = create_phi_node (make_ssa_name (integer_type_node),
+			      gimple_bb (stmt));
+      for (unsigned i = 0; i < cnt; i++)
+	{
+	  add_phi_arg (phi1, bqp[i].val, bqp[i].e, UNKNOWN_LOCATION);
+	  add_phi_arg (phi2, bqp[i].addend, bqp[i].e, UNKNOWN_LOCATION);
+	}
+      if (arg1 == NULL_TREE)
+	{
+	  g = gimple_build_builtin_unreachable (m_loc);
+	  insert_before (g);
+	}
+      m_gsi = gsi_for_stmt (stmt);
+      g = gimple_build_call (fndecl, 1, gimple_phi_result (phi1));
+      gimple_call_set_lhs (g, make_ssa_name (integer_type_node));
+      insert_before (g);
+      if (arg1 == NULL_TREE)
+	g = gimple_build_assign (lhs, PLUS_EXPR,
+				 gimple_phi_result (phi2),
+				 gimple_call_lhs (g));
+      else
+	{
+	  g = gimple_build_assign (make_ssa_name (integer_type_node),
+				   PLUS_EXPR, gimple_phi_result (phi2),
+				   gimple_call_lhs (g));
+	  insert_before (g);
+	  edge e1 = split_block (gimple_bb (stmt), g);
+	  edge e2 = make_edge (bb, e1->dest, EDGE_FALLTHRU);
+	  e2->probability = profile_probability::always ();
+	  set_immediate_dominator (CDI_DOMINATORS, e1->dest,
+				   get_immediate_dominator (CDI_DOMINATORS,
+							    e1->src));
+	  phi3 = create_phi_node (make_ssa_name (integer_type_node), e1->dest);
+	  add_phi_arg (phi3, gimple_assign_lhs (g), e1, UNKNOWN_LOCATION);
+	  add_phi_arg (phi3, arg1, e2, UNKNOWN_LOCATION);
+	  m_gsi = gsi_for_stmt (stmt);
+	  g = gimple_build_assign (lhs, gimple_phi_result (phi3));
+	}
+      gsi_replace (&m_gsi, g, true);
+      break;
+    case IFN_CLRSB:
+      bb = gsi_bb (m_gsi);
+      remove_edge (find_edge (bb, edge_bb));
+      edge e;
+      e = make_edge (bb, gimple_bb (stmt), EDGE_FALLTHRU);
+      e->probability = profile_probability::always ();
+      set_immediate_dominator (CDI_DOMINATORS, gimple_bb (stmt),
+			       get_immediate_dominator (CDI_DOMINATORS,
+							edge_bb));
+      phi1 = create_phi_node (make_ssa_name (m_limb_type),
+			      edge_bb);
+      phi2 = create_phi_node (make_ssa_name (integer_type_node),
+			      edge_bb);
+      phi3 = create_phi_node (make_ssa_name (integer_type_node),
+			      gimple_bb (stmt));
+      for (unsigned i = 0; i < cnt; i++)
+	{
+	  add_phi_arg (phi1, bqp[i].val, bqp[2 * i + 1].e, UNKNOWN_LOCATION);
+	  add_phi_arg (phi2, bqp[i].addend, bqp[2 * i + 1].e,
+		       UNKNOWN_LOCATION);
+	  tree a = bqp[i].addend;
+	  if (i && kind == bitint_prec_large)
+	    a = int_const_binop (PLUS_EXPR, a, integer_minus_one_node);
+	  if (i)
+	    add_phi_arg (phi3, a, bqp[2 * i].e, UNKNOWN_LOCATION);
+	}
+      add_phi_arg (phi3, build_int_cst (integer_type_node, prec - 1), e,
+		   UNKNOWN_LOCATION);
+      m_gsi = gsi_after_labels (edge_bb);
+      g = gimple_build_call (fndecl, 1,
+			     add_cast (signed_type_for (m_limb_type),
+				       gimple_phi_result (phi1)));
+      gimple_call_set_lhs (g, make_ssa_name (integer_type_node));
+      insert_before (g);
+      g = gimple_build_assign (make_ssa_name (integer_type_node),
+			       PLUS_EXPR, gimple_call_lhs (g),
+			       gimple_phi_result (phi2));
+      insert_before (g);
+      if (kind != bitint_prec_large)
+	{
+	  g = gimple_build_assign (make_ssa_name (integer_type_node),
+				   PLUS_EXPR, gimple_assign_lhs (g),
+				   integer_one_node);
+	  insert_before (g);
+	}
+      add_phi_arg (phi3, gimple_assign_lhs (g),
+		   find_edge (edge_bb, gimple_bb (stmt)), UNKNOWN_LOCATION);
+      m_gsi = gsi_for_stmt (stmt);
+      g = gimple_build_assign (lhs, gimple_phi_result (phi3));
+      gsi_replace (&m_gsi, g, true);
+      break;
+    case IFN_PARITY:
+      g = gimple_build_call (fndecl, 1, res);
+      gimple_call_set_lhs (g, lhs);
+      gsi_replace (&m_gsi, g, true);
+      break;
+    case IFN_POPCOUNT:
+      g = gimple_build_assign (lhs, res);
+      gsi_replace (&m_gsi, g, true);
+      break;
+    default:
+      gcc_unreachable ();
+    }
+}
+
 /* Lower a call statement with one or more large/huge _BitInt
    arguments or large/huge _BitInt return value.  */
 
@@ -4476,6 +4995,14 @@ bitint_large_huge::lower_call (tree obj, gimple *stmt)
       case IFN_UBSAN_CHECK_MUL:
 	lower_mul_overflow (obj, stmt);
 	return;
+      case IFN_CLZ:
+      case IFN_CTZ:
+      case IFN_CLRSB:
+      case IFN_FFS:
+      case IFN_PARITY:
+      case IFN_POPCOUNT:
+	lower_bit_query (stmt);
+	return;
       default:
 	break;
       }
diff --git a/gcc/gimple-range-op.cc b/gcc/gimple-range-op.cc
index 67b3c3d..26c42da 100644
--- a/gcc/gimple-range-op.cc
+++ b/gcc/gimple-range-op.cc
@@ -908,39 +908,34 @@ public:
   cfn_clz (bool internal) { m_gimple_call_internal_p = internal; }
   using range_operator::fold_range;
   virtual bool fold_range (irange &r, tree type, const irange &lh,
-			   const irange &, relation_trio) const;
+			   const irange &rh, relation_trio) const;
 private:
   bool m_gimple_call_internal_p;
 } op_cfn_clz (false), op_cfn_clz_internal (true);
 
 bool
 cfn_clz::fold_range (irange &r, tree type, const irange &lh,
-		     const irange &, relation_trio) const
+		     const irange &rh, relation_trio) const
 {
   // __builtin_c[lt]z* return [0, prec-1], except when the
   // argument is 0, but that is undefined behavior.
   //
   // For __builtin_c[lt]z* consider argument of 0 always undefined
-  // behavior, for internal fns depending on C?Z_DEFINED_VALUE_AT_ZERO.
+  // behavior, for internal fns likewise, unless it has 2 arguments,
+  // then the second argument is the value at zero.
   if (lh.undefined_p ())
     return false;
   int prec = TYPE_PRECISION (lh.type ());
   int mini = 0;
   int maxi = prec - 1;
-  int zerov = 0;
-  scalar_int_mode mode = SCALAR_INT_TYPE_MODE (lh.type ());
   if (m_gimple_call_internal_p)
     {
-      if (optab_handler (clz_optab, mode) != CODE_FOR_nothing
-	  && CLZ_DEFINED_VALUE_AT_ZERO (mode, zerov) == 2)
-	{
-	  // Only handle the single common value.
-	  if (zerov == prec)
-	    maxi = prec;
-	  else
-	    // Magic value to give up, unless we can prove arg is non-zero.
-	    mini = -2;
-	}
+      // Only handle the single common value.
+      if (rh.lower_bound () == prec)
+	maxi = prec;
+      else
+	// Magic value to give up, unless we can prove arg is non-zero.
+	mini = -2;
     }
 
   // From clz of minimum we can compute result maximum.
@@ -985,37 +980,31 @@ public:
   cfn_ctz (bool internal) { m_gimple_call_internal_p = internal; }
   using range_operator::fold_range;
   virtual bool fold_range (irange &r, tree type, const irange &lh,
-			   const irange &, relation_trio) const;
+			   const irange &rh, relation_trio) const;
 private:
   bool m_gimple_call_internal_p;
 } op_cfn_ctz (false), op_cfn_ctz_internal (true);
 
 bool
 cfn_ctz::fold_range (irange &r, tree type, const irange &lh,
-		     const irange &, relation_trio) const
+		     const irange &rh, relation_trio) const
 {
   if (lh.undefined_p ())
     return false;
   int prec = TYPE_PRECISION (lh.type ());
   int mini = 0;
   int maxi = prec - 1;
-  int zerov = 0;
-  scalar_int_mode mode = SCALAR_INT_TYPE_MODE (lh.type ());
 
   if (m_gimple_call_internal_p)
     {
-      if (optab_handler (ctz_optab, mode) != CODE_FOR_nothing
-	  && CTZ_DEFINED_VALUE_AT_ZERO (mode, zerov) == 2)
-	{
-	  // Handle only the two common values.
-	  if (zerov == -1)
-	    mini = -1;
-	  else if (zerov == prec)
-	    maxi = prec;
-	  else
-	    // Magic value to give up, unless we can prove arg is non-zero.
-	    mini = -2;
-	}
+      // Handle only the two common values.
+      if (rh.lower_bound () == -1)
+	mini = -1;
+      else if (rh.lower_bound () == prec)
+	maxi = prec;
+      else
+	// Magic value to give up, unless we can prove arg is non-zero.
+	mini = -2;
     }
   // If arg is non-zero, then use [0, prec - 1].
   if (!range_includes_zero_p (&lh))
@@ -1288,16 +1277,24 @@ gimple_range_op_handler::maybe_builtin_call ()
 
     CASE_CFN_CLZ:
       m_op1 = gimple_call_arg (call, 0);
-      if (gimple_call_internal_p (call))
-	m_operator = &op_cfn_clz_internal;
+      if (gimple_call_internal_p (call)
+	  && gimple_call_num_args (call) == 2)
+	{
+	  m_op2 = gimple_call_arg (call, 1);
+	  m_operator = &op_cfn_clz_internal;
+	}
       else
 	m_operator = &op_cfn_clz;
       break;
 
     CASE_CFN_CTZ:
       m_op1 = gimple_call_arg (call, 0);
-      if (gimple_call_internal_p (call))
-	m_operator = &op_cfn_ctz_internal;
+      if (gimple_call_internal_p (call)
+	  && gimple_call_num_args (call) == 2)
+	{
+	  m_op2 = gimple_call_arg (call, 1);
+	  m_operator = &op_cfn_ctz_internal;
+	}
       else
 	m_operator = &op_cfn_ctz;
       break;
diff --git a/gcc/match.pd b/gcc/match.pd
index 281c6c0..59791f4 100644
--- a/gcc/match.pd
+++ b/gcc/match.pd
@@ -8536,31 +8536,34 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)
    (op (clz:s@2 @0) INTEGER_CST@1)
    (if (integer_zerop (@1) && single_use (@2))
     /* clz(X) == 0 is (int)X < 0 and clz(X) != 0 is (int)X >= 0.  */
-    (with { tree type0 = TREE_TYPE (@0);
-	    tree stype = signed_type_for (type0);
-	    HOST_WIDE_INT val = 0;
-	    /* Punt on hypothetical weird targets.  */
-	    if (clz == CFN_CLZ
-		&& CLZ_DEFINED_VALUE_AT_ZERO (SCALAR_TYPE_MODE (type0),
-					      val) == 2
-		&& val == 0)
-	      stype = NULL_TREE;
-	  }
-     (if (stype)
-      (cmp (convert:stype @0) { build_zero_cst (stype); })))
+    (with { tree stype = signed_type_for (TREE_TYPE (@0)); }
+     (cmp (convert:stype @0) { build_zero_cst (stype); }))
     /* clz(X) == (prec-1) is X == 1 and clz(X) != (prec-1) is X != 1.  */
-    (with { bool ok = true;
-	    HOST_WIDE_INT val = 0;
-	    tree type0 = TREE_TYPE (@0);
-	    /* Punt on hypothetical weird targets.  */
-	    if (clz == CFN_CLZ
-		&& CLZ_DEFINED_VALUE_AT_ZERO (SCALAR_TYPE_MODE (type0),
-					      val) == 2
-		&& val == TYPE_PRECISION (type0) - 1)
-	      ok = false;
-	  }
-     (if (ok && wi::to_wide (@1) == (TYPE_PRECISION (type0) - 1))
-      (op @0 { build_one_cst (type0); })))))))
+    (if (wi::to_wide (@1) == TYPE_PRECISION (TREE_TYPE (@0)) - 1)
+     (op @0 { build_one_cst (TREE_TYPE (@0)); }))))))
+(for op (eq ne)
+     cmp (lt ge)
+ (simplify
+  (op (IFN_CLZ:s@2 @0 @3) INTEGER_CST@1)
+  (if (integer_zerop (@1) && single_use (@2))
+   /* clz(X) == 0 is (int)X < 0 and clz(X) != 0 is (int)X >= 0.  */
+   (with { tree type0 = TREE_TYPE (@0);
+	   tree stype = signed_type_for (TREE_TYPE (@0));
+	   /* Punt if clz(0) == 0.  */
+	   if (integer_zerop (@3))
+	     stype = NULL_TREE;
+	 }
+    (if (stype)
+     (cmp (convert:stype @0) { build_zero_cst (stype); })))
+   /* clz(X) == (prec-1) is X == 1 and clz(X) != (prec-1) is X != 1.  */
+   (with { bool ok = true;
+	   tree type0 = TREE_TYPE (@0);
+	   /* Punt if clz(0) == prec - 1.  */
+	   if (wi::to_widest (@3) == TYPE_PRECISION (type0) - 1)
+	     ok = false;
+	 }
+    (if (ok && wi::to_wide (@1) == (TYPE_PRECISION (type0) - 1))
+     (op @0 { build_one_cst (type0); }))))))
 
 /* CTZ simplifications.  */
 (for ctz (CTZ)
@@ -8585,22 +8588,14 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)
 		      val++;
 		  }
 	      }
-	    bool zero_res = false;
-	    HOST_WIDE_INT zero_val = 0;
 	    tree type0 = TREE_TYPE (@0);
 	    int prec = TYPE_PRECISION (type0);
-	    if (ctz == CFN_CTZ
-		&& CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_TYPE_MODE (type0),
-					      zero_val) == 2)
-	      zero_res = true;
 	  }
-     (if (val <= 0)
-      (if (ok && (!zero_res || zero_val >= val))
-       { constant_boolean_node (cmp == EQ_EXPR ? true : false, type); })
-      (if (val >= prec)
-       (if (ok && (!zero_res || zero_val < val))
-	{ constant_boolean_node (cmp == EQ_EXPR ? false : true, type); })
-       (if (ok && (!zero_res || zero_val < 0 || zero_val >= prec))
+     (if (ok && prec <= MAX_FIXED_MODE_SIZE)
+      (if (val <= 0)
+       { constant_boolean_node (cmp == EQ_EXPR ? true : false, type); }
+       (if (val >= prec)
+	{ constant_boolean_node (cmp == EQ_EXPR ? false : true, type); }
 	(cmp (bit_and @0 { wide_int_to_tree (type0,
 					     wi::mask (val, false, prec)); })
 	     { build_zero_cst (type0); })))))))
@@ -8608,19 +8603,68 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)
   (simplify
    /* __builtin_ctz (x) == C -> (x & ((1 << (C + 1)) - 1)) == (1 << C).  */
    (op (ctz:s @0) INTEGER_CST@1)
-    (with { bool zero_res = false;
-	    HOST_WIDE_INT zero_val = 0;
-	    tree type0 = TREE_TYPE (@0);
+    (with { tree type0 = TREE_TYPE (@0);
 	    int prec = TYPE_PRECISION (type0);
-	    if (ctz == CFN_CTZ
-		&& CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_TYPE_MODE (type0),
-					      zero_val) == 2)
-	      zero_res = true;
 	  }
+     (if (prec <= MAX_FIXED_MODE_SIZE)
+      (if (tree_int_cst_sgn (@1) < 0 || wi::to_widest (@1) >= prec)
+       { constant_boolean_node (op == EQ_EXPR ? false : true, type); }
+       (op (bit_and @0 { wide_int_to_tree (type0,
+					   wi::mask (tree_to_uhwi (@1) + 1,
+						     false, prec)); })
+	   { wide_int_to_tree (type0,
+			       wi::shifted_mask (tree_to_uhwi (@1), 1,
+						 false, prec)); })))))))
+(for op (ge gt le lt)
+     cmp (eq eq ne ne)
+ (simplify
+  /* __builtin_ctz (x) >= C -> (x & ((1 << C) - 1)) == 0.  */
+  (op (IFN_CTZ:s @0 @2) INTEGER_CST@1)
+   (with { bool ok = true;
+	   HOST_WIDE_INT val = 0;
+	   if (!tree_fits_shwi_p (@1))
+	     ok = false;
+	   else
+	     {
+	       val = tree_to_shwi (@1);
+	       /* Canonicalize to >= or <.  */
+	       if (op == GT_EXPR || op == LE_EXPR)
+		 {
+		   if (val == HOST_WIDE_INT_MAX)
+		     ok = false;
+		   else
+		     val++;
+		 }
+	     }
+	   HOST_WIDE_INT zero_val = tree_to_shwi (@2);
+	   tree type0 = TREE_TYPE (@0);
+	   int prec = TYPE_PRECISION (type0);
+	   if (prec > MAX_FIXED_MODE_SIZE)
+	     ok = false;
+	  }
+     (if (val <= 0)
+      (if (ok && zero_val >= val)
+       { constant_boolean_node (cmp == EQ_EXPR ? true : false, type); })
+      (if (val >= prec)
+       (if (ok && zero_val < val)
+	{ constant_boolean_node (cmp == EQ_EXPR ? false : true, type); })
+       (if (ok && (zero_val < 0 || zero_val >= prec))
+	(cmp (bit_and @0 { wide_int_to_tree (type0,
+					     wi::mask (val, false, prec)); })
+	     { build_zero_cst (type0); })))))))
+(for op (eq ne)
+ (simplify
+  /* __builtin_ctz (x) == C -> (x & ((1 << (C + 1)) - 1)) == (1 << C).  */
+  (op (IFN_CTZ:s @0 @2) INTEGER_CST@1)
+   (with { HOST_WIDE_INT zero_val = tree_to_shwi (@2);
+	   tree type0 = TREE_TYPE (@0);
+	   int prec = TYPE_PRECISION (type0);
+	 }
+    (if (prec <= MAX_FIXED_MODE_SIZE)
      (if (tree_int_cst_sgn (@1) < 0 || wi::to_widest (@1) >= prec)
-      (if (!zero_res || zero_val != wi::to_widest (@1))
+      (if (zero_val != wi::to_widest (@1))
        { constant_boolean_node (op == EQ_EXPR ? false : true, type); })
-      (if (!zero_res || zero_val < 0 || zero_val >= prec)
+      (if (zero_val < 0 || zero_val >= prec)
        (op (bit_and @0 { wide_int_to_tree (type0,
 					   wi::mask (tree_to_uhwi (@1) + 1,
 						     false, prec)); })
@@ -8757,13 +8801,38 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)
   (cond (ne @0 integer_zerop@1) (func (convert?@3 @0)) INTEGER_CST@2)
   (with { int val;
 	  internal_fn ifn = IFN_LAST;
-	  if (direct_internal_fn_supported_p (IFN_CLZ, type, OPTIMIZE_FOR_BOTH)
-	      && CLZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (type),
-					    val) == 2)
+	  if (TREE_CODE (TREE_TYPE (@3)) == BITINT_TYPE)
+	    {
+	      if (tree_fits_shwi_p (@2))
+		{
+		  HOST_WIDE_INT valw = tree_to_shwi (@2);
+		  if ((int) valw == valw)
+		    {
+		      val = valw;
+		      ifn = IFN_CLZ;
+		    }
+		}
+	    }
+	  else if (direct_internal_fn_supported_p (IFN_CLZ, TREE_TYPE (@3),
+						   OPTIMIZE_FOR_BOTH)
+		   && CLZ_DEFINED_VALUE_AT_ZERO
+			(SCALAR_INT_TYPE_MODE (TREE_TYPE (@3)), val) == 2)
 	    ifn = IFN_CLZ;
 	}
    (if (ifn == IFN_CLZ && wi::to_widest (@2) == val)
-    (IFN_CLZ @3)))))
+    (IFN_CLZ @3 @2)))))
+(simplify
+ (cond (ne @0 integer_zerop@1) (IFN_CLZ (convert?@3 @0) INTEGER_CST@2) @2)
+  (with { int val;
+	  internal_fn ifn = IFN_LAST;
+	  if (TREE_CODE (TREE_TYPE (@3)) == BITINT_TYPE)
+	    ifn = IFN_CLZ;
+	  else if (direct_internal_fn_supported_p (IFN_CLZ, TREE_TYPE (@3),
+						   OPTIMIZE_FOR_BOTH))
+	    ifn = IFN_CLZ;
+	}
+   (if (ifn == IFN_CLZ)
+    (IFN_CLZ @3 @2))))
 
 /* a != 0 ? CTZ(a) : CST -> .CTZ(a) where CST is the result of the internal function for 0. */
 (for func (CTZ)
@@ -8771,13 +8840,38 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)
   (cond (ne @0 integer_zerop@1) (func (convert?@3 @0)) INTEGER_CST@2)
   (with { int val;
 	  internal_fn ifn = IFN_LAST;
-	  if (direct_internal_fn_supported_p (IFN_CTZ, type, OPTIMIZE_FOR_BOTH)
-	      && CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (type),
-					    val) == 2)
+	  if (TREE_CODE (TREE_TYPE (@3)) == BITINT_TYPE)
+	    {
+	      if (tree_fits_shwi_p (@2))
+		{
+		  HOST_WIDE_INT valw = tree_to_shwi (@2);
+		  if ((int) valw == valw)
+		    {
+		      val = valw;
+		      ifn = IFN_CTZ;
+		    }
+		}
+	    }
+	  else if (direct_internal_fn_supported_p (IFN_CTZ, TREE_TYPE (@3),
+						   OPTIMIZE_FOR_BOTH)
+		   && CTZ_DEFINED_VALUE_AT_ZERO
+			(SCALAR_INT_TYPE_MODE (TREE_TYPE (@3)), val) == 2)
 	    ifn = IFN_CTZ;
 	}
    (if (ifn == IFN_CTZ && wi::to_widest (@2) == val)
-    (IFN_CTZ @3)))))
+    (IFN_CTZ @3 @2)))))
+(simplify
+ (cond (ne @0 integer_zerop@1) (IFN_CTZ (convert?@3 @0) INTEGER_CST@2) @2)
+  (with { int val;
+	  internal_fn ifn = IFN_LAST;
+	  if (TREE_CODE (TREE_TYPE (@3)) == BITINT_TYPE)
+	    ifn = IFN_CTZ;
+	  else if (direct_internal_fn_supported_p (IFN_CTZ, TREE_TYPE (@3),
+						   OPTIMIZE_FOR_BOTH))
+	    ifn = IFN_CTZ;
+	}
+   (if (ifn == IFN_CTZ)
+    (IFN_CTZ @3 @2))))
 #endif
 
 /* Common POPCOUNT/PARITY simplifications.  */
diff --git a/gcc/testsuite/c-c++-common/pr111309-1.c b/gcc/testsuite/c-c++-common/pr111309-1.c
new file mode 100644
index 0000000..fdf3bf1
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/pr111309-1.c
@@ -0,0 +1,470 @@
+/* PR c/111309 */
+/* { dg-do run } */
+/* { dg-options "-O2" } */
+
+__attribute__((noipa)) int
+clzc (unsigned char x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzc2 (unsigned char x, int y)
+{
+  return __builtin_clzg (x, y);
+}
+
+__attribute__((noipa)) int
+clzs (unsigned short x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzs2 (unsigned short x)
+{
+  return __builtin_clzg (x, -2);
+}
+
+__attribute__((noipa)) int
+clzi (unsigned int x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzi2 (unsigned int x, int y)
+{
+  return __builtin_clzg (x, y);
+}
+
+__attribute__((noipa)) int
+clzl (unsigned long x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzl2 (unsigned long x)
+{
+  return __builtin_clzg (x, -1);
+}
+
+__attribute__((noipa)) int
+clzL (unsigned long long x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzL2 (unsigned long long x, int y)
+{
+  return __builtin_clzg (x, y);
+}
+
+#ifdef __SIZEOF_INT128__
+__attribute__((noipa)) int
+clzI (unsigned __int128 x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzI2 (unsigned __int128 x, int y)
+{
+  return __builtin_clzg (x, y);
+}
+#endif
+
+__attribute__((noipa)) int
+ctzc (unsigned char x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzc2 (unsigned char x, int y)
+{
+  return __builtin_ctzg (x, y);
+}
+
+__attribute__((noipa)) int
+ctzs (unsigned short x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzs2 (unsigned short x, int y)
+{
+  return __builtin_ctzg (x, y);
+}
+
+__attribute__((noipa)) int
+ctzi (unsigned int x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzi2 (unsigned int x, int y)
+{
+  return __builtin_ctzg (x, y);
+}
+
+__attribute__((noipa)) int
+ctzl (unsigned long x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzl2 (unsigned long x, int y)
+{
+  return __builtin_ctzg (x, y);
+}
+
+__attribute__((noipa)) int
+ctzL (unsigned long long x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzL2 (unsigned long long x, int y)
+{
+  return __builtin_ctzg (x, y);
+}
+
+#ifdef __SIZEOF_INT128__
+__attribute__((noipa)) int
+ctzI (unsigned __int128 x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzI2 (unsigned __int128 x)
+{
+  return __builtin_ctzg (x, __SIZEOF_INT128__ * __CHAR_BIT__);
+}
+#endif
+
+__attribute__((noipa)) int
+clrsbc (signed char x)
+{
+  return __builtin_clrsbg (x);
+}
+
+__attribute__((noipa)) int
+clrsbs (signed short x)
+{
+  return __builtin_clrsbg (x);
+}
+
+__attribute__((noipa)) int
+clrsbi (signed int x)
+{
+  return __builtin_clrsbg (x);
+}
+
+__attribute__((noipa)) int
+clrsbl (signed long x)
+{
+  return __builtin_clrsbg (x);
+}
+
+__attribute__((noipa)) int
+clrsbL (signed long long x)
+{
+  return __builtin_clrsbg (x);
+}
+
+#ifdef __SIZEOF_INT128__
+__attribute__((noipa)) int
+clrsbI (signed __int128 x)
+{
+  return __builtin_clrsbg (x);
+}
+#endif
+
+__attribute__((noipa)) int
+ffsc (signed char x)
+{
+  return __builtin_ffsg (x);
+}
+
+__attribute__((noipa)) int
+ffss (signed short x)
+{
+  return __builtin_ffsg (x);
+}
+
+__attribute__((noipa)) int
+ffsi (signed int x)
+{
+  return __builtin_ffsg (x);
+}
+
+__attribute__((noipa)) int
+ffsl (signed long x)
+{
+  return __builtin_ffsg (x);
+}
+
+__attribute__((noipa)) int
+ffsL (signed long long x)
+{
+  return __builtin_ffsg (x);
+}
+
+#ifdef __SIZEOF_INT128__
+__attribute__((noipa)) int
+ffsI (signed __int128 x)
+{
+  return __builtin_ffsg (x);
+}
+#endif
+
+__attribute__((noipa)) int
+parityc (unsigned char x)
+{
+  return __builtin_parityg (x);
+}
+
+__attribute__((noipa)) int
+paritys (unsigned short x)
+{
+  return __builtin_parityg (x);
+}
+
+__attribute__((noipa)) int
+parityi (unsigned int x)
+{
+  return __builtin_parityg (x);
+}
+
+__attribute__((noipa)) int
+parityl (unsigned long x)
+{
+  return __builtin_parityg (x);
+}
+
+__attribute__((noipa)) int
+parityL (unsigned long long x)
+{
+  return __builtin_parityg (x);
+}
+
+#ifdef __SIZEOF_INT128__
+__attribute__((noipa)) int
+parityI (unsigned __int128 x)
+{
+  return __builtin_parityg (x);
+}
+#endif
+
+__attribute__((noipa)) int
+popcountc (unsigned char x)
+{
+  return __builtin_popcountg (x);
+}
+
+__attribute__((noipa)) int
+popcounts (unsigned short x)
+{
+  return __builtin_popcountg (x);
+}
+
+__attribute__((noipa)) int
+popcounti (unsigned int x)
+{
+  return __builtin_popcountg (x);
+}
+
+__attribute__((noipa)) int
+popcountl (unsigned long x)
+{
+  return __builtin_popcountg (x);
+}
+
+__attribute__((noipa)) int
+popcountL (unsigned long long x)
+{
+  return __builtin_popcountg (x);
+}
+
+#ifdef __SIZEOF_INT128__
+__attribute__((noipa)) int
+popcountI (unsigned __int128 x)
+{
+  return __builtin_popcountg (x);
+}
+#endif
+
+int
+main ()
+{
+  if (__builtin_clzg ((unsigned char) 1) != __CHAR_BIT__ - 1
+      || __builtin_clzg ((unsigned short) 2, -2) != __SIZEOF_SHORT__ * __CHAR_BIT__ - 2
+      || __builtin_clzg (0U, 42) != 42
+      || __builtin_clzg (0U, -1) != -1
+      || __builtin_clzg (1U) != __SIZEOF_INT__ * __CHAR_BIT__ - 1
+      || __builtin_clzg (2UL, -1) != __SIZEOF_LONG__ * __CHAR_BIT__ - 2
+      || __builtin_clzg (5ULL) != __SIZEOF_LONG_LONG__ * __CHAR_BIT__ - 3
+#ifdef __SIZEOF_INT128__
+      || __builtin_clzg ((unsigned __int128) 9) != __SIZEOF_INT128__ * __CHAR_BIT__ - 4
+#endif
+      || __builtin_clzg (~0U, -5) != 0
+      || __builtin_clzg (~0ULL >> 2) != 2
+      || __builtin_ctzg ((unsigned char) 1) != 0
+      || __builtin_ctzg ((unsigned short) 28) != 2
+      || __builtin_ctzg (0U, 32) != 32
+      || __builtin_ctzg (0U, -42) != -42
+      || __builtin_ctzg (1U) != 0
+      || __builtin_ctzg (16UL, -1) != 4
+      || __builtin_ctzg (5ULL << 52, 0) != 52
+#ifdef __SIZEOF_INT128__
+      || __builtin_ctzg (((unsigned __int128) 9) << 72) != 72
+#endif
+      || __builtin_clrsbg ((signed char) 0) != __CHAR_BIT__ - 1
+      || __builtin_clrsbg ((signed short) -1) != __SIZEOF_SHORT__ * __CHAR_BIT__ - 1
+      || __builtin_clrsbg (0) != __SIZEOF_INT__ * __CHAR_BIT__ - 1
+      || __builtin_clrsbg (-1L) != __SIZEOF_LONG__ * __CHAR_BIT__ - 1
+      || __builtin_clrsbg (0LL) != __SIZEOF_LONG_LONG__ * __CHAR_BIT__ - 1
+#ifdef __SIZEOF_INT128__
+      || __builtin_clrsbg ((__int128) -1) != __SIZEOF_INT128__ * __CHAR_BIT__ - 1
+#endif
+      || __builtin_clrsbg (0x1afb) != __SIZEOF_INT__ * __CHAR_BIT__ - 14
+      || __builtin_clrsbg (-2) != __SIZEOF_INT__ * __CHAR_BIT__ - 2
+      || __builtin_clrsbg (1L) != __SIZEOF_LONG__ * __CHAR_BIT__ - 2
+      || __builtin_clrsbg (-4LL) != __SIZEOF_LONG_LONG__ * __CHAR_BIT__ - 3
+      || __builtin_ffsg ((signed char) 0) != 0
+      || __builtin_ffsg ((signed short) 0) != 0
+      || __builtin_ffsg (0) != 0
+      || __builtin_ffsg (0L) != 0
+      || __builtin_ffsg (0LL) != 0
+#ifdef __SIZEOF_INT128__
+      || __builtin_ffsg ((__int128) 0) != 0
+#endif
+      || __builtin_ffsg ((signed char) 4) != 3
+      || __builtin_ffsg ((signed short) 8) != 4
+      || __builtin_ffsg (1) != 1
+      || __builtin_ffsg (2L) != 2
+      || __builtin_ffsg (28LL) != 3
+      || __builtin_parityg ((unsigned char) 1) != 1
+      || __builtin_parityg ((unsigned short) 2) != 1
+      || __builtin_parityg (0U) != 0
+      || __builtin_parityg (3U) != 0
+      || __builtin_parityg (0UL) != 0
+      || __builtin_parityg (7UL) != 1
+      || __builtin_parityg (0ULL) != 0
+#ifdef __SIZEOF_INT128__
+      || __builtin_parityg ((unsigned __int128) 0) != 0
+#endif
+      || __builtin_parityg ((unsigned char) ~0U) != 0
+      || __builtin_parityg ((unsigned short) ~0U) != 0
+      || __builtin_parityg (~0U) != 0
+      || __builtin_parityg (~0UL) != 0
+      || __builtin_parityg (~0ULL) != 0
+#ifdef __SIZEOF_INT128__
+      || __builtin_parityg (~(unsigned __int128) 0) != 0
+#endif
+      || __builtin_popcountg (0U) != 0
+      || __builtin_popcountg (0UL) != 0
+      || __builtin_popcountg (0ULL) != 0
+#ifdef __SIZEOF_INT128__
+      || __builtin_popcountg ((unsigned __int128) 0) != 0
+#endif
+      || __builtin_popcountg ((unsigned char) ~0U) != __CHAR_BIT__
+      || __builtin_popcountg ((unsigned short) ~0U) != __SIZEOF_SHORT__ * __CHAR_BIT__
+      || __builtin_popcountg (~0U) != __SIZEOF_INT__ * __CHAR_BIT__
+      || __builtin_popcountg (~0UL) != __SIZEOF_LONG__ * __CHAR_BIT__
+      || __builtin_popcountg (~0ULL) != __SIZEOF_LONG_LONG__ * __CHAR_BIT__
+#ifdef __SIZEOF_INT128__
+      || __builtin_popcountg (~(unsigned __int128) 0) != __SIZEOF_INT128__ * __CHAR_BIT__
+#endif
+      || 0)
+  __builtin_abort ();
+  if (clzc (1) != __CHAR_BIT__ - 1
+      || clzs2 (2) != __SIZEOF_SHORT__ * __CHAR_BIT__ - 2
+      || clzi2 (0U, 42) != 42
+      || clzi2 (0U, -1) != -1
+      || clzi (1U) != __SIZEOF_INT__ * __CHAR_BIT__ - 1
+      || clzl2 (2UL) != __SIZEOF_LONG__ * __CHAR_BIT__ - 2
+      || clzL (5ULL) != __SIZEOF_LONG_LONG__ * __CHAR_BIT__ - 3
+#ifdef __SIZEOF_INT128__
+      || clzI ((unsigned __int128) 9) != __SIZEOF_INT128__ * __CHAR_BIT__ - 4
+#endif
+      || clzi2 (~0U, -5) != 0
+      || clzL (~0ULL >> 2) != 2
+      || ctzc (1) != 0
+      || ctzs (28) != 2
+      || ctzi2 (0U, 32) != 32
+      || ctzi2 (0U, -42) != -42
+      || ctzi (1U) != 0
+      || ctzl2 (16UL, -1) != 4
+      || ctzL2 (5ULL << 52, 0) != 52
+#ifdef __SIZEOF_INT128__
+      || ctzI (((unsigned __int128) 9) << 72) != 72
+#endif
+      || clrsbc (0) != __CHAR_BIT__ - 1
+      || clrsbs (-1) != __SIZEOF_SHORT__ * __CHAR_BIT__ - 1
+      || clrsbi (0) != __SIZEOF_INT__ * __CHAR_BIT__ - 1
+      || clrsbl (-1L) != __SIZEOF_LONG__ * __CHAR_BIT__ - 1
+      || clrsbL (0LL) != __SIZEOF_LONG_LONG__ * __CHAR_BIT__ - 1
+#ifdef __SIZEOF_INT128__
+      || clrsbI (-1) != __SIZEOF_INT128__ * __CHAR_BIT__ - 1
+#endif
+      || clrsbi (0x1afb) != __SIZEOF_INT__ * __CHAR_BIT__ - 14
+      || clrsbi (-2) != __SIZEOF_INT__ * __CHAR_BIT__ - 2
+      || clrsbl (1L) != __SIZEOF_LONG__ * __CHAR_BIT__ - 2
+      || clrsbL (-4LL) != __SIZEOF_LONG_LONG__ * __CHAR_BIT__ - 3
+      || ffsc (0) != 0
+      || ffss (0) != 0
+      || ffsi (0) != 0
+      || ffsl (0L) != 0
+      || ffsL (0LL) != 0
+#ifdef __SIZEOF_INT128__
+      || ffsI (0) != 0
+#endif
+      || ffsc (4) != 3
+      || ffss (8) != 4
+      || ffsi (1) != 1
+      || ffsl (2L) != 2
+      || ffsL (28LL) != 3
+      || parityc (1) != 1
+      || paritys (2) != 1
+      || parityi (0U) != 0
+      || parityi (3U) != 0
+      || parityl (0UL) != 0
+      || parityl (7UL) != 1
+      || parityL (0ULL) != 0
+#ifdef __SIZEOF_INT128__
+      || parityI (0) != 0
+#endif
+      || parityc ((unsigned char) ~0U) != 0
+      || paritys ((unsigned short) ~0U) != 0
+      || parityi (~0U) != 0
+      || parityl (~0UL) != 0
+      || parityL (~0ULL) != 0
+#ifdef __SIZEOF_INT128__
+      || parityI (~(unsigned __int128) 0) != 0
+#endif
+      || popcounti (0U) != 0
+      || popcountl (0UL) != 0
+      || popcountL (0ULL) != 0
+#ifdef __SIZEOF_INT128__
+      || popcountI (0) != 0
+#endif
+      || popcountc ((unsigned char) ~0U) != __CHAR_BIT__
+      || popcounts ((unsigned short) ~0U) != __SIZEOF_SHORT__ * __CHAR_BIT__
+      || popcounti (~0U) != __SIZEOF_INT__ * __CHAR_BIT__
+      || popcountl (~0UL) != __SIZEOF_LONG__ * __CHAR_BIT__
+      || popcountL (~0ULL) != __SIZEOF_LONG_LONG__ * __CHAR_BIT__
+#ifdef __SIZEOF_INT128__
+      || popcountI (~(unsigned __int128) 0) != __SIZEOF_INT128__ * __CHAR_BIT__
+#endif
+      || 0)
+  __builtin_abort ();
+}
diff --git a/gcc/testsuite/c-c++-common/pr111309-2.c b/gcc/testsuite/c-c++-common/pr111309-2.c
new file mode 100644
index 0000000..535208b
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/pr111309-2.c
@@ -0,0 +1,85 @@
+/* PR c/111309 */
+/* { dg-do compile } */
+/* { dg-additional-options "-std=c99" { target c } } */
+
+#ifndef __cplusplus
+#define bool _Bool
+#define true ((_Bool) 1)
+#define false ((_Bool) 0)
+#endif
+
+void
+foo (void)
+{
+  enum E { E0 = 0 };
+  struct S { int s; } s;
+  __builtin_clzg ();		/* { dg-error "too few arguments" } */
+  __builtin_clzg (0U, 1, 2);	/* { dg-error "too many arguments" } */
+  __builtin_clzg (0);		/* { dg-error "has signed type" } */
+  __builtin_clzg (0.0);		/* { dg-error "does not have integral type" } */
+  __builtin_clzg (s);		/* { dg-error "does not have integral type" } */
+  __builtin_clzg (true);	/* { dg-error "has boolean type" } */
+  __builtin_clzg (E0);		/* { dg-error "has signed type" "" { target c } } */
+				/* { dg-error "has enumerated type" "" { target c++ } .-1 } */
+  __builtin_clzg (0, 0);	/* { dg-error "has signed type" } */
+  __builtin_clzg (0.0, 0);	/* { dg-error "does not have integral type" } */
+  __builtin_clzg (s, 0);	/* { dg-error "does not have integral type" } */
+  __builtin_clzg (true, 0);	/* { dg-error "has boolean type" } */
+  __builtin_clzg (E0, 0);	/* { dg-error "has signed type" "" { target c } } */
+				/* { dg-error "has enumerated type" "" { target c++ } .-1 } */
+  __builtin_clzg (0U, 2.0);	/* { dg-error "does not have integral type" } */
+  __builtin_clzg (0U, s);	/* { dg-error "does not have integral type" } */
+  __builtin_clzg (0U, 2LL);	/* { dg-error "does not have 'int' type" } */
+  __builtin_clzg (0U, 2U);	/* { dg-error "does not have 'int' type" } */
+  __builtin_clzg (0U, true);
+  __builtin_clzg (0U, E0);	/* { dg-error "does not have 'int' type" "" { target c++ } } */
+  __builtin_ctzg ();		/* { dg-error "too few arguments" } */
+  __builtin_ctzg (0U, 1, 2);	/* { dg-error "too many arguments" } */
+  __builtin_ctzg (0);		/* { dg-error "has signed type" } */
+  __builtin_ctzg (0.0);		/* { dg-error "does not have integral type" } */
+  __builtin_ctzg (s);		/* { dg-error "does not have integral type" } */
+  __builtin_ctzg (true);	/* { dg-error "has boolean type" } */
+  __builtin_ctzg (E0);		/* { dg-error "has signed type" "" { target c } } */
+				/* { dg-error "has enumerated type" "" { target c++ } .-1 } */
+  __builtin_ctzg (0, 0);	/* { dg-error "has signed type" } */
+  __builtin_ctzg (0.0, 0);	/* { dg-error "does not have integral type" } */
+  __builtin_ctzg (s, 0);	/* { dg-error "does not have integral type" } */
+  __builtin_ctzg (true, 0);	/* { dg-error "has boolean type" } */
+  __builtin_ctzg (E0, 0);	/* { dg-error "has signed type" "" { target c } } */
+				/* { dg-error "has enumerated type" "" { target c++ } .-1 } */
+  __builtin_ctzg (0U, 2.0);	/* { dg-error "does not have integral type" } */
+  __builtin_ctzg (0U, 2LL);	/* { dg-error "does not have 'int' type" } */
+  __builtin_ctzg (0U, 2U);	/* { dg-error "does not have 'int' type" } */
+  __builtin_ctzg (0U, true);
+  __builtin_ctzg (0U, E0);	/* { dg-error "does not have 'int' type" "" { target c++ } } */
+  __builtin_clrsbg ();		/* { dg-error "too few arguments" } */
+  __builtin_clrsbg (0, 1);	/* { dg-error "too many arguments" } */
+  __builtin_clrsbg (0U);	/* { dg-error "has unsigned type" } */
+  __builtin_clrsbg (0.0);	/* { dg-error "does not have integral type" } */
+  __builtin_clrsbg (s);		/* { dg-error "does not have integral type" } */
+  __builtin_clrsbg (true);	/* { dg-error "has boolean type" } */
+  __builtin_clrsbg (E0);	/* { dg-error "has enumerated type" "" { target c++ } } */
+  __builtin_ffsg ();		/* { dg-error "too few arguments" } */
+  __builtin_ffsg (0, 1);	/* { dg-error "too many arguments" } */
+  __builtin_ffsg (0U);		/* { dg-error "has unsigned type" } */
+  __builtin_ffsg (0.0);		/* { dg-error "does not have integral type" } */
+  __builtin_ffsg (s);		/* { dg-error "does not have integral type" } */
+  __builtin_ffsg (true);	/* { dg-error "has boolean type" } */
+  __builtin_ffsg (E0);		/* { dg-error "has enumerated type" "" { target c++ } } */
+  __builtin_parityg ();		/* { dg-error "too few arguments" } */
+  __builtin_parityg (0U, 1);	/* { dg-error "too many arguments" } */
+  __builtin_parityg (0);	/* { dg-error "has signed type" } */
+  __builtin_parityg (0.0);	/* { dg-error "does not have integral type" } */
+  __builtin_parityg (s);	/* { dg-error "does not have integral type" } */
+  __builtin_parityg (true);	/* { dg-error "has boolean type" } */
+  __builtin_parityg (E0);	/* { dg-error "has signed type" "" { target c } } */
+				/* { dg-error "has enumerated type" "" { target c++ } .-1 } */
+  __builtin_popcountg ();	/* { dg-error "too few arguments" } */
+  __builtin_popcountg (0U, 1);	/* { dg-error "too many arguments" } */
+  __builtin_popcountg (0);	/* { dg-error "has signed type" } */
+  __builtin_popcountg (0.0);	/* { dg-error "does not have integral type" } */
+  __builtin_popcountg (s);	/* { dg-error "does not have integral type" } */
+  __builtin_popcountg (true);	/* { dg-error "has boolean type" } */
+  __builtin_popcountg (E0);	/* { dg-error "has signed type" "" { target c } } */
+				/* { dg-error "has enumerated type" "" { target c++ } .-1 } */
+}
diff --git a/gcc/testsuite/gcc.dg/torture/bitint-43.c b/gcc/testsuite/gcc.dg/torture/bitint-43.c
new file mode 100644
index 0000000..4265bff
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/bitint-43.c
@@ -0,0 +1,306 @@
+/* PR c/111309 */
+/* { dg-do run { target bitint } } */
+/* { dg-options "-std=c2x -pedantic-errors" } */
+/* { dg-skip-if "" { ! run_expensive_tests }  { "*" } { "-O0" "-O2" } } */
+/* { dg-skip-if "" { ! run_expensive_tests } { "-flto" } { "" } } */
+
+#if __BITINT_MAXWIDTH__ >= 156
+__attribute__((noipa)) int
+clz156 (unsigned _BitInt(156) x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzd156 (unsigned _BitInt(156) x)
+{
+  return __builtin_clzg (x, __builtin_popcountg ((typeof (x)) ~(typeof (x)) 0));
+}
+
+__attribute__((noipa)) int
+clzD156 (unsigned _BitInt(156) x, int y)
+{
+  return __builtin_clzg (x, y);
+}
+
+__attribute__((noipa)) int
+ctz156 (unsigned _BitInt(156) x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzd156 (unsigned _BitInt(156) x)
+{
+  return __builtin_ctzg (x, __builtin_popcountg ((typeof (x)) ~(typeof (x)) 0));
+}
+
+__attribute__((noipa)) int
+ctzD156 (unsigned _BitInt(156) x, int y)
+{
+  return __builtin_ctzg (x, y);
+}
+
+__attribute__((noipa)) int
+clrsb156 (_BitInt(156) x)
+{
+  return __builtin_clrsbg (x);
+}
+
+__attribute__((noipa)) int
+ffs156 (_BitInt(156) x)
+{
+  return __builtin_ffsg (x);
+}
+
+__attribute__((noipa)) int
+parity156 (unsigned _BitInt(156) x)
+{
+  return __builtin_parityg (x);
+}
+
+__attribute__((noipa)) int
+popcount156 (unsigned _BitInt(156) x)
+{
+  return __builtin_popcountg (x);
+}
+#endif
+
+#if __BITINT_MAXWIDTH__ >= 192
+__attribute__((noipa)) int
+clz192 (unsigned _BitInt(192) x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzd192 (unsigned _BitInt(192) x)
+{
+  return __builtin_clzg (x, __builtin_popcountg ((typeof (x)) ~(typeof (x)) 0));
+}
+
+__attribute__((noipa)) int
+clzD192 (unsigned _BitInt(192) x, int y)
+{
+  return __builtin_clzg (x, y);
+}
+
+__attribute__((noipa)) int
+ctz192 (unsigned _BitInt(192) x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzd192 (unsigned _BitInt(192) x)
+{
+  return __builtin_ctzg (x, __builtin_popcountg ((typeof (x)) ~(typeof (x)) 0));
+}
+
+__attribute__((noipa)) int
+ctzD192 (unsigned _BitInt(192) x, int y)
+{
+  return __builtin_ctzg (x, y);
+}
+
+__attribute__((noipa)) int
+clrsb192 (_BitInt(192) x)
+{
+  return __builtin_clrsbg (x);
+}
+
+__attribute__((noipa)) int
+ffs192 (_BitInt(192) x)
+{
+  return __builtin_ffsg (x);
+}
+
+__attribute__((noipa)) int
+parity192 (unsigned _BitInt(192) x)
+{
+  return __builtin_parityg (x);
+}
+
+__attribute__((noipa)) int
+popcount192 (unsigned _BitInt(192) x)
+{
+  return __builtin_popcountg (x);
+}
+#endif
+
+int
+main ()
+{
+#if __BITINT_MAXWIDTH__ >= 156
+  if (clzd156 (0) != 156
+      || clzD156 (0, -1) != -1
+      || ctzd156 (0) != 156
+      || ctzD156 (0, 42) != 42
+      || clrsb156 (0) != 156 - 1
+      || ffs156 (0) != 0
+      || parity156 (0) != 0
+      || popcount156 (0) != 0
+      || __builtin_clzg ((unsigned _BitInt(156)) 0, 156 + 32) != 156 + 32
+      || __builtin_ctzg ((unsigned _BitInt(156)) 0, 156) != 156
+      || __builtin_clrsbg ((_BitInt(156)) 0) != 156 - 1
+      || __builtin_ffsg ((_BitInt(156)) 0) != 0
+      || __builtin_parityg ((unsigned _BitInt(156)) 0) != 0
+      || __builtin_popcountg ((unsigned _BitInt(156)) 0) != 0)
+    __builtin_abort ();
+  if (clz156 (-1) != 0
+      || clzd156 (-1) != 0
+      || clzD156 (-1, 0) != 0
+      || ctz156 (-1) != 0
+      || ctzd156 (-1) != 0
+      || ctzD156 (-1, 17) != 0
+      || clrsb156 (-1) != 156 - 1
+      || ffs156 (-1) != 1
+      || parity156 (-1) != 0
+      || popcount156 (-1) != 156
+      || __builtin_clzg ((unsigned _BitInt(156)) -1) != 0
+      || __builtin_clzg ((unsigned _BitInt(156)) -1, 156 + 32) != 0
+      || __builtin_ctzg ((unsigned _BitInt(156)) -1) != 0
+      || __builtin_ctzg ((unsigned _BitInt(156)) -1, 156) != 0
+      || __builtin_clrsbg ((_BitInt(156)) -1) != 156 - 1
+      || __builtin_ffsg ((_BitInt(156)) -1) != 1
+      || __builtin_parityg ((unsigned _BitInt(156)) -1) != 0
+      || __builtin_popcountg ((unsigned _BitInt(156)) -1) != 156)
+    __builtin_abort ();
+  if (clz156 (((unsigned _BitInt(156)) -1) >> 24) != 24
+      || clz156 (((unsigned _BitInt(156)) -1) >> 79) != 79
+      || clz156 (1) != 156 - 1
+      || clzd156 (((unsigned _BitInt(156)) -1) >> 139) != 139
+      || clzd156 (2) != 156 - 2
+      || ctz156 (((unsigned _BitInt(156)) -1) << 42) != 42
+      || ctz156 (((unsigned _BitInt(156)) -1) << 57) != 57
+      || ctz156 (0x4000000000000000000000uwb) != 86
+      || ctzd156 (((unsigned _BitInt(156)) -1) << 149) != 149
+      || ctzd156 (2) != 1
+      || clrsb156 ((unsigned _BitInt(156 - 4)) -1) != 3
+      || clrsb156 ((unsigned _BitInt(156 - 28)) -1) != 27
+      || clrsb156 ((unsigned _BitInt(156 - 29)) -1) != 28
+      || clrsb156 (~(unsigned _BitInt(156)) (unsigned _BitInt(156 - 68)) -1) != 67
+      || clrsb156 (~(unsigned _BitInt(156)) (unsigned _BitInt(156 - 92)) -1) != 91
+      || clrsb156 (~(unsigned _BitInt(156)) (unsigned _BitInt(156 - 93)) -1) != 92
+      || ffs156 (((unsigned _BitInt(156)) -1) << 42) != 43
+      || ffs156 (((unsigned _BitInt(156)) -1) << 57) != 58
+      || ffs156 (0x4000000000000000000000uwb) != 87
+      || ffs156 (((unsigned _BitInt(156)) -1) << 149) != 150
+      || ffs156 (2) != 2
+      || __builtin_clzg (((unsigned _BitInt(156)) -1) >> 24) != 24
+      || __builtin_clzg (((unsigned _BitInt(156)) -1) >> 79) != 79
+      || __builtin_clzg ((unsigned _BitInt(156)) 1) != 156 - 1
+      || __builtin_clzg (((unsigned _BitInt(156)) -1) >> 139, 156) != 139
+      || __builtin_clzg ((unsigned _BitInt(156)) 2, 156) != 156 - 2
+      || __builtin_ctzg (((unsigned _BitInt(156)) -1) << 42) != 42
+      || __builtin_ctzg (((unsigned _BitInt(156)) -1) << 57) != 57
+      || __builtin_ctzg ((unsigned _BitInt(156)) 0x4000000000000000000000uwb) != 86
+      || __builtin_ctzg (((unsigned _BitInt(156)) -1) << 149, 156) != 149
+      || __builtin_ctzg ((unsigned _BitInt(156)) 2, 156) != 1
+      || __builtin_clrsbg ((_BitInt(156)) (unsigned _BitInt(156 - 4)) -1) != 3
+      || __builtin_clrsbg ((_BitInt(156)) (unsigned _BitInt(156 - 28)) -1) != 27
+      || __builtin_clrsbg ((_BitInt(156)) (unsigned _BitInt(156 - 29)) -1) != 28
+      || __builtin_clrsbg ((_BitInt(156)) ~(unsigned _BitInt(156)) (unsigned _BitInt(156 - 68)) -1) != 67
+      || __builtin_clrsbg ((_BitInt(156)) ~(unsigned _BitInt(156)) (unsigned _BitInt(156 - 92)) -1) != 91
+      || __builtin_clrsbg ((_BitInt(156)) ~(unsigned _BitInt(156)) (unsigned _BitInt(156 - 93)) -1) != 92
+      || __builtin_ffsg ((_BitInt(156)) (((unsigned _BitInt(156)) -1) << 42)) != 43
+      || __builtin_ffsg ((_BitInt(156)) (((unsigned _BitInt(156)) -1) << 57)) != 58
+      || __builtin_ffsg ((_BitInt(156)) 0x4000000000000000000000uwb) != 87
+      || __builtin_ffsg ((_BitInt(156)) (((unsigned _BitInt(156)) -1) << 149)) != 150
+      || __builtin_ffsg ((_BitInt(156)) 2) != 2)
+    __builtin_abort ();
+  if (parity156 (23008250258685373142923325827291949461178444434uwb) != __builtin_parityg (23008250258685373142923325827291949461178444434uwb)
+      || parity156 (41771568792516301628132437740665810252917251244uwb) != __builtin_parityg (41771568792516301628132437740665810252917251244uwb)
+      || parity156 (5107402473866766219120283991834936835726115452uwb) != __builtin_parityg (5107402473866766219120283991834936835726115452uwb)
+      || popcount156 (50353291748276374580944955711958129678996395562uwb) != __builtin_popcountg (50353291748276374580944955711958129678996395562uwb)
+      || popcount156 (29091263616891212550063067166307725491211684496uwb) != __builtin_popcountg (29091263616891212550063067166307725491211684496uwb)
+      || popcount156 (64973284306583205619384799873110935608793072026uwb) != __builtin_popcountg (64973284306583205619384799873110935608793072026uwb))
+    __builtin_abort ();
+#endif
+#if __BITINT_MAXWIDTH__ >= 192
+  if (clzd192 (0) != 192
+      || clzD192 (0, 42) != 42
+      || ctzd192 (0) != 192
+      || ctzD192 (0, -1) != -1
+      || clrsb192 (0) != 192 - 1
+      || ffs192 (0) != 0
+      || parity192 (0) != 0
+      || popcount192 (0) != 0
+      || __builtin_clzg ((unsigned _BitInt(192)) 0, 192 + 32) != 192 + 32
+      || __builtin_ctzg ((unsigned _BitInt(192)) 0, 192) != 192
+      || __builtin_clrsbg ((_BitInt(192)) 0) != 192 - 1
+      || __builtin_ffsg ((_BitInt(192)) 0) != 0
+      || __builtin_parityg ((unsigned _BitInt(192)) 0) != 0
+      || __builtin_popcountg ((unsigned _BitInt(192)) 0) != 0)
+    __builtin_abort ();
+  if (clz192 (-1) != 0
+      || clzd192 (-1) != 0
+      || clzD192 (-1, 15) != 0
+      || ctz192 (-1) != 0
+      || ctzd192 (-1) != 0
+      || ctzD192 (-1, -57) != 0
+      || clrsb192 (-1) != 192 - 1
+      || ffs192 (-1) != 1
+      || parity192 (-1) != 0
+      || popcount192 (-1) != 192
+      || __builtin_clzg ((unsigned _BitInt(192)) -1) != 0
+      || __builtin_clzg ((unsigned _BitInt(192)) -1, 192 + 32) != 0
+      || __builtin_ctzg ((unsigned _BitInt(192)) -1) != 0
+      || __builtin_ctzg ((unsigned _BitInt(192)) -1, 192) != 0
+      || __builtin_clrsbg ((_BitInt(192)) -1) != 192 - 1
+      || __builtin_ffsg ((_BitInt(192)) -1) != 1
+      || __builtin_parityg ((unsigned _BitInt(192)) -1) != 0
+      || __builtin_popcountg ((unsigned _BitInt(192)) -1) != 192)
+    __builtin_abort ();
+  if (clz192 (((unsigned _BitInt(192)) -1) >> 24) != 24
+      || clz192 (((unsigned _BitInt(192)) -1) >> 79) != 79
+      || clz192 (1) != 192 - 1
+      || clzd192 (((unsigned _BitInt(192)) -1) >> 139) != 139
+      || clzd192 (2) != 192 - 2
+      || ctz192 (((unsigned _BitInt(192)) -1) << 42) != 42
+      || ctz192 (((unsigned _BitInt(192)) -1) << 57) != 57
+      || ctz192 (0x4000000000000000000000uwb) != 86
+      || ctzd192 (((unsigned _BitInt(192)) -1) << 149) != 149
+      || ctzd192 (2) != 1
+      || clrsb192 ((unsigned _BitInt(192 - 4)) -1) != 3
+      || clrsb192 ((unsigned _BitInt(192 - 28)) -1) != 27
+      || clrsb192 ((unsigned _BitInt(192 - 29)) -1) != 28
+      || clrsb192 (~(unsigned _BitInt(192)) (unsigned _BitInt(192 - 68)) -1) != 67
+      || clrsb192 (~(unsigned _BitInt(192)) (unsigned _BitInt(192 - 92)) -1) != 91
+      || clrsb192 (~(unsigned _BitInt(192)) (unsigned _BitInt(192 - 93)) -1) != 92
+      || ffs192 (((unsigned _BitInt(192)) -1) << 42) != 43
+      || ffs192 (((unsigned _BitInt(192)) -1) << 57) != 58
+      || ffs192 (0x4000000000000000000000uwb) != 87
+      || ffs192 (((unsigned _BitInt(192)) -1) << 149) != 150
+      || ffs192 (2) != 2
+      || __builtin_clzg (((unsigned _BitInt(192)) -1) >> 24) != 24
+      || __builtin_clzg (((unsigned _BitInt(192)) -1) >> 79) != 79
+      || __builtin_clzg ((unsigned _BitInt(192)) 1) != 192 - 1
+      || __builtin_clzg (((unsigned _BitInt(192)) -1) >> 139, 192) != 139
+      || __builtin_clzg ((unsigned _BitInt(192)) 2, 192) != 192 - 2
+      || __builtin_ctzg (((unsigned _BitInt(192)) -1) << 42) != 42
+      || __builtin_ctzg (((unsigned _BitInt(192)) -1) << 57) != 57
+      || __builtin_ctzg ((unsigned _BitInt(192)) 0x4000000000000000000000uwb) != 86
+      || __builtin_ctzg (((unsigned _BitInt(192)) -1) << 149, 192) != 149
+      || __builtin_ctzg ((unsigned _BitInt(192)) 2, 192) != 1
+      || __builtin_clrsbg ((_BitInt(192)) (unsigned _BitInt(192 - 4)) -1) != 3
+      || __builtin_clrsbg ((_BitInt(192)) (unsigned _BitInt(192 - 28)) -1) != 27
+      || __builtin_clrsbg ((_BitInt(192)) (unsigned _BitInt(192 - 29)) -1) != 28
+      || __builtin_clrsbg ((_BitInt(192)) ~(unsigned _BitInt(192)) (unsigned _BitInt(192 - 68)) -1) != 67
+      || __builtin_clrsbg ((_BitInt(192)) ~(unsigned _BitInt(192)) (unsigned _BitInt(192 - 92)) -1) != 91
+      || __builtin_clrsbg ((_BitInt(192)) ~(unsigned _BitInt(192)) (unsigned _BitInt(192 - 93)) -1) != 92
+      || __builtin_ffsg ((_BitInt(192)) (((unsigned _BitInt(192)) -1) << 42)) != 43
+      || __builtin_ffsg ((_BitInt(192)) (((unsigned _BitInt(192)) -1) << 57)) != 58
+      || __builtin_ffsg ((_BitInt(192)) 0x4000000000000000000000uwb) != 87
+      || __builtin_ffsg ((_BitInt(192)) (((unsigned _BitInt(192)) -1) << 149)) != 150
+      || __builtin_ffsg ((_BitInt(192)) 2) != 2)
+    __builtin_abort ();
+  if (parity192 (4692147078159863499615754634965484598760535154638668598762uwb) != __builtin_parityg (4692147078159863499615754634965484598760535154638668598762uwb)
+      || parity192 (1669461228546917627909935444501097256112222796898845183538uwb) != __builtin_parityg (1669461228546917627909935444501097256112222796898845183538uwb)
+      || parity192 (5107402473866766219120283991834936835726115452uwb) != __builtin_parityg (5107402473866766219120283991834936835726115452uwb)
+      || popcount192 (4033871057575185619108386380181511734118888391160164588976uwb) != __builtin_popcountg (4033871057575185619108386380181511734118888391160164588976uwb)
+      || popcount192 (58124766715713711628758119849579188845074973856704521119uwb) != __builtin_popcountg (58124766715713711628758119849579188845074973856704521119uwb)
+      || popcount192 (289948065236269174335700831610076764076947650072787325852uwb) != __builtin_popcountg (289948065236269174335700831610076764076947650072787325852uwb))
+    __builtin_abort ();
+#endif
+}
diff --git a/gcc/testsuite/gcc.dg/torture/bitint-44.c b/gcc/testsuite/gcc.dg/torture/bitint-44.c
new file mode 100644
index 0000000..938c0e9
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/bitint-44.c
@@ -0,0 +1,306 @@
+/* PR c/111309 */
+/* { dg-do run { target bitint } } */
+/* { dg-options "-std=c2x -pedantic-errors" } */
+/* { dg-skip-if "" { ! run_expensive_tests }  { "*" } { "-O0" "-O2" } } */
+/* { dg-skip-if "" { ! run_expensive_tests } { "-flto" } { "" } } */
+
+#if __BITINT_MAXWIDTH__ >= 512
+__attribute__((noipa)) int
+clz512 (unsigned _BitInt(512) x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzd512 (unsigned _BitInt(512) x)
+{
+  return __builtin_clzg (x, __builtin_popcountg ((typeof (x)) ~(typeof (x)) 0));
+}
+
+__attribute__((noipa)) int
+clzD512 (unsigned _BitInt(512) x, int y)
+{
+  return __builtin_clzg (x, y);
+}
+
+__attribute__((noipa)) int
+ctz512 (unsigned _BitInt(512) x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzd512 (unsigned _BitInt(512) x)
+{
+  return __builtin_ctzg (x, __builtin_popcountg ((typeof (x)) ~(typeof (x)) 0));
+}
+
+__attribute__((noipa)) int
+ctzD512 (unsigned _BitInt(512) x, int y)
+{
+  return __builtin_ctzg (x, y);
+}
+
+__attribute__((noipa)) int
+clrsb512 (_BitInt(512) x)
+{
+  return __builtin_clrsbg (x);
+}
+
+__attribute__((noipa)) int
+ffs512 (_BitInt(512) x)
+{
+  return __builtin_ffsg (x);
+}
+
+__attribute__((noipa)) int
+parity512 (unsigned _BitInt(512) x)
+{
+  return __builtin_parityg (x);
+}
+
+__attribute__((noipa)) int
+popcount512 (unsigned _BitInt(512) x)
+{
+  return __builtin_popcountg (x);
+}
+#endif
+
+#if __BITINT_MAXWIDTH__ >= 523
+__attribute__((noipa)) int
+clz523 (unsigned _BitInt(523) x)
+{
+  return __builtin_clzg (x);
+}
+
+__attribute__((noipa)) int
+clzd523 (unsigned _BitInt(523) x)
+{
+  return __builtin_clzg (x, __builtin_popcountg ((typeof (x)) ~(typeof (x)) 0));
+}
+
+__attribute__((noipa)) int
+clzD523 (unsigned _BitInt(523) x, int y)
+{
+  return __builtin_clzg (x, y);
+}
+
+__attribute__((noipa)) int
+ctz523 (unsigned _BitInt(523) x)
+{
+  return __builtin_ctzg (x);
+}
+
+__attribute__((noipa)) int
+ctzd523 (unsigned _BitInt(523) x)
+{
+  return __builtin_ctzg (x, __builtin_popcountg ((typeof (x)) ~(typeof (x)) 0));
+}
+
+__attribute__((noipa)) int
+ctzD523 (unsigned _BitInt(523) x, int y)
+{
+  return __builtin_ctzg (x, y);
+}
+
+__attribute__((noipa)) int
+clrsb523 (_BitInt(523) x)
+{
+  return __builtin_clrsbg (x);
+}
+
+__attribute__((noipa)) int
+ffs523 (_BitInt(523) x)
+{
+  return __builtin_ffsg (x);
+}
+
+__attribute__((noipa)) int
+parity523 (unsigned _BitInt(523) x)
+{
+  return __builtin_parityg (x);
+}
+
+__attribute__((noipa)) int
+popcount523 (unsigned _BitInt(523) x)
+{
+  return __builtin_popcountg (x);
+}
+#endif
+
+int
+main ()
+{
+#if __BITINT_MAXWIDTH__ >= 512
+  if (clzd512 (0) != 512
+      || clzD512 (0, -1) != -1
+      || ctzd512 (0) != 512
+      || ctzD512 (0, 42) != 42
+      || clrsb512 (0) != 512 - 1
+      || ffs512 (0) != 0
+      || parity512 (0) != 0
+      || popcount512 (0) != 0
+      || __builtin_clzg ((unsigned _BitInt(512)) 0, 512 + 32) != 512 + 32
+      || __builtin_ctzg ((unsigned _BitInt(512)) 0, 512) != 512
+      || __builtin_clrsbg ((_BitInt(512)) 0) != 512 - 1
+      || __builtin_ffsg ((_BitInt(512)) 0) != 0
+      || __builtin_parityg ((unsigned _BitInt(512)) 0) != 0
+      || __builtin_popcountg ((unsigned _BitInt(512)) 0) != 0)
+    __builtin_abort ();
+  if (clz512 (-1) != 0
+      || clzd512 (-1) != 0
+      || clzD512 (-1, 0) != 0
+      || ctz512 (-1) != 0
+      || ctzd512 (-1) != 0
+      || ctzD512 (-1, 17) != 0
+      || clrsb512 (-1) != 512 - 1
+      || ffs512 (-1) != 1
+      || parity512 (-1) != 0
+      || popcount512 (-1) != 512
+      || __builtin_clzg ((unsigned _BitInt(512)) -1) != 0
+      || __builtin_clzg ((unsigned _BitInt(512)) -1, 512 + 32) != 0
+      || __builtin_ctzg ((unsigned _BitInt(512)) -1) != 0
+      || __builtin_ctzg ((unsigned _BitInt(512)) -1, 512) != 0
+      || __builtin_clrsbg ((_BitInt(512)) -1) != 512 - 1
+      || __builtin_ffsg ((_BitInt(512)) -1) != 1
+      || __builtin_parityg ((unsigned _BitInt(512)) -1) != 0
+      || __builtin_popcountg ((unsigned _BitInt(512)) -1) != 512)
+    __builtin_abort ();
+  if (clz512 (((unsigned _BitInt(512)) -1) >> 24) != 24
+      || clz512 (((unsigned _BitInt(512)) -1) >> 79) != 79
+      || clz512 (1) != 512 - 1
+      || clzd512 (((unsigned _BitInt(512)) -1) >> 139) != 139
+      || clzd512 (2) != 512 - 2
+      || ctz512 (((unsigned _BitInt(512)) -1) << 42) != 42
+      || ctz512 (((unsigned _BitInt(512)) -1) << 57) != 57
+      || ctz512 (0x4000000000000000000000uwb) != 86
+      || ctzd512 (((unsigned _BitInt(512)) -1) << 149) != 149
+      || ctzd512 (2) != 1
+      || clrsb512 ((unsigned _BitInt(512 - 4)) -1) != 3
+      || clrsb512 ((unsigned _BitInt(512 - 28)) -1) != 27
+      || clrsb512 ((unsigned _BitInt(512 - 29)) -1) != 28
+      || clrsb512 (~(unsigned _BitInt(512)) (unsigned _BitInt(512 - 68)) -1) != 67
+      || clrsb512 (~(unsigned _BitInt(512)) (unsigned _BitInt(512 - 92)) -1) != 91
+      || clrsb512 (~(unsigned _BitInt(512)) (unsigned _BitInt(512 - 93)) -1) != 92
+      || ffs512 (((unsigned _BitInt(512)) -1) << 42) != 43
+      || ffs512 (((unsigned _BitInt(512)) -1) << 57) != 58
+      || ffs512 (0x4000000000000000000000uwb) != 87
+      || ffs512 (((unsigned _BitInt(512)) -1) << 149) != 150
+      || ffs512 (2) != 2
+      || __builtin_clzg (((unsigned _BitInt(512)) -1) >> 24) != 24
+      || __builtin_clzg (((unsigned _BitInt(512)) -1) >> 79) != 79
+      || __builtin_clzg ((unsigned _BitInt(512)) 1) != 512 - 1
+      || __builtin_clzg (((unsigned _BitInt(512)) -1) >> 139, 512) != 139
+      || __builtin_clzg ((unsigned _BitInt(512)) 2, 512) != 512 - 2
+      || __builtin_ctzg (((unsigned _BitInt(512)) -1) << 42) != 42
+      || __builtin_ctzg (((unsigned _BitInt(512)) -1) << 57) != 57
+      || __builtin_ctzg ((unsigned _BitInt(512)) 0x4000000000000000000000uwb) != 86
+      || __builtin_ctzg (((unsigned _BitInt(512)) -1) << 149, 512) != 149
+      || __builtin_ctzg ((unsigned _BitInt(512)) 2, 512) != 1
+      || __builtin_clrsbg ((_BitInt(512)) (unsigned _BitInt(512 - 4)) -1) != 3
+      || __builtin_clrsbg ((_BitInt(512)) (unsigned _BitInt(512 - 28)) -1) != 27
+      || __builtin_clrsbg ((_BitInt(512)) (unsigned _BitInt(512 - 29)) -1) != 28
+      || __builtin_clrsbg ((_BitInt(512)) ~(unsigned _BitInt(512)) (unsigned _BitInt(512 - 68)) -1) != 67
+      || __builtin_clrsbg ((_BitInt(512)) ~(unsigned _BitInt(512)) (unsigned _BitInt(512 - 92)) -1) != 91
+      || __builtin_clrsbg ((_BitInt(512)) ~(unsigned _BitInt(512)) (unsigned _BitInt(512 - 93)) -1) != 92
+      || __builtin_ffsg ((_BitInt(512)) (((unsigned _BitInt(512)) -1) << 42)) != 43
+      || __builtin_ffsg ((_BitInt(512)) (((unsigned _BitInt(512)) -1) << 57)) != 58
+      || __builtin_ffsg ((_BitInt(512)) 0x4000000000000000000000uwb) != 87
+      || __builtin_ffsg ((_BitInt(512)) (((unsigned _BitInt(512)) -1) << 149)) != 150
+      || __builtin_ffsg ((_BitInt(512)) 2) != 2)
+    __builtin_abort ();
+  if (parity512 (8278593062772967967574644592392030907507244457324713380127157444008480135136016412791369421272159911061801023217823646324038055629840240503699995274750141uwb) != __builtin_parityg (8278593062772967967574644592392030907507244457324713380127157444008480135136016412791369421272159911061801023217823646324038055629840240503699995274750141uwb)
+      || parity512 (663951521760319802637316646127146913163123967584512032007606686578544864655291546789196279408181546344880831465704154822174055168766759305688225967189384uwb) != __builtin_parityg (663951521760319802637316646127146913163123967584512032007606686578544864655291546789196279408181546344880831465704154822174055168766759305688225967189384uwb)
+      || parity512 (8114152627481936575035564712656624361256533214211179387274127464949371919139038942819974113641465089580051998523156404968195970853124179018281296621919217uwb) != __builtin_parityg (8114152627481936575035564712656624361256533214211179387274127464949371919139038942819974113641465089580051998523156404968195970853124179018281296621919217uwb)
+      || popcount512 (697171368046392901434470580443928282938585745214587494987284546386421344865289735592202298494880955572094546861862007016154025065165834164941207378563932uwb) != __builtin_popcountg (697171368046392901434470580443928282938585745214587494987284546386421344865289735592202298494880955572094546861862007016154025065165834164941207378563932uwb)
+      || popcount512 (12625357869391866487124235043239209385173615631331705015179232007319637649427586947822360147798041278948617160703315666047585702906648747835331939389354450uwb) != __builtin_popcountg (12625357869391866487124235043239209385173615631331705015179232007319637649427586947822360147798041278948617160703315666047585702906648747835331939389354450uwb)
+      || popcount512 (12989863959706456104163426941303698078341934896544520782734564901708926112239778316241786242633862403309192697330635825122310265805838908726925342761646021uwb) != __builtin_popcountg (12989863959706456104163426941303698078341934896544520782734564901708926112239778316241786242633862403309192697330635825122310265805838908726925342761646021uwb))
+    __builtin_abort ();
+#endif
+#if __BITINT_MAXWIDTH__ >= 523
+  if (clzd523 (0) != 523
+      || clzD523 (0, 42) != 42
+      || ctzd523 (0) != 523
+      || ctzD523 (0, -1) != -1
+      || clrsb523 (0) != 523 - 1
+      || ffs523 (0) != 0
+      || parity523 (0) != 0
+      || popcount523 (0) != 0
+      || __builtin_clzg ((unsigned _BitInt(523)) 0, 523 + 32) != 523 + 32
+      || __builtin_ctzg ((unsigned _BitInt(523)) 0, 523) != 523
+      || __builtin_clrsbg ((_BitInt(523)) 0) != 523 - 1
+      || __builtin_ffsg ((_BitInt(523)) 0) != 0
+      || __builtin_parityg ((unsigned _BitInt(523)) 0) != 0
+      || __builtin_popcountg ((unsigned _BitInt(523)) 0) != 0)
+    __builtin_abort ();
+  if (clz523 (-1) != 0
+      || clzd523 (-1) != 0
+      || clzD523 (-1, 15) != 0
+      || ctz523 (-1) != 0
+      || ctzd523 (-1) != 0
+      || ctzD523 (-1, -57) != 0
+      || clrsb523 (-1) != 523 - 1
+      || ffs523 (-1) != 1
+      || parity523 (-1) != 1
+      || popcount523 (-1) != 523
+      || __builtin_clzg ((unsigned _BitInt(523)) -1) != 0
+      || __builtin_clzg ((unsigned _BitInt(523)) -1, 523 + 32) != 0
+      || __builtin_ctzg ((unsigned _BitInt(523)) -1) != 0
+      || __builtin_ctzg ((unsigned _BitInt(523)) -1, 523) != 0
+      || __builtin_clrsbg ((_BitInt(523)) -1) != 523 - 1
+      || __builtin_ffsg ((_BitInt(523)) -1) != 1
+      || __builtin_parityg ((unsigned _BitInt(523)) -1) != 1
+      || __builtin_popcountg ((unsigned _BitInt(523)) -1) != 523)
+    __builtin_abort ();
+  if (clz523 (((unsigned _BitInt(523)) -1) >> 24) != 24
+      || clz523 (((unsigned _BitInt(523)) -1) >> 79) != 79
+      || clz523 (1) != 523 - 1
+      || clzd523 (((unsigned _BitInt(523)) -1) >> 139) != 139
+      || clzd523 (2) != 523 - 2
+      || ctz523 (((unsigned _BitInt(523)) -1) << 42) != 42
+      || ctz523 (((unsigned _BitInt(523)) -1) << 57) != 57
+      || ctz523 (0x4000000000000000000000uwb) != 86
+      || ctzd523 (((unsigned _BitInt(523)) -1) << 149) != 149
+      || ctzd523 (2) != 1
+      || clrsb523 ((unsigned _BitInt(523 - 4)) -1) != 3
+      || clrsb523 ((unsigned _BitInt(523 - 28)) -1) != 27
+      || clrsb523 ((unsigned _BitInt(523 - 29)) -1) != 28
+      || clrsb523 (~(unsigned _BitInt(523)) (unsigned _BitInt(523 - 68)) -1) != 67
+      || clrsb523 (~(unsigned _BitInt(523)) (unsigned _BitInt(523 - 92)) -1) != 91
+      || clrsb523 (~(unsigned _BitInt(523)) (unsigned _BitInt(523 - 93)) -1) != 92
+      || ffs523 (((unsigned _BitInt(523)) -1) << 42) != 43
+      || ffs523 (((unsigned _BitInt(523)) -1) << 57) != 58
+      || ffs523 (0x4000000000000000000000uwb) != 87
+      || ffs523 (((unsigned _BitInt(523)) -1) << 149) != 150
+      || ffs523 (2) != 2
+      || __builtin_clzg (((unsigned _BitInt(523)) -1) >> 24) != 24
+      || __builtin_clzg (((unsigned _BitInt(523)) -1) >> 79) != 79
+      || __builtin_clzg ((unsigned _BitInt(523)) 1) != 523 - 1
+      || __builtin_clzg (((unsigned _BitInt(523)) -1) >> 139, 523) != 139
+      || __builtin_clzg ((unsigned _BitInt(523)) 2, 523) != 523 - 2
+      || __builtin_ctzg (((unsigned _BitInt(523)) -1) << 42) != 42
+      || __builtin_ctzg (((unsigned _BitInt(523)) -1) << 57) != 57
+      || __builtin_ctzg ((unsigned _BitInt(523)) 0x4000000000000000000000uwb) != 86
+      || __builtin_ctzg (((unsigned _BitInt(523)) -1) << 149, 523) != 149
+      || __builtin_ctzg ((unsigned _BitInt(523)) 2, 523) != 1
+      || __builtin_clrsbg ((_BitInt(523)) (unsigned _BitInt(523 - 4)) -1) != 3
+      || __builtin_clrsbg ((_BitInt(523)) (unsigned _BitInt(523 - 28)) -1) != 27
+      || __builtin_clrsbg ((_BitInt(523)) (unsigned _BitInt(523 - 29)) -1) != 28
+      || __builtin_clrsbg ((_BitInt(523)) ~(unsigned _BitInt(523)) (unsigned _BitInt(523 - 68)) -1) != 67
+      || __builtin_clrsbg ((_BitInt(523)) ~(unsigned _BitInt(523)) (unsigned _BitInt(523 - 92)) -1) != 91
+      || __builtin_clrsbg ((_BitInt(523)) ~(unsigned _BitInt(523)) (unsigned _BitInt(523 - 93)) -1) != 92
+      || __builtin_ffsg ((_BitInt(523)) (((unsigned _BitInt(523)) -1) << 42)) != 43
+      || __builtin_ffsg ((_BitInt(523)) (((unsigned _BitInt(523)) -1) << 57)) != 58
+      || __builtin_ffsg ((_BitInt(523)) 0x4000000000000000000000uwb) != 87
+      || __builtin_ffsg ((_BitInt(523)) (((unsigned _BitInt(523)) -1) << 149)) != 150
+      || __builtin_ffsg ((_BitInt(523)) 2) != 2)
+    __builtin_abort ();
+  if (parity523 (14226628251091586975416900831427560438504550751597528218770815297642064445318137709184907300499591292677456563377096100346699421879373024906380724757049700104uwb) != __builtin_parityg (14226628251091586975416900831427560438504550751597528218770815297642064445318137709184907300499591292677456563377096100346699421879373024906380724757049700104uwb)
+      || parity523 (20688958227123188226117538663818621034852702121556301239818743230005799574164516085541310491875153692467123662601853835357822935286851364843928714141587045255uwb) != __builtin_parityg (20688958227123188226117538663818621034852702121556301239818743230005799574164516085541310491875153692467123662601853835357822935286851364843928714141587045255uwb)
+      || parity523 (8927708174664018648856542263215989788443763271738485875573765922613438023117960552135374015673598803453205044464280019640319125968982118836809392169156450404uwb) != __builtin_parityg (8927708174664018648856542263215989788443763271738485875573765922613438023117960552135374015673598803453205044464280019640319125968982118836809392169156450404uwb)
+      || popcount523 (27178327344587654457581274852432957423537947348354896748701960885269035920194935311522194372418922852798513401240689173265979378157685169921449935364246334672uwb) != __builtin_popcountg (27178327344587654457581274852432957423537947348354896748701960885269035920194935311522194372418922852798513401240689173265979378157685169921449935364246334672uwb)
+      || popcount523 (5307736750284212829931201546806718535860789684371772688568780952567669490917265125893664418036905110148872995350655890585853451175740907670080602411287166989uwb) != __builtin_popcountg (5307736750284212829931201546806718535860789684371772688568780952567669490917265125893664418036905110148872995350655890585853451175740907670080602411287166989uwb)
+      || popcount523 (21261096432069432668470452941790780841888331284195411465624030283325239673941548816191698556934198698768393659379577567450765073013688585051560340496749593370uwb) != __builtin_popcountg (21261096432069432668470452941790780841888331284195411465624030283325239673941548816191698556934198698768393659379577567450765073013688585051560340496749593370uwb))
+    __builtin_abort ();
+#endif
+}
diff --git a/gcc/tree-ssa-forwprop.cc b/gcc/tree-ssa-forwprop.cc
index d4e9202..d39dfc1 100644
--- a/gcc/tree-ssa-forwprop.cc
+++ b/gcc/tree-ssa-forwprop.cc
@@ -2381,6 +2381,7 @@ simplify_count_trailing_zeroes (gimple_stmt_iterator *gsi)
       HOST_WIDE_INT type_size = tree_to_shwi (TYPE_SIZE (type));
       bool zero_ok
 	= CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (type), ctz_val) == 2;
+      int nargs = 2;
 
       /* If the input value can't be zero, don't special case ctz (0).  */
       if (tree_expr_nonzero_p (res_ops[0]))
@@ -2388,6 +2389,7 @@ simplify_count_trailing_zeroes (gimple_stmt_iterator *gsi)
 	  zero_ok = true;
 	  zero_val = 0;
 	  ctz_val = 0;
+	  nargs = 1;
 	}
 
       /* Skip if there is no value defined at zero, or if we can't easily
@@ -2399,7 +2401,11 @@ simplify_count_trailing_zeroes (gimple_stmt_iterator *gsi)
 
       gimple_seq seq = NULL;
       gimple *g;
-      gcall *call = gimple_build_call_internal (IFN_CTZ, 1, res_ops[0]);
+      gcall *call
+	= gimple_build_call_internal (IFN_CTZ, nargs, res_ops[0],
+				      nargs == 1 ? NULL_TREE
+				      : build_int_cst (integer_type_node,
+						       ctz_val));
       gimple_set_location (call, gimple_location (stmt));
       gimple_set_lhs (call, make_ssa_name (integer_type_node));
       gimple_seq_add_stmt (&seq, call);
diff --git a/gcc/tree-ssa-loop-niter.cc b/gcc/tree-ssa-loop-niter.cc
index 718582a..2098bef 100644
--- a/gcc/tree-ssa-loop-niter.cc
+++ b/gcc/tree-ssa-loop-niter.cc
@@ -2235,14 +2235,18 @@ build_cltz_expr (tree src, bool leading, bool define_at_zero)
   tree call;
   if (use_ifn)
     {
-      call = build_call_expr_internal_loc (UNKNOWN_LOCATION, ifn,
-					   integer_type_node, 1, src);
       int val;
       int optab_defined_at_zero
 	= (leading
 	   ? CLZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (utype), val)
 	   : CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (utype), val));
-      if (define_at_zero && !(optab_defined_at_zero == 2 && val == prec))
+      tree arg2 = NULL_TREE;
+      if (define_at_zero && optab_defined_at_zero == 2 && val == prec)
+	arg2 = build_int_cst (integer_type_node, val);
+      call = build_call_expr_internal_loc (UNKNOWN_LOCATION, ifn,
+					   integer_type_node, arg2 ? 2 : 1,
+					   src, arg2);
+      if (define_at_zero && arg2 == NULL_TREE)
 	{
 	  tree is_zero = fold_build2 (NE_EXPR, boolean_type_node, src,
 				      build_zero_cst (TREE_TYPE (src)));
diff --git a/gcc/tree-ssa-phiopt.cc b/gcc/tree-ssa-phiopt.cc
index bb55a4f..ac80517 100644
--- a/gcc/tree-ssa-phiopt.cc
+++ b/gcc/tree-ssa-phiopt.cc
@@ -2863,18 +2863,26 @@ cond_removal_in_builtin_zero_pattern (basic_block cond_bb,
     }
 
   /* Check that we have a popcount/clz/ctz builtin.  */
-  if (!is_gimple_call (call) || gimple_call_num_args (call) != 1)
+  if (!is_gimple_call (call))
     return false;
 
-  arg = gimple_call_arg (call, 0);
   lhs = gimple_get_lhs (call);
 
   if (lhs == NULL_TREE)
     return false;
 
   combined_fn cfn = gimple_call_combined_fn (call);
+  if (gimple_call_num_args (call) != 1
+      && (gimple_call_num_args (call) != 2
+	  || cfn == CFN_CLZ
+	  || cfn == CFN_CTZ))
+    return false;
+
+  arg = gimple_call_arg (call, 0);
+
   internal_fn ifn = IFN_LAST;
   int val = 0;
+  bool any_val = false;
   switch (cfn)
     {
     case CFN_BUILT_IN_BSWAP16:
@@ -2889,6 +2897,23 @@ cond_removal_in_builtin_zero_pattern (basic_block cond_bb,
       if (INTEGRAL_TYPE_P (TREE_TYPE (arg)))
 	{
 	  tree type = TREE_TYPE (arg);
+	  if (TREE_CODE (type) == BITINT_TYPE)
+	    {
+	      if (gimple_call_num_args (call) == 1)
+		{
+		  any_val = true;
+		  ifn = IFN_CLZ;
+		  break;
+		}
+	      if (!tree_fits_shwi_p (gimple_call_arg (call, 1)))
+		return false;
+	      HOST_WIDE_INT at_zero = tree_to_shwi (gimple_call_arg (call, 1));
+	      if ((int) at_zero != at_zero)
+		return false;
+	      ifn = IFN_CLZ;
+	      val = at_zero;
+	      break;
+	    }
 	  if (direct_internal_fn_supported_p (IFN_CLZ, type, OPTIMIZE_FOR_BOTH)
 	      && CLZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (type),
 					    val) == 2)
@@ -2902,6 +2927,23 @@ cond_removal_in_builtin_zero_pattern (basic_block cond_bb,
       if (INTEGRAL_TYPE_P (TREE_TYPE (arg)))
 	{
 	  tree type = TREE_TYPE (arg);
+	  if (TREE_CODE (type) == BITINT_TYPE)
+	    {
+	      if (gimple_call_num_args (call) == 1)
+		{
+		  any_val = true;
+		  ifn = IFN_CTZ;
+		  break;
+		}
+	      if (!tree_fits_shwi_p (gimple_call_arg (call, 1)))
+		return false;
+	      HOST_WIDE_INT at_zero = tree_to_shwi (gimple_call_arg (call, 1));
+	      if ((int) at_zero != at_zero)
+		return false;
+	      ifn = IFN_CTZ;
+	      val = at_zero;
+	      break;
+	    }
 	  if (direct_internal_fn_supported_p (IFN_CTZ, type, OPTIMIZE_FOR_BOTH)
 	      && CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (type),
 					    val) == 2)
@@ -2960,8 +3002,18 @@ cond_removal_in_builtin_zero_pattern (basic_block cond_bb,
 
   /* Check PHI arguments.  */
   if (lhs != arg0
-      || TREE_CODE (arg1) != INTEGER_CST
-      || wi::to_wide (arg1) != val)
+      || TREE_CODE (arg1) != INTEGER_CST)
+    return false;
+  if (any_val)
+    {
+      if (!tree_fits_shwi_p (arg1))
+	return false;
+      HOST_WIDE_INT at_zero = tree_to_shwi (arg1);
+      if ((int) at_zero != at_zero)
+	return false;
+      val = at_zero;
+    }
+  else if (wi::to_wide (arg1) != val)
     return false;
 
   /* And insert the popcount/clz/ctz builtin and cast stmt before the
@@ -2974,13 +3026,15 @@ cond_removal_in_builtin_zero_pattern (basic_block cond_bb,
       reset_flow_sensitive_info (gimple_get_lhs (cast));
     }
   gsi_from = gsi_for_stmt (call);
-  if (ifn == IFN_LAST || gimple_call_internal_p (call))
+  if (ifn == IFN_LAST
+      || (gimple_call_internal_p (call) && gimple_call_num_args (call) == 2))
     gsi_move_before (&gsi_from, &gsi);
   else
     {
       /* For __builtin_c[lt]z* force .C[LT]Z ifn, because only
 	 the latter is well defined at zero.  */
-      call = gimple_build_call_internal (ifn, 1, gimple_call_arg (call, 0));
+      call = gimple_build_call_internal (ifn, 2, gimple_call_arg (call, 0),
+					 build_int_cst (integer_type_node, val));
       gimple_call_set_lhs (call, lhs);
       gsi_insert_before (&gsi, call, GSI_SAME_STMT);
       gsi_remove (&gsi_from, true);
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 6b6b412..7debe7f 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -1818,7 +1818,7 @@ vect_recog_ctz_ffs_pattern (vec_info *vinfo, stmt_vec_info stmt_vinfo,
   tree new_var;
   internal_fn ifn = IFN_LAST, ifnnew = IFN_LAST;
   bool defined_at_zero = true, defined_at_zero_new = false;
-  int val = 0, val_new = 0;
+  int val = 0, val_new = 0, val_cmp = 0;
   int prec;
   int sub = 0, add = 0;
   location_t loc;
@@ -1826,7 +1826,8 @@ vect_recog_ctz_ffs_pattern (vec_info *vinfo, stmt_vec_info stmt_vinfo,
   if (!is_gimple_call (call_stmt))
     return NULL;
 
-  if (gimple_call_num_args (call_stmt) != 1)
+  if (gimple_call_num_args (call_stmt) != 1
+      && gimple_call_num_args (call_stmt) != 2)
     return NULL;
 
   rhs_oprnd = gimple_call_arg (call_stmt, 0);
@@ -1846,9 +1847,10 @@ vect_recog_ctz_ffs_pattern (vec_info *vinfo, stmt_vec_info stmt_vinfo,
     CASE_CFN_CTZ:
       ifn = IFN_CTZ;
       if (!gimple_call_internal_p (call_stmt)
-	  || CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (rhs_type),
-					val) != 2)
+	  || gimple_call_num_args (call_stmt) != 2)
 	defined_at_zero = false;
+      else
+	val = tree_to_shwi (gimple_call_arg (call_stmt, 1));
       break;
     CASE_CFN_FFS:
       ifn = IFN_FFS;
@@ -1907,6 +1909,7 @@ vect_recog_ctz_ffs_pattern (vec_info *vinfo, stmt_vec_info stmt_vinfo,
 
   vect_pattern_detected ("vec_recog_ctz_ffs_pattern", call_stmt);
 
+  val_cmp = val_new;
   if ((ifnnew == IFN_CLZ
        && defined_at_zero
        && defined_at_zero_new
@@ -1918,7 +1921,7 @@ vect_recog_ctz_ffs_pattern (vec_info *vinfo, stmt_vec_info stmt_vinfo,
 	 .CTZ (X) = .POPCOUNT ((X - 1) & ~X).  */
       if (ifnnew == IFN_CLZ)
 	sub = prec;
-      val_new = prec;
+      val_cmp = prec;
 
       if (!TYPE_UNSIGNED (rhs_type))
 	{
@@ -1955,7 +1958,7 @@ vect_recog_ctz_ffs_pattern (vec_info *vinfo, stmt_vec_info stmt_vinfo,
       /* .CTZ (X) = (PREC - 1) - .CLZ (X & -X)
 	 .FFS (X) = PREC - .CLZ (X & -X).  */
       sub = prec - (ifn == IFN_CTZ);
-      val_new = sub - val_new;
+      val_cmp = sub - val_new;
 
       tree neg = vect_recog_temp_ssa_var (rhs_type, NULL);
       pattern_stmt = gimple_build_assign (neg, NEGATE_EXPR, rhs_oprnd);
@@ -1974,7 +1977,7 @@ vect_recog_ctz_ffs_pattern (vec_info *vinfo, stmt_vec_info stmt_vinfo,
       /* .CTZ (X) = PREC - .POPCOUNT (X | -X)
 	 .FFS (X) = (PREC + 1) - .POPCOUNT (X | -X).  */
       sub = prec + (ifn == IFN_FFS);
-      val_new = sub;
+      val_cmp = sub;
 
       tree neg = vect_recog_temp_ssa_var (rhs_type, NULL);
       pattern_stmt = gimple_build_assign (neg, NEGATE_EXPR, rhs_oprnd);
@@ -1992,12 +1995,18 @@ vect_recog_ctz_ffs_pattern (vec_info *vinfo, stmt_vec_info stmt_vinfo,
     {
       /* .FFS (X) = .CTZ (X) + 1.  */
       add = 1;
-      val_new++;
+      val_cmp++;
     }
 
   /* Create B = .IFNNEW (A).  */
   new_var = vect_recog_temp_ssa_var (lhs_type, NULL);
-  pattern_stmt = gimple_build_call_internal (ifnnew, 1, rhs_oprnd);
+  if ((ifnnew == IFN_CLZ || ifnnew == IFN_CTZ) && defined_at_zero_new)
+    pattern_stmt
+      = gimple_build_call_internal (ifnnew, 2, rhs_oprnd,
+				    build_int_cst (integer_type_node,
+						   val_new));
+  else
+    pattern_stmt = gimple_build_call_internal (ifnnew, 1, rhs_oprnd);
   gimple_call_set_lhs (pattern_stmt, new_var);
   gimple_set_location (pattern_stmt, loc);
   *type_out = vec_type;
@@ -2023,7 +2032,7 @@ vect_recog_ctz_ffs_pattern (vec_info *vinfo, stmt_vec_info stmt_vinfo,
     }
 
   if (defined_at_zero
-      && (!defined_at_zero_new || val != val_new))
+      && (!defined_at_zero_new || val != val_cmp))
     {
       append_pattern_def_seq (vinfo, stmt_vinfo, pattern_stmt, vec_type);
       tree ret_var = vect_recog_temp_ssa_var (lhs_type, NULL);
@@ -2143,7 +2152,8 @@ vect_recog_popcount_clz_ctz_ffs_pattern (vec_info *vinfo,
       return NULL;
     }
 
-  if (gimple_call_num_args (call_stmt) != 1)
+  if (gimple_call_num_args (call_stmt) != 1
+      && gimple_call_num_args (call_stmt) != 2)
     return NULL;
 
   rhs_oprnd = gimple_call_arg (call_stmt, 0);
@@ -2181,17 +2191,14 @@ vect_recog_popcount_clz_ctz_ffs_pattern (vec_info *vinfo,
 	  return NULL;
 	addend = (TYPE_PRECISION (TREE_TYPE (rhs_oprnd))
 		  - TYPE_PRECISION (lhs_type));
-	if (gimple_call_internal_p (call_stmt))
+	if (gimple_call_internal_p (call_stmt)
+	    && gimple_call_num_args (call_stmt) == 2)
 	  {
 	    int val1, val2;
-	    int d1
-	      = CLZ_DEFINED_VALUE_AT_ZERO
-		  (SCALAR_INT_TYPE_MODE (TREE_TYPE (rhs_oprnd)), val1);
+	    val1 = tree_to_shwi (gimple_call_arg (call_stmt, 1));
 	    int d2
 	      = CLZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (lhs_type),
 					   val2);
-	    if (d1 != 2)
-	      break;
 	    if (d2 != 2 || val1 != val2 + addend)
 	      return NULL;
 	  }
@@ -2200,17 +2207,14 @@ vect_recog_popcount_clz_ctz_ffs_pattern (vec_info *vinfo,
 	/* ctzll (x) == ctz (x) for unsigned or signed x != 0, so ok
 	   if it is undefined at zero or if it matches also for the
 	   defined value there.  */
-	if (gimple_call_internal_p (call_stmt))
+	if (gimple_call_internal_p (call_stmt)
+	    && gimple_call_num_args (call_stmt) == 2)
 	  {
 	    int val1, val2;
-	    int d1
-	      = CTZ_DEFINED_VALUE_AT_ZERO
-		  (SCALAR_INT_TYPE_MODE (TREE_TYPE (rhs_oprnd)), val1);
+	    val1 = tree_to_shwi (gimple_call_arg (call_stmt, 1));
 	    int d2
 	      = CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (lhs_type),
 					   val2);
-	    if (d1 != 2)
-	      break;
 	    if (d2 != 2 || val1 != val2)
 	      return NULL;
 	  }
@@ -2260,7 +2264,20 @@ vect_recog_popcount_clz_ctz_ffs_pattern (vec_info *vinfo,
 
   /* Create B = .POPCOUNT (A).  */
   new_var = vect_recog_temp_ssa_var (lhs_type, NULL);
-  pattern_stmt = gimple_build_call_internal (ifn, 1, unprom_diff.op);
+  tree arg2 = NULL_TREE;
+  int val;
+  if (ifn == IFN_CLZ
+      && CLZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (lhs_type),
+				    val) == 2)
+    arg2 = build_int_cst (integer_type_node, val);
+  else if (ifn == IFN_CTZ
+	   && CTZ_DEFINED_VALUE_AT_ZERO (SCALAR_INT_TYPE_MODE (lhs_type),
+					 val) == 2)
+    arg2 = build_int_cst (integer_type_node, val);
+  if (arg2)
+    pattern_stmt = gimple_build_call_internal (ifn, 2, unprom_diff.op, arg2);
+  else
+    pattern_stmt = gimple_build_call_internal (ifn, 1, unprom_diff.op);
   gimple_call_set_lhs (pattern_stmt, new_var);
   gimple_set_location (pattern_stmt, gimple_location (last_stmt));
   *type_out = vec_type;
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index ee89f47..96e4a6cf 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -3115,6 +3115,7 @@ vectorizable_call (vec_info *vinfo,
   enum { NARROW, NONE, WIDEN } modifier;
   size_t i, nargs;
   tree lhs;
+  tree clz_ctz_arg1 = NULL_TREE;
 
   if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo)
     return false;
@@ -3160,6 +3161,14 @@ vectorizable_call (vec_info *vinfo,
       nargs = 0;
       rhs_type = unsigned_type_node;
     }
+  /* Similarly pretend IFN_CLZ and IFN_CTZ only has one argument, the second
+     argument just says whether it is well-defined at zero or not and what
+     value should be returned for it.  */
+  if ((cfn == CFN_CLZ || cfn == CFN_CTZ) && nargs == 2)
+    {
+      nargs = 1;
+      clz_ctz_arg1 = gimple_call_arg (stmt, 1);
+    }
 
   int mask_opno = -1;
   if (internal_fn_p (cfn))
@@ -3425,6 +3434,8 @@ vectorizable_call (vec_info *vinfo,
       ifn = cond_fn;
       vect_nargs += 2;
     }
+  if (clz_ctz_arg1)
+    ++vect_nargs;
 
   if (modifier == NONE || ifn != IFN_LAST)
     {
@@ -3462,6 +3473,9 @@ vectorizable_call (vec_info *vinfo,
 		    }
 		  if (masked_loop_p && reduc_idx >= 0)
 		    vargs[varg++] = vargs[reduc_idx + 1];
+		  if (clz_ctz_arg1)
+		    vargs[varg++] = clz_ctz_arg1;
+
 		  gimple *new_stmt;
 		  if (modifier == NARROW)
 		    {
@@ -3548,6 +3562,8 @@ vectorizable_call (vec_info *vinfo,
 	    }
 	  if (masked_loop_p && reduc_idx >= 0)
 	    vargs[varg++] = vargs[reduc_idx + 1];
+	  if (clz_ctz_arg1)
+	    vargs[varg++] = clz_ctz_arg1;
 
 	  if (len_opno >= 0 && len_loop_p)
 	    {