sparc.opt (msubxc): New option.

	* config/sparc/sparc.opt (msubxc): New option.
	* doc/invoke.texi (SPARC options): Document it and tidy up.
	* doc/tm.texi.in (Condition Codes): Adjust SPARC example.
	* doc/tm.texi: Regenerate.
	* config/sparc/sparc-modes.def (CC_NOOV): Rename into...
	(CCNZ): ...this.
	(CCX_NOOV): Rename into...
	(CCXNZ): ...this.
	(CCC): New.
	(CCXC): Likewise.
	* config/sparc/predicates.m (fcc_register_operand): Simplify.
	(fcc0_register_operand): Likewise.
	(icc_register_operand): New.
	(icc_or_fcc_register_operand): Simplify.
	(nz_comparison_operator): New.
	(c_comparison_operator): Likewise.
	(noov_compare_operator): Rename into...
	(icc_comparison_operator): ...this.  Use above predicates.
	(noov_compare64_operator): Rename into...
	(v9_comparison_operator): ...this and tidy up.
	(fcc_comparison_operator): New.
	(icc_or_fcc_comparison_operator): Likewise.
	(v9_register_compare_operator): Rename info...
	(v9_register_comparison_operator): ...this.
	* config/sparc/sparc.c (TARGET_FIXED_CONDITION_CODE_REGS): Define.
	(sparc_option_override): Remove redundant VIS masks and add MASK_SUBXC
	for Niagara-7.
	(sparc_fixed_condition_code_regs): New function.
	(select_cc_mode): Remove ATTRIBUTE_UNUSED.  Adjust for CCNZ/CCXNZ
	renaming and add support for CCC/CCXC.
	(output_cbranch): Likewise.
	(sparc_print_operand): Likewise.
	(gen_v9_scc): Remove obsolete assertion.
	(emit_scc_insn): Emit RTL directly for EQ and NE.  Add direct support
	for EQ in DImode if TARGET_SUBXC.  Remove test on TARGET_VIS3 for GEU.
	(output_cbcond): Remove bogus handling of CC modes.
	(sparc_register_move_cost): Return 100 for NO_REGS.
	* config/sparc/sparc.md (W): New mode iterator.
	(length): Adjust for noov_compare64_operator renaming.
	(cmpsi_sne): New instruction.
	(cmpdi_sne): Likewise.
	(seqdi_special): Delete.
	(seqdi_special): Likewise.
	(snesi<P:mode>_special): Likewise.
	(snedi_special): Likewise.
	(snedi_special_vis3): Likewise.
	(snesi patterns): Use W iterator.
	(snedi patterns): Likewise.  Add TARGET_SUBXC patterns.
	(sltu patterns): Likewise.
	(sgeu patterns): Likewise.
	(scc splitter): Do not split GEU in DImode if TARGET_SUBXC.
	(normal_branch): Use icc_comparison_operator predicate.
	(inverted_branch): Likewise.
	(cbcond_sp32): Use comparison_operator predicate.
	(cbcond_sp64): Likewise.
	(normal_int_branch_sp64): Adjust for renaming
	(inverted_int_branch_sp64): Likewise.
	(mov<I:mode>_cc_reg_sp64): Likewise.
	(movsf_cc_reg_sp6): Likewise.
	(movdf_cc_reg_sp64): Likewise.
	(movtf_cc_reg_hq_sp64): Likewise.
	(movtf_cc_reg_sp64): Likewise.
	(mov<I:mode>_cc_v9): Use icc_or_fcc_comparison_operator predicate.
	(movsf_cc_v9): Likewise.
	(movdf_cc_v9): Likewise.
	(movtf_cc_hq_v9): Likewise.
	(movtf_cc_v9): Likewise.
	(adddi3): Call gen_adddi3_sp32.
	(adddi3_insn_sp32): Rename to...
	(adddi3_sp32): ...this.  Accept only register_operand as operand #1
	and use CCCmode for the carry.
	(addx_extend_sp32): Use CCCmode for the carry.
	(addx_extend_sp64): Delete.
	(adddi3_extend_sp32): Use CCCmode for the carry.
	(cmp_plus patterns): Use CCNZ/CCXNZ mode and add C variants.
	(subdi3): Call gen_subdi3_sp32.
	(subdi3_insn_sp32): Rename to...
	(subdi3_sp32): ...this and use CCmode for the carry.
	(subx_extend_sp32): Use CCCmode for the carry.
	(subx_extend_sp64): Delete.
	(subdi3_extend_sp32): Use CCmode for the carry.
	(cmp_minus patterns): Use CCNZ/CCXNZ mode and add C variants.
	(negdi3): Call gen_negdi3_sp32.
	(negdi3_sp32): Use CCCmode for the carry.
	(cmp_neg patterns): Use CCNZ/CCXNZ mode and add C variants.
	(cmp_nz_ashift_1): Use CCNZ mode.
	(cmp_nz_set_ashift_1): Likewise.
	(ctrapsi4): Use comparison_operator predicate.
	(ctrapdi4): Likewise.
	(trapsi_insn): Use icc_comparison_operator predicate.
	(trapdi_insn): Likewise.
	(edge8 patterns): Use CCNZmode.
	(edge16 patterns): Likewise.
	(edge32 patterns): Likewise.

From-SVN: r240971

24 files changed, 1530 insertions, 868 deletions

diff --git a/gcc/ChangeLog b/gcc/ChangeLog
index e2cc1bb..b00cb2c 100644
--- a/gcc/ChangeLog
+++ b/gcc/ChangeLog
@@ -1,5 +1,102 @@
 2016-10-11  Eric Botcazou  <ebotcazou@adacore.com>
 
+	* config/sparc/sparc.opt (msubxc): New option.
+	* doc/invoke.texi (SPARC options): Document it and tidy up.
+	* doc/tm.texi.in (Condition Codes): Adjust SPARC example.
+	* doc/tm.texi: Regenerate.
+	* config/sparc/sparc-modes.def (CC_NOOV): Rename into...
+	(CCNZ): ...this.
+	(CCX_NOOV): Rename into...
+	(CCXNZ): ...this.
+	(CCC): New.
+	(CCXC): Likewise.
+	* config/sparc/predicates.m (fcc_register_operand): Simplify.
+	(fcc0_register_operand): Likewise.
+	(icc_register_operand): New.
+	(icc_or_fcc_register_operand): Simplify.
+	(nz_comparison_operator): New.
+	(c_comparison_operator): Likewise.
+	(noov_compare_operator): Rename into...
+	(icc_comparison_operator): ...this.  Use above predicates.
+	(noov_compare64_operator): Rename into...
+	(v9_comparison_operator): ...this and tidy up.
+	(fcc_comparison_operator): New.
+	(icc_or_fcc_comparison_operator): Likewise.
+	(v9_register_compare_operator): Rename info...
+	(v9_register_comparison_operator): ...this.
+	* config/sparc/sparc.c (TARGET_FIXED_CONDITION_CODE_REGS): Define.
+	(sparc_option_override): Remove redundant VIS masks and add MASK_SUBXC
+	for Niagara-7.
+	(sparc_fixed_condition_code_regs): New function.
+	(select_cc_mode): Remove ATTRIBUTE_UNUSED.  Adjust for CCNZ/CCXNZ
+	renaming and add support for CCC/CCXC.
+	(output_cbranch): Likewise.
+	(sparc_print_operand): Likewise.
+	(gen_v9_scc): Remove obsolete assertion.
+	(emit_scc_insn): Emit RTL directly for EQ and NE.  Add direct support
+	for EQ in DImode if TARGET_SUBXC.  Remove test on TARGET_VIS3 for GEU.
+	(output_cbcond): Remove bogus handling of CC modes.
+	(sparc_register_move_cost): Return 100 for NO_REGS.
+	* config/sparc/sparc.md (W): New mode iterator.
+	(length): Adjust for noov_compare64_operator renaming.
+	(cmpsi_sne): New instruction.
+	(cmpdi_sne): Likewise.
+	(seqdi_special): Delete.
+	(seqdi_special): Likewise.
+	(snesi<P:mode>_special): Likewise.
+	(snedi_special): Likewise.
+	(snedi_special_vis3): Likewise.
+	(snesi patterns): Use W iterator.
+	(snedi patterns): Likewise.  Add TARGET_SUBXC patterns.
+	(sltu patterns): Likewise.
+	(sgeu patterns): Likewise.
+	(scc splitter): Do not split GEU in DImode if TARGET_SUBXC.
+	(normal_branch): Use icc_comparison_operator predicate.
+	(inverted_branch): Likewise.
+	(cbcond_sp32): Use comparison_operator predicate.
+	(cbcond_sp64): Likewise.
+	(normal_int_branch_sp64): Adjust for renaming
+	(inverted_int_branch_sp64): Likewise.
+	(mov<I:mode>_cc_reg_sp64): Likewise.
+	(movsf_cc_reg_sp6): Likewise.
+	(movdf_cc_reg_sp64): Likewise.
+	(movtf_cc_reg_hq_sp64): Likewise.
+	(movtf_cc_reg_sp64): Likewise.
+	(mov<I:mode>_cc_v9): Use icc_or_fcc_comparison_operator predicate.
+	(movsf_cc_v9): Likewise.
+	(movdf_cc_v9): Likewise.
+	(movtf_cc_hq_v9): Likewise.
+	(movtf_cc_v9): Likewise.
+	(adddi3): Call gen_adddi3_sp32.
+	(adddi3_insn_sp32): Rename to...
+	(adddi3_sp32): ...this.  Accept only register_operand as operand #1
+	and use CCCmode for the carry.
+	(addx_extend_sp32): Use CCCmode for the carry.
+	(addx_extend_sp64): Delete.
+	(adddi3_extend_sp32): Use CCCmode for the carry.
+	(cmp_plus patterns): Use CCNZ/CCXNZ mode and add C variants.
+	(subdi3): Call gen_subdi3_sp32.
+	(subdi3_insn_sp32): Rename to...
+	(subdi3_sp32): ...this and use CCmode for the carry.
+	(subx_extend_sp32): Use CCCmode for the carry.
+	(subx_extend_sp64): Delete.
+	(subdi3_extend_sp32): Use CCmode for the carry.
+	(cmp_minus patterns): Use CCNZ/CCXNZ mode and add C variants.
+	(negdi3): Call gen_negdi3_sp32.
+	(negdi3_sp32): Use CCCmode for the carry.
+	(cmp_neg patterns): Use CCNZ/CCXNZ mode and add C variants.
+	(cmp_nz_ashift_1): Use CCNZ mode.
+	(cmp_nz_set_ashift_1): Likewise.
+	(ctrapsi4): Use comparison_operator predicate.
+	(ctrapdi4): Likewise.
+	(trapsi_insn): Use icc_comparison_operator predicate.
+	(trapdi_insn): Likewise.
+	(edge8 patterns): Use CCNZmode.
+	(edge16 patterns): Likewise.
+	(edge32 patterns): Likewise.
+
+2016-10-11  Eric Botcazou  <ebotcazou@adacore.com>
+
 	* config/visium/visium-modes.def (CC_NOOV): Rename into...
 	(CCNZ): ...this.
 	(CC_BTST): Rename into...
diff --git a/gcc/config/sparc/predicates.md b/gcc/config/sparc/predicates.md
index 787e049..dd57bf5 100644
--- a/gcc/config/sparc/predicates.md
+++ b/gcc/config/sparc/predicates.md
@@ -234,53 +234,23 @@
 
 ;; Return true if OP is a floating point condition code register.
 (define_predicate "fcc_register_operand"
-  (match_code "reg")
-{
-  if (mode != VOIDmode && mode != GET_MODE (op))
-    return false;
-  if (mode == VOIDmode
-      && (GET_MODE (op) != CCFPmode && GET_MODE (op) != CCFPEmode))
-    return false;
-
-#if 0 /* ??? 1 when %fcc0-3 are pseudos first.  See gen_compare_reg().  */
-  if (reg_renumber == 0)
-    return REGNO (op) >= FIRST_PSEUDO_REGISTER;
-  return REGNO_OK_FOR_CCFP_P (REGNO (op));
-#else
-  return ((unsigned) REGNO (op) - SPARC_FIRST_V9_FCC_REG) < 4;
-#endif
-})
+  (and (match_code "reg")
+       (match_test "((unsigned) REGNO (op) - SPARC_FIRST_V9_FCC_REG) < 4")))
 
 ;; Return true if OP is the floating point condition code register fcc0.
 (define_predicate "fcc0_register_operand"
-  (match_code "reg")
-{
-  if (mode != VOIDmode && mode != GET_MODE (op))
-    return false;
-  if (mode == VOIDmode
-      && (GET_MODE (op) != CCFPmode && GET_MODE (op) != CCFPEmode))
-    return false;
+  (and (match_code "reg")
+       (match_test "REGNO (op) == SPARC_FCC_REG")))
 
-  return REGNO (op) == SPARC_FCC_REG;
-})
+;; Return true if OP is an integer condition code register.
+(define_predicate "icc_register_operand"
+  (and (match_code "reg")
+       (match_test "REGNO (op) == SPARC_ICC_REG")))
 
 ;; Return true if OP is an integer or floating point condition code register.
 (define_predicate "icc_or_fcc_register_operand"
-  (match_code "reg")
-{
-  if (REGNO (op) == SPARC_ICC_REG)
-    {
-      if (mode != VOIDmode && mode != GET_MODE (op))
-	return false;
-      if (mode == VOIDmode
-	  && GET_MODE (op) != CCmode && GET_MODE (op) != CCXmode)
-	return false;
-
-      return true;
-    }
-
-  return fcc_register_operand (op, mode);
-})
+  (ior (match_operand 0 "icc_register_operand")
+       (match_operand 0 "fcc_register_operand")))
 
 
 ;; Predicates for arithmetic instructions.
@@ -442,46 +412,74 @@
 
 ;; Predicates for operators.
 
-;; Return true if OP is a comparison operator.  This allows the use of
-;; MATCH_OPERATOR to recognize all the branch insns.
-(define_predicate "noov_compare_operator"
-  (match_code "ne,eq,ge,gt,le,lt,geu,gtu,leu,ltu")
+;; Return true if OP is a valid comparison operator for CCNZmode.
+(define_predicate "nz_comparison_operator"
+  (match_code "eq,ne,lt,ge"))
+
+;; Return true if OP is a valid comparison operator for CCCmode.
+(define_predicate "c_comparison_operator"
+  (match_code "ltu,geu"))
+
+;; Return true if OP is an integer comparison operator.  This allows
+;; the use of MATCH_OPERATOR to recognize all the branch insns.
+(define_predicate "icc_comparison_operator"
+  (match_operand 0 "ordered_comparison_operator")
 {
-  enum rtx_code code = GET_CODE (op);
-  if (GET_MODE (XEXP (op, 0)) == CC_NOOVmode
-      || GET_MODE (XEXP (op, 0)) == CCX_NOOVmode)
-    /* These are the only branches which work with CC_NOOVmode.  */
-    return (code == EQ || code == NE || code == GE || code == LT);
-  return true;
+  switch (GET_MODE (XEXP (op, 0)))
+    {
+    case CCmode:
+    case CCXmode:
+      return true;
+    case CCNZmode:
+    case CCXNZmode:
+      return nz_comparison_operator (op, mode);
+    case CCCmode:
+    case CCXCmode:
+      return c_comparison_operator (op, mode);
+    default:
+      return false;
+    }
 })
 
-;; Return true if OP is a 64-bit comparison operator.  This allows the use of
-;; MATCH_OPERATOR to recognize all the branch insns.
-(define_predicate "noov_compare64_operator"
-  (and (match_code "ne,eq,ge,gt,le,lt,geu,gtu,leu,ltu")
-       (match_test "TARGET_V9"))
+;; Return true if OP is a FP comparison operator.
+(define_predicate "fcc_comparison_operator"
+  (match_operand 0 "comparison_operator")
 {
-  enum rtx_code code = GET_CODE (op);
-  if (GET_MODE (XEXP (op, 0)) == CCX_NOOVmode)
-    /* These are the only branches which work with CCX_NOOVmode.  */
-    return (code == EQ || code == NE || code == GE || code == LT);
-  return (GET_MODE (XEXP (op, 0)) == CCXmode);
+  switch (GET_MODE (XEXP (op, 0)))
+    {
+    case CCFPmode:
+    case CCFPEmode:
+      return true;
+    default:
+      return false;
+    }
 })
 
+;; Return true if OP is an integer or FP comparison operator.  This allows
+;; the use of MATCH_OPERATOR to recognize all the conditional move insns.
+(define_predicate "icc_or_fcc_comparison_operator"
+  (ior (match_operand 0 "icc_comparison_operator")
+       (match_operand 0 "fcc_comparison_operator")))
+
+;; Return true if OP is an integer comparison operator for V9.
+(define_predicate "v9_comparison_operator"
+  (and (match_operand 0 "ordered_comparison_operator")
+       (match_test "TARGET_V9")))
+
 ;; Return true if OP is a comparison operator suitable for use in V9
 ;; conditional move or branch on register contents instructions.
-(define_predicate "v9_register_compare_operator"
+(define_predicate "v9_register_comparison_operator"
   (match_code "eq,ne,ge,lt,le,gt"))
 
 ;; Return true if OP is an operator which can set the condition codes
-;; explicitly.  We do not include PLUS and MINUS because these
-;; require CC_NOOVmode, which we handle explicitly.
+;; explicitly.  We do not include PLUS/MINUS/NEG/ASHIFT because these
+;; require CCNZmode, which we handle explicitly.
 (define_predicate "cc_arith_operator"
   (match_code "and,ior,xor"))
 
 ;; Return true if OP is an operator which can bitwise complement its
 ;; second operand and set the condition codes explicitly.
 ;; XOR is not here because combine canonicalizes (xor (not ...) ...)
-;; and (xor ... (not ...)) to (not (xor ...)).  */
+;; and (xor ... (not ...)) to (not (xor ...)).
 (define_predicate "cc_arith_not_operator"
   (match_code "and,ior"))
diff --git a/gcc/config/sparc/sparc-modes.def b/gcc/config/sparc/sparc-modes.def
index 0324180..9c6fe03 100644
--- a/gcc/config/sparc/sparc-modes.def
+++ b/gcc/config/sparc/sparc-modes.def
@@ -1,7 +1,7 @@
 /* Definitions of target machine for GCC, for Sun SPARC.
    Copyright (C) 2002-2016 Free Software Foundation, Inc.
    Contributed by Michael Tiemann (tiemann@cygnus.com).
-   64 bit SPARC V9 support by Michael Tiemann, Jim Wilson, and Doug Evans,
+   64-bit SPARC V9 support by Michael Tiemann, Jim Wilson, and Doug Evans,
    at Cygnus Support.
 
 This file is part of GCC.
@@ -25,20 +25,27 @@ FLOAT_MODE (TF, 16, ieee_quad_format);
 
 /* Add any extra modes needed to represent the condition code.
 
-   On the SPARC, we have a "no-overflow" mode which is used when an add or
-   subtract insn is used to set the condition code.  Different branches are
-   used in this case for some operations.
+   We have a CCNZ mode which is used for implicit comparisons with zero when
+   arithmetic instructions set the condition code.  Only the N and Z flags
+   are valid in this mode, which means that only the =,!= and <,>= operators
+   can be used in conjunction with it.
+
+   We also have a CCCmode which is used by the arithmetic instructions when
+   they explicitly set the C flag (unsigned overflow).  Only the unsigned
+   <,>= operators can be used in conjunction with it.
 
    We also have two modes to indicate that the relevant condition code is
    in the floating-point condition code register.  One for comparisons which
    will generate an exception if the result is unordered (CCFPEmode) and
    one for comparisons which will never trap (CCFPmode).
 
-   CCXmode and CCX_NOOVmode are only used by v9.  */
+   CC modes are used for the 32-bit ICC, CCX modes for the 64-bit XCC.  */
 
 CC_MODE (CCX);
-CC_MODE (CC_NOOV);
-CC_MODE (CCX_NOOV);
+CC_MODE (CCNZ);
+CC_MODE (CCXNZ);
+CC_MODE (CCC);
+CC_MODE (CCXC);
 CC_MODE (CCFP);
 CC_MODE (CCFPE);
 
diff --git a/gcc/config/sparc/sparc.c b/gcc/config/sparc/sparc.c
index c1c196b..d8ceff9 100644
--- a/gcc/config/sparc/sparc.c
+++ b/gcc/config/sparc/sparc.c
@@ -647,6 +647,7 @@ static reg_class_t sparc_secondary_reload (bool, rtx, reg_class_t,
 					   secondary_reload_info *);
 static machine_mode sparc_cstore_mode (enum insn_code icode);
 static void sparc_atomic_assign_expand_fenv (tree *, tree *, tree *);
+static bool sparc_fixed_condition_code_regs (unsigned int *, unsigned int *);
 
 #ifdef SUBTARGET_ATTRIBUTE_TABLE
 /* Table of valid machine attributes.  */
@@ -857,6 +858,9 @@ char sparc_hard_reg_printed[8];
 #undef TARGET_ATOMIC_ASSIGN_EXPAND_FENV
 #define TARGET_ATOMIC_ASSIGN_EXPAND_FENV sparc_atomic_assign_expand_fenv
 
+#undef TARGET_FIXED_CONDITION_CODE_REGS
+#define TARGET_FIXED_CONDITION_CODE_REGS sparc_fixed_condition_code_regs
+
 struct gcc_target targetm = TARGET_INITIALIZER;
 
 /* Return the memory reference contained in X if any, zero otherwise.  */
@@ -1314,13 +1318,13 @@ sparc_option_override (void)
       MASK_V9|MASK_POPC|MASK_VIS2 },
     /* UltraSPARC T3 */
     { "niagara3",	MASK_ISA,
-      MASK_V9|MASK_POPC|MASK_VIS2|MASK_VIS3|MASK_FMAF },
+      MASK_V9|MASK_POPC|MASK_VIS3|MASK_FMAF },
     /* UltraSPARC T4 */
     { "niagara4",	MASK_ISA,
-      MASK_V9|MASK_POPC|MASK_VIS2|MASK_VIS3|MASK_FMAF|MASK_CBCOND },
+      MASK_V9|MASK_POPC|MASK_VIS3|MASK_FMAF|MASK_CBCOND },
     /* UltraSPARC M7 */
     { "niagara7",	MASK_ISA,
-      MASK_V9|MASK_POPC|MASK_VIS2|MASK_VIS3|MASK_VIS4|MASK_FMAF|MASK_CBCOND },
+      MASK_V9|MASK_POPC|MASK_VIS4|MASK_FMAF|MASK_CBCOND|MASK_SUBXC }
   };
   const struct cpu_table *cpu;
   unsigned int i;
@@ -1451,7 +1455,7 @@ sparc_option_override (void)
 		   & ~MASK_CBCOND
 #endif
 #ifndef HAVE_AS_SPARC5_VIS4
-		   & ~MASK_VIS4
+		   & ~(MASK_VIS4 | MASK_SUBXC)
 #endif
 #ifndef HAVE_AS_LEON
 		   & ~(MASK_LEON | MASK_LEON3)
@@ -2742,14 +2746,24 @@ sparc_emit_set_const64 (rtx op0, rtx op1)
   sparc_emit_set_const64_longway (op0, temp, high_bits, low_bits);
 }
 
+/* Implement TARGET_FIXED_CONDITION_CODE_REGS.  */
+
+static bool
+sparc_fixed_condition_code_regs (unsigned int *p1, unsigned int *p2)
+{
+  *p1 = SPARC_ICC_REG;
+  *p2 = SPARC_FCC_REG;
+  return true;
+}
+
 /* Given a comparison code (EQ, NE, etc.) and the first operand of a COMPARE,
    return the mode to be used for the comparison.  For floating-point,
-   CCFP[E]mode is used.  CC_NOOVmode should be used when the first operand
+   CCFP[E]mode is used.  CCNZmode should be used when the first operand
    is a PLUS, MINUS, NEG, or ASHIFT.  CCmode should be used when no special
    processing is needed.  */
 
 machine_mode
-select_cc_mode (enum rtx_code op, rtx x, rtx y ATTRIBUTE_UNUSED)
+select_cc_mode (enum rtx_code op, rtx x, rtx y)
 {
   if (GET_MODE_CLASS (GET_MODE (x)) == MODE_FLOAT)
     {
@@ -2781,12 +2795,21 @@ select_cc_mode (enum rtx_code op, rtx x, rtx y ATTRIBUTE_UNUSED)
 	   || GET_CODE (x) == NEG || GET_CODE (x) == ASHIFT)
     {
       if (TARGET_ARCH64 && GET_MODE (x) == DImode)
-	return CCX_NOOVmode;
+	return CCXNZmode;
       else
-	return CC_NOOVmode;
+	return CCNZmode;
     }
   else
     {
+      /* This is for the cmp<mode>_sne pattern.  */
+      if (GET_CODE (x) == NOT && y == constm1_rtx)
+	{
+	  if (TARGET_ARCH64 && GET_MODE (x) == DImode)
+	    return CCXCmode;
+	  else
+	    return CCCmode;
+	}
+
       if (TARGET_ARCH64 && GET_MODE (x) == DImode)
 	return CCXmode;
       else
@@ -2951,9 +2974,6 @@ gen_v9_scc (rtx dest, enum rtx_code compare_code, rtx x, rtx y)
       x = gen_compare_reg_1 (compare_code, x, y);
       y = const0_rtx;
 
-      gcc_assert (GET_MODE (x) != CC_NOOVmode
-		  && GET_MODE (x) != CCX_NOOVmode);
-
       emit_insn (gen_rtx_SET (dest, const0_rtx));
       emit_insn (gen_rtx_SET (dest,
 			  gen_rtx_IF_THEN_ELSE (GET_MODE (dest),
@@ -2971,10 +2991,9 @@ gen_v9_scc (rtx dest, enum rtx_code compare_code, rtx x, rtx y)
 bool
 emit_scc_insn (rtx operands[])
 {
-  rtx tem;
-  rtx x;
-  rtx y;
+  rtx tem, x, y;
   enum rtx_code code;
+  machine_mode mode;
 
   /* The quad-word fp compare library routines all return nonzero to indicate
      true, which is different from the equivalent libgcc routines, so we must
@@ -2990,59 +3009,42 @@ emit_scc_insn (rtx operands[])
   code = GET_CODE (operands[1]);
   x = operands[2];
   y = operands[3];
+  mode = GET_MODE (x);
 
   /* For seq/sne on v9 we use the same code as v8 (the addx/subx method has
      more applications).  The exception to this is "reg != 0" which can
      be done in one instruction on v9 (so we do it).  */
-  if (code == EQ)
+  if ((code == EQ || code == NE) && (mode == SImode || mode == DImode))
     {
-      if (GET_MODE (x) == SImode)
-        {
-	  rtx pat;
-	  if (TARGET_ARCH64)
-	    pat = gen_seqsidi_special (operands[0], x, y);
-	  else
-	    pat = gen_seqsisi_special (operands[0], x, y);
-          emit_insn (pat);
-          return true;
-        }
-      else if (GET_MODE (x) == DImode)
-        {
-	  rtx pat = gen_seqdi_special (operands[0], x, y);
-          emit_insn (pat);
-          return true;
-        }
-    }
+      if (y != const0_rtx)
+	x = force_reg (mode, gen_rtx_XOR (mode, x, y));
 
-  if (code == NE)
-    {
-      if (GET_MODE (x) == SImode)
-        {
-          rtx pat;
-	  if (TARGET_ARCH64)
-	    pat = gen_snesidi_special (operands[0], x, y);
-	  else
-	    pat = gen_snesisi_special (operands[0], x, y);
-          emit_insn (pat);
-          return true;
-        }
-      else if (GET_MODE (x) == DImode)
-        {
-	  rtx pat;
-	  if (TARGET_VIS3)
-	    pat = gen_snedi_special_vis3 (operands[0], x, y);
-	  else
-	    pat = gen_snedi_special (operands[0], x, y);
-          emit_insn (pat);
-          return true;
-        }
+      rtx pat = gen_rtx_SET (operands[0],
+			     gen_rtx_fmt_ee (code, GET_MODE (operands[0]),
+					     x, const0_rtx));
+
+      /* If we can use addx/subx or addxc/subxc, add a clobber for CC.  */
+      if (mode == SImode
+	  || (code == NE && TARGET_VIS3)
+	  || (code == EQ && TARGET_SUBXC))
+	{
+	  rtx clobber
+	    = gen_rtx_CLOBBER (VOIDmode,
+			       gen_rtx_REG (mode == SImode ? CCmode : CCXmode,
+					    SPARC_ICC_REG));
+	  pat = gen_rtx_PARALLEL (VOIDmode, gen_rtvec (2, pat, clobber));
+	}
+
+      emit_insn (pat);
+      return true;
     }
 
-  if (TARGET_V9
-      && TARGET_ARCH64
-      && GET_MODE (x) == DImode
-      && !(TARGET_VIS3
-	   && (code == GTU || code == LTU))
+  /* We can do LTU in DImode using the addxc instruction with VIS3
+     and GEU in DImode using the subxc instruction with SUBXC.  */
+  if (TARGET_ARCH64
+      && mode == DImode
+      && !((code == LTU || code == GTU) && TARGET_VIS3)
+      && !((code == GEU || code == LEU) && TARGET_SUBXC)
       && gen_v9_scc (operands[0], code, x, y))
     return true;
 
@@ -3061,8 +3063,7 @@ emit_scc_insn (rtx operands[])
         }
     }
 
-  if (code == LTU
-      || (!TARGET_VIS3 && code == GEU))
+  if (code == LTU || code == GEU)
     {
       emit_insn (gen_rtx_SET (operands[0],
 			      gen_rtx_fmt_ee (code, GET_MODE (operands[0]),
@@ -7715,7 +7716,6 @@ output_cbranch (rtx op, rtx dest, int label, int reversed, int annul,
 	case LTGT:
 	  branch = "fblg";
 	  break;
-
 	default:
 	  gcc_unreachable ();
 	}
@@ -7741,7 +7741,7 @@ output_cbranch (rtx op, rtx dest, int label, int reversed, int annul,
 	  branch = "be";
 	  break;
 	case GE:
-	  if (mode == CC_NOOVmode || mode == CCX_NOOVmode)
+	  if (mode == CCNZmode || mode == CCXNZmode)
 	    branch = "bpos";
 	  else
 	    branch = "bge";
@@ -7753,7 +7753,7 @@ output_cbranch (rtx op, rtx dest, int label, int reversed, int annul,
 	  branch = "ble";
 	  break;
 	case LT:
-	  if (mode == CC_NOOVmode || mode == CCX_NOOVmode)
+	  if (mode == CCNZmode || mode == CCXNZmode)
 	    branch = "bneg";
 	  else
 	    branch = "bl";
@@ -7770,7 +7770,6 @@ output_cbranch (rtx op, rtx dest, int label, int reversed, int annul,
 	case LTU:
 	  branch = "blu";
 	  break;
-
 	default:
 	  gcc_unreachable ();
 	}
@@ -7801,28 +7800,37 @@ output_cbranch (rtx op, rtx dest, int label, int reversed, int annul,
 	    v8 = 1;
 	}
 
-      if (mode == CCFPmode || mode == CCFPEmode)
-	{
-	  static char v9_fcc_labelno[] = "%%fccX, ";
-	  /* Set the char indicating the number of the fcc reg to use.  */
-	  v9_fcc_labelno[5] = REGNO (cc_reg) - SPARC_FIRST_V9_FCC_REG + '0';
-	  labelno = v9_fcc_labelno;
-	  if (v8)
-	    {
-	      gcc_assert (REGNO (cc_reg) == SPARC_FCC_REG);
-	      labelno = "";
-	    }
-	}
-      else if (mode == CCXmode || mode == CCX_NOOVmode)
-	{
-	  labelno = "%%xcc, ";
-	  gcc_assert (! v8);
-	}
-      else
+      switch (mode)
 	{
+	case CCmode:
+	case CCNZmode:
+	case CCCmode:
 	  labelno = "%%icc, ";
 	  if (v8)
 	    labelno = "";
+	  break;
+	case CCXmode:
+	case CCXNZmode:
+	case CCXCmode:
+	  labelno = "%%xcc, ";
+	  gcc_assert (!v8);
+	  break;
+	case CCFPmode:
+	case CCFPEmode:
+	  {
+	    static char v9_fcc_labelno[] = "%%fccX, ";
+	    /* Set the char indicating the number of the fcc reg to use.  */
+	    v9_fcc_labelno[5] = REGNO (cc_reg) - SPARC_FIRST_V9_FCC_REG + '0';
+	    labelno = v9_fcc_labelno;
+	    if (v8)
+	      {
+		gcc_assert (REGNO (cc_reg) == SPARC_FCC_REG);
+		labelno = "";
+	      }
+	  }
+	  break;
+	default:
+	  gcc_unreachable ();
 	}
 
       if (*labelno && insn && (note = find_reg_note (insn, REG_BR_PROB, NULL_RTX)))
@@ -8129,10 +8137,7 @@ output_cbcond (rtx op, rtx dest, rtx_insn *insn)
       break;
 
     case GE:
-      if (mode == CC_NOOVmode || mode == CCX_NOOVmode)
-	cond_str = "pos";
-      else
-	cond_str = "ge";
+      cond_str = "ge";
       break;
 
     case GT:
@@ -8144,10 +8149,7 @@ output_cbcond (rtx op, rtx dest, rtx_insn *insn)
       break;
 
     case LT:
-      if (mode == CC_NOOVmode || mode == CCX_NOOVmode)
-	cond_str = "neg";
-      else
-	cond_str = "l";
+      cond_str = "l";
       break;
 
     case GEU:
@@ -8699,6 +8701,8 @@ sparc_print_operand_punct_valid_p (unsigned char code)
 static void
 sparc_print_operand (FILE *file, rtx x, int code)
 {
+  const char *s;
+
   switch (code)
     {
     case '#':
@@ -8806,14 +8810,22 @@ sparc_print_operand (FILE *file, rtx x, int code)
       /* Print a condition code register.  */
       if (REGNO (x) == SPARC_ICC_REG)
 	{
-	  /* We don't handle CC[X]_NOOVmode because they're not supposed
-	     to occur here.  */
-	  if (GET_MODE (x) == CCmode)
-	    fputs ("%icc", file);
-	  else if (GET_MODE (x) == CCXmode)
-	    fputs ("%xcc", file);
-	  else
-	    gcc_unreachable ();
+	  switch (GET_MODE (x))
+	    {
+	    case CCmode:
+	    case CCNZmode:
+	    case CCCmode:
+	      s = "%icc";
+	      break;
+	    case CCXmode:
+	    case CCXNZmode:
+	    case CCXCmode:
+	      s = "%xcc";
+	      break;
+	    default:
+	      gcc_unreachable ();
+	    }
+	  fputs (s, file);
 	}
       else
 	/* %fccN register */
@@ -8838,67 +8850,147 @@ sparc_print_operand (FILE *file, rtx x, int code)
     case 'A':
       switch (GET_CODE (x))
 	{
-	case IOR: fputs ("or", file); break;
-	case AND: fputs ("and", file); break;
-	case XOR: fputs ("xor", file); break;
-	default: output_operand_lossage ("invalid %%A operand");
+	case IOR:
+	  s = "or";
+	  break;
+	case AND:
+	  s = "and";
+	  break;
+	case XOR:
+	  s = "xor";
+	  break;
+	default:
+	  output_operand_lossage ("invalid %%A operand");
+	  s = "";
+	  break;
 	}
+      fputs (s, file);
       return;
 
     case 'B':
       switch (GET_CODE (x))
 	{
-	case IOR: fputs ("orn", file); break;
-	case AND: fputs ("andn", file); break;
-	case XOR: fputs ("xnor", file); break;
-	default: output_operand_lossage ("invalid %%B operand");
+	case IOR:
+	  s = "orn";
+	  break;
+	case AND:
+	  s = "andn";
+	  break;
+	case XOR:
+	  s = "xnor";
+	  break;
+	default:
+	  output_operand_lossage ("invalid %%B operand");
+	  s = "";
+	  break;
 	}
+      fputs (s, file);
       return;
 
       /* This is used by the conditional move instructions.  */
     case 'C':
       {
-	enum rtx_code rc = GET_CODE (x);
-	
-	switch (rc)
+	machine_mode mode = GET_MODE (XEXP (x, 0));
+	switch (GET_CODE (x))
 	  {
-	  case NE: fputs ("ne", file); break;
-	  case EQ: fputs ("e", file); break;
-	  case GE: fputs ("ge", file); break;
-	  case GT: fputs ("g", file); break;
-	  case LE: fputs ("le", file); break;
-	  case LT: fputs ("l", file); break;
-	  case GEU: fputs ("geu", file); break;
-	  case GTU: fputs ("gu", file); break;
-	  case LEU: fputs ("leu", file); break;
-	  case LTU: fputs ("lu", file); break;
-	  case LTGT: fputs ("lg", file); break;
-	  case UNORDERED: fputs ("u", file); break;
-	  case ORDERED: fputs ("o", file); break;
-	  case UNLT: fputs ("ul", file); break;
-	  case UNLE: fputs ("ule", file); break;
-	  case UNGT: fputs ("ug", file); break;
-	  case UNGE: fputs ("uge", file); break;
-	  case UNEQ: fputs ("ue", file); break;
-	  default: output_operand_lossage ("invalid %%C operand");
+	  case NE:
+	    s = "ne";
+	    break;
+	  case EQ:
+	    s = "e";
+	    break;
+	  case GE:
+	    if (mode == CCNZmode || mode == CCXNZmode)
+	      s = "pos";
+	    else
+	      s = "ge";
+	    break;
+	  case GT:
+	    s = "g";
+	    break;
+	  case LE:
+	    s = "le";
+	    break;
+	  case LT:
+	    if (mode == CCNZmode || mode == CCXNZmode)
+	      s = "neg";
+	    else
+	      s = "l";
+	    break;
+	  case GEU:
+	    s = "geu";
+	    break;
+	  case GTU:
+	    s = "gu";
+	    break;
+	  case LEU:
+	    s = "leu";
+	    break;
+	  case LTU:
+	    s = "lu";
+	    break;
+	  case LTGT:
+	    s = "lg";
+	    break;
+	  case UNORDERED:
+	    s = "u";
+	    break;
+	  case ORDERED:
+	    s = "o";
+	    break;
+	  case UNLT:
+	    s = "ul";
+	    break;
+	  case UNLE:
+	    s = "ule";
+	    break;
+	  case UNGT:
+	    s = "ug";
+	    break;
+	  case UNGE:
+	    s = "uge"
+	    ; break;
+	  case UNEQ:
+	    s = "ue";
+	    break;
+	  default:
+	    output_operand_lossage ("invalid %%C operand");
+	    s = "";
+	    break;
 	  }
+	fputs (s, file);
 	return;
       }
 
       /* This are used by the movr instruction pattern.  */
     case 'D':
       {
-	enum rtx_code rc = GET_CODE (x);
-	switch (rc)
+	switch (GET_CODE (x))
 	  {
-	  case NE: fputs ("ne", file); break;
-	  case EQ: fputs ("e", file); break;
-	  case GE: fputs ("gez", file); break;
-	  case LT: fputs ("lz", file); break;
-	  case LE: fputs ("lez", file); break;
-	  case GT: fputs ("gz", file); break;
-	  default: output_operand_lossage ("invalid %%D operand");
+	  case NE:
+	    s = "ne";
+	    break;
+	  case EQ:
+	    s = "e";
+	    break;
+	  case GE:
+	    s = "gez";
+	    break;
+	  case LT:
+	    s = "lz";
+	    break;
+	  case LE:
+	    s = "lez";
+	    break;
+	  case GT:
+	    s = "gz";
+	    break;
+	  default:
+	    output_operand_lossage ("invalid %%D operand");
+	    s = "";
+	    break;
 	  }
+	fputs (s, file);
 	return;
       }
 
@@ -11238,6 +11330,10 @@ sparc_register_move_cost (machine_mode mode ATTRIBUTE_UNUSED,
 {
   bool need_memory = false;
 
+  /* This helps postreload CSE to eliminate redundant comparisons.  */
+  if (from == NO_REGS || to == NO_REGS)
+    return 100;
+
   if (from == FPCC_REGS || to == FPCC_REGS)
     need_memory = true;
   else if ((FP_REG_CLASS_P (from) && general_or_i64_p (to))
diff --git a/gcc/config/sparc/sparc.h b/gcc/config/sparc/sparc.h
index 4674c30..5389216 100644
--- a/gcc/config/sparc/sparc.h
+++ b/gcc/config/sparc/sparc.h
@@ -1517,7 +1517,7 @@ do {									   \
 
 /* Given a comparison code (EQ, NE, etc.) and the first operand of a COMPARE,
    return the mode to be used for the comparison.  For floating-point,
-   CCFP[E]mode is used.  CC_NOOVmode should be used when the first operand
+   CCFP[E]mode is used.  CCNZmode should be used when the first operand
    is a PLUS, MINUS, NEG, or ASHIFT.  CCmode should be used when no special
    processing is needed.  */
 #define SELECT_CC_MODE(OP,X,Y)  select_cc_mode ((OP), (X), (Y))
diff --git a/gcc/config/sparc/sparc.md b/gcc/config/sparc/sparc.md
index 29e4966..0540745 100644
--- a/gcc/config/sparc/sparc.md
+++ b/gcc/config/sparc/sparc.md
@@ -203,6 +203,7 @@
 
 (define_mode_iterator P [(SI "Pmode == SImode") (DI "Pmode == DImode")])
 (define_mode_iterator I [QI HI SI DI])
+(define_mode_iterator W [SI (DI "TARGET_ARCH64")])
 (define_mode_iterator F [SF DF TF])
 
 ;; The upper 32 fp regs on the v9 can't hold SFmode values.  To deal with this
@@ -338,7 +339,7 @@
 	       (const_int 2)
 	       (const_int 1)))
 	 (eq_attr "branch_type" "icc")
-	   (if_then_else (match_operand 0 "noov_compare64_operator" "")
+	   (if_then_else (match_operand 0 "v9_comparison_operator" "")
 	     (if_then_else (lt (pc) (match_dup 1))
 	       (if_then_else (lt (minus (match_dup 1) (pc)) (const_int 260000))
 		 (if_then_else (eq_attr "empty_delay_slot" "true")
@@ -525,8 +526,6 @@
 ;; DEFINE_SPLITs for some of the scc insns that actually require
 ;; more than one machine instruction.  DEFINE_EXPANDs are further down.
 
-;; The compare DEFINE_INSNs.
-
 (define_insn "*cmpsi_insn"
   [(set (reg:CC CC_REG)
 	(compare:CC (match_operand:SI 0 "register_operand" "r")
@@ -543,6 +542,22 @@
   "cmp\t%0, %1"
   [(set_attr "type" "compare")])
 
+(define_insn "*cmpsi_sne"
+  [(set (reg:CCC CC_REG)
+	(compare:CCC (not:SI (match_operand:SI 0 "arith_operand" "rI"))
+		     (const_int -1)))]
+  ""
+  "cmp\t%%g0, %0"
+  [(set_attr "type" "compare")])
+
+(define_insn "*cmpdi_sne"
+  [(set (reg:CCXC CC_REG)
+	(compare:CCXC (not:DI (match_operand:DI 0 "arith_operand" "rI"))
+		      (const_int -1)))]
+  "TARGET_ARCH64"
+  "cmp\t%%g0, %0"
+  [(set_attr "type" "compare")])
+
 (define_insn "*cmpsf_fpe"
   [(set (match_operand:CCFPE 0 "fcc_register_operand" "=c")
 	(compare:CCFPE (match_operand:SF 1 "register_operand" "f")
@@ -652,497 +667,671 @@
           (match_operand:F 3 "register_operand" "")]))
    (clobber (match_operand:SI 0 "cstore_result_operand"))]
   "TARGET_FPU"
-  { if (emit_scc_insn (operands)) DONE; else FAIL; })
-
-;; Seq_special[_xxx] and sne_special[_xxx] clobber the CC reg, because they
-;; generate addcc/subcc instructions.
-
-(define_expand "seqsi<P:mode>_special"
-  [(set (match_dup 3)
-	(xor:SI (match_operand:SI 1 "register_operand" "")
-		(match_operand:SI 2 "register_operand" "")))
-   (parallel [(set (match_operand:P 0 "register_operand" "")
-		   (eq:P (match_dup 3) (const_int 0)))
-	      (clobber (reg:CC CC_REG))])]
-  ""
-  { operands[3] = gen_reg_rtx (SImode); })
-
-(define_expand "seqdi_special"
-  [(set (match_dup 3)
-	(xor:DI (match_operand:DI 1 "register_operand" "")
-		(match_operand:DI 2 "register_operand" "")))
-   (set (match_operand:DI 0 "register_operand" "")
-	(eq:DI (match_dup 3) (const_int 0)))]
-  "TARGET_ARCH64"
-  { operands[3] = gen_reg_rtx (DImode); })
-
-(define_expand "snesi<P:mode>_special"
-  [(set (match_dup 3)
-	(xor:SI (match_operand:SI 1 "register_operand" "")
-		(match_operand:SI 2 "register_operand" "")))
-   (parallel [(set (match_operand:P 0 "register_operand" "")
-		   (ne:P (match_dup 3) (const_int 0)))
-	      (clobber (reg:CC CC_REG))])]
-  ""
-  { operands[3] = gen_reg_rtx (SImode); })
-
-(define_expand "snedi_special"
-  [(set (match_dup 3)
-	(xor:DI (match_operand:DI 1 "register_operand" "")
-		(match_operand:DI 2 "register_operand" "")))
-   (set (match_operand:DI 0 "register_operand" "")
-	(ne:DI (match_dup 3) (const_int 0)))]
-  "TARGET_ARCH64 && ! TARGET_VIS3"
-  { operands[3] = gen_reg_rtx (DImode); })
-
-(define_expand "snedi_special_vis3"
-  [(set (match_dup 3)
-	(xor:DI (match_operand:DI 1 "register_operand" "")
-		(match_operand:DI 2 "register_operand" "")))
-   (parallel [(set (match_operand:DI 0 "register_operand" "")
-		   (ne:DI (match_dup 3) (const_int 0)))
-	      (clobber (reg:CCX CC_REG))])]
-  "TARGET_ARCH64 && TARGET_VIS3"
-  { operands[3] = gen_reg_rtx (DImode); })
-
-
-;; Now the DEFINE_INSNs for the scc cases.
+{
+  if (emit_scc_insn (operands)) DONE; else FAIL;
+})
 
-;; The SEQ and SNE patterns are special because they can be done
-;; without any branching and do not involve a COMPARE.  We want
-;; them to always use the splits below so the results can be
-;; scheduled.
+;; The SNE and SEQ patterns are special because they can be done
+;; without any branching and do not involve a COMPARE.
 
-(define_insn_and_split "*snesi<P:mode>_zero"
-  [(set (match_operand:P 0 "register_operand" "=r")
-	(ne:P (match_operand:SI 1 "register_operand" "r")
-	       (const_int 0)))
+(define_insn_and_split "*snesi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(ne:W (match_operand:SI 1 "register_operand" "r")
+	      (const_int 0)))
    (clobber (reg:CC CC_REG))]
   ""
   "#"
   ""
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (ltu:P (reg:CC CC_REG) (const_int 0)))]
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (ltu:W (reg:CCC CC_REG) (const_int 0)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*neg_snesisi_zero"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(neg:SI (ne:SI (match_operand:SI 1 "register_operand" "r")
-		       (const_int 0))))
+(define_insn_and_split "*neg_snesi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(neg:W (ne:W (match_operand:SI 1 "register_operand" "r")
+		     (const_int 0))))
    (clobber (reg:CC CC_REG))]
   ""
   "#"
   ""
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (neg:SI (ltu:SI (reg:CC CC_REG) (const_int 0))))]
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (neg:W (ltu:W (reg:CCC CC_REG) (const_int 0))))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*neg_snesidi_zero"
-  [(set (match_operand:DI 0 "register_operand" "=r")
-	(neg:DI (ne:DI (match_operand:SI 1 "register_operand" "r")
-		       (const_int 0))))
-   (clobber (reg:CC CC_REG))]
-  "TARGET_ARCH64"
+(define_insn_and_split "*snedi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=&r")
+        (ne:W (match_operand:DI 1 "register_operand" "r")
+              (const_int 0)))]
+  "TARGET_ARCH64 && !TARGET_VIS3"
   "#"
+  "&& ! reg_overlap_mentioned_p (operands[1], operands[0])"
+  [(set (match_dup 0) (const_int 0))
+   (set (match_dup 0) (if_then_else:W (ne:DI (match_dup 1) (const_int 0))
+                                      (const_int 1)
+                                      (match_dup 0)))]
   ""
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (sign_extend:DI (neg:SI (ltu:SI (reg:CC CC_REG)
-                                                      (const_int 0)))))]
+  [(set_attr "length" "2")])
+
+(define_insn_and_split "*snedi<W:mode>_zero_vis3"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(ne:W (match_operand:DI 1 "register_operand" "r")
+	      (const_int 0)))
+   (clobber (reg:CCX CC_REG))]
+  "TARGET_ARCH64 && TARGET_VIS3"
+  "#"
+  ""
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (ltu:W (reg:CCXC CC_REG) (const_int 0)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*snedi_zero"
-  [(set (match_operand:DI 0 "register_operand" "=&r")
-        (ne:DI (match_operand:DI 1 "register_operand" "r")
-               (const_int 0)))]
-  "TARGET_ARCH64 && ! TARGET_VIS3"
+(define_insn_and_split "*neg_snedi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=&r")
+        (neg:W (ne:W (match_operand:DI 1 "register_operand" "r")
+                     (const_int 0))))]
+  "TARGET_ARCH64 && !TARGET_SUBXC"
   "#"
   "&& ! reg_overlap_mentioned_p (operands[1], operands[0])"
   [(set (match_dup 0) (const_int 0))
-   (set (match_dup 0) (if_then_else:DI (ne:DI (match_dup 1)
-                                              (const_int 0))
-                                       (const_int 1)
-                                       (match_dup 0)))]
+   (set (match_dup 0) (if_then_else:W (ne:DI (match_dup 1) (const_int 0))
+                                      (const_int -1)
+                                      (match_dup 0)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*snedi_zero_vis3"
-  [(set (match_operand:DI 0 "register_operand" "=r")
-	(ne:DI (match_operand:DI 1 "register_operand" "r")
-	       (const_int 0)))
+(define_insn_and_split "*neg_snedi<W:mode>_zero_subxc"
+  [(set (match_operand:W 0 "register_operand" "=&r")
+        (neg:W (ne:W (match_operand:DI 1 "register_operand" "r")
+                     (const_int 0))))
    (clobber (reg:CCX CC_REG))]
-  "TARGET_ARCH64 && TARGET_VIS3"
+  "TARGET_ARCH64 && TARGET_SUBXC"
   "#"
   ""
-  [(set (reg:CCX_NOOV CC_REG) (compare:CCX_NOOV (neg:DI (match_dup 1))
-					        (const_int 0)))
-   (set (match_dup 0) (ltu:DI (reg:CCX CC_REG) (const_int 0)))]
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (neg:W (ltu:W (reg:CCXC CC_REG) (const_int 0))))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*neg_snedi_zero"
-  [(set (match_operand:DI 0 "register_operand" "=&r")
-        (neg:DI (ne:DI (match_operand:DI 1 "register_operand" "r")
-                       (const_int 0))))]
-  "TARGET_ARCH64"
+(define_insn_and_split "*seqsi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(eq:W (match_operand:SI 1 "register_operand" "r")
+	      (const_int 0)))
+   (clobber (reg:CC CC_REG))]
+  ""
   "#"
-  "&& ! reg_overlap_mentioned_p (operands[1], operands[0])"
-  [(set (match_dup 0) (const_int 0))
-   (set (match_dup 0) (if_then_else:DI (ne:DI (match_dup 1)
-                                              (const_int 0))
-                                       (const_int -1)
-                                       (match_dup 0)))]
+  ""
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (geu:W (reg:CCC CC_REG) (const_int 0)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*snedi_zero_trunc"
-  [(set (match_operand:SI 0 "register_operand" "=&r")
-        (ne:SI (match_operand:DI 1 "register_operand" "r")
-               (const_int 0)))]
-  "TARGET_ARCH64 && ! TARGET_VIS3"
+(define_insn_and_split "*neg_seqsi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(neg:W (eq:W (match_operand:SI 1 "register_operand" "r")
+		     (const_int 0))))
+   (clobber (reg:CC CC_REG))]
+  ""
+  "#"
+  ""
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (neg:W (geu:W (reg:CCC CC_REG) (const_int 0))))]
+  ""
+  [(set_attr "length" "2")])
+
+(define_insn_and_split "*seqdi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=&r")
+        (eq:W (match_operand:DI 1 "register_operand" "r")
+              (const_int 0)))]
+  "TARGET_ARCH64 && !TARGET_SUBXC"
   "#"
   "&& ! reg_overlap_mentioned_p (operands[1], operands[0])"
   [(set (match_dup 0) (const_int 0))
-   (set (match_dup 0) (if_then_else:SI (ne:DI (match_dup 1)
-                                              (const_int 0))
-                                       (const_int 1)
-                                       (match_dup 0)))]
+   (set (match_dup 0) (if_then_else:W (eq:DI (match_dup 1) (const_int 0))
+                                      (const_int 1)
+                                      (match_dup 0)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*snedi_zero_trunc_vis3"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(ne:SI (match_operand:DI 1 "register_operand" "r")
-	       (const_int 0)))
+(define_insn_and_split "*seqdi<W:mode>_zero_subxc"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(eq:W (match_operand:DI 1 "register_operand" "r")
+	      (const_int 0)))
    (clobber (reg:CCX CC_REG))]
-  "TARGET_ARCH64 && TARGET_VIS3"
+  "TARGET_ARCH64 && TARGET_SUBXC"
   "#"
   ""
-  [(set (reg:CCX_NOOV CC_REG) (compare:CCX_NOOV (neg:DI (match_dup 1))
-					        (const_int 0)))
-   (set (match_dup 0) (ltu:SI (reg:CCX CC_REG) (const_int 0)))]
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (geu:W (reg:CCXC CC_REG) (const_int 0)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*seqsi<P:mode>_zero"
-  [(set (match_operand:P 0 "register_operand" "=r")
-	(eq:P (match_operand:SI 1 "register_operand" "r")
-	       (const_int 0)))
-   (clobber (reg:CC CC_REG))]
+(define_insn_and_split "*neg_seqdi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=&r")
+        (neg:W (eq:W (match_operand:DI 1 "register_operand" "r")
+                     (const_int 0))))]
+  "TARGET_ARCH64 && !TARGET_VIS3"
+  "#"
+  "&& ! reg_overlap_mentioned_p (operands[1], operands[0])"
+  [(set (match_dup 0) (const_int 0))
+   (set (match_dup 0) (if_then_else:W (eq:DI (match_dup 1) (const_int 0))
+                                      (const_int -1)
+                                      (match_dup 0)))]
   ""
+  [(set_attr "length" "2")]) 
+
+(define_insn_and_split "*neg_seqdi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(neg:W (eq:W (match_operand:DI 1 "register_operand" "r")
+		     (const_int 0))))
+   (clobber (reg:CCX CC_REG))]
+  "TARGET_ARCH64 && TARGET_VIS3"
   "#"
   ""
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (geu:P (reg:CC CC_REG) (const_int 0)))]
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (neg:W (geu:W (reg:CCXC CC_REG) (const_int 0))))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*neg_seqsisi_zero"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(neg:SI (eq:SI (match_operand:SI 1 "register_operand" "r")
-		       (const_int 0))))
+;; We can also do (x + (i == 0)) and related, so put them in.
+
+(define_insn_and_split "*plus_snesi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (ne:W (match_operand:SI 1 "register_operand" "r")
+		      (const_int 0))
+		(match_operand:W 2 "register_operand" "r")))
    (clobber (reg:CC CC_REG))]
   ""
   "#"
   ""
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (neg:SI (geu:SI (reg:CC CC_REG) (const_int 0))))]
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (plus:W (ltu:W (reg:CCC CC_REG) (const_int 0))
+			      (match_dup 2)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*neg_seqsidi_zero"
-  [(set (match_operand:DI 0 "register_operand" "=r")
-	(neg:DI (eq:DI (match_operand:SI 1 "register_operand" "r")
-		       (const_int 0))))
+(define_insn_and_split "*plus_plus_snesi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (plus:W (ne:W (match_operand:SI 1 "register_operand" "r")
+			      (const_int 0))
+			(match_operand:W 2 "register_operand" "r"))
+                 (match_operand:W 3 "register_operand" "r")))
    (clobber (reg:CC CC_REG))]
-  "TARGET_ARCH64"
+  ""
   "#"
-  "&& 1"
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (sign_extend:DI (neg:SI (geu:SI (reg:CC CC_REG)
-                                                      (const_int 0)))))]
+  ""
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (plus:W (plus:W (ltu:W (reg:CCC CC_REG) (const_int 0))
+				      (match_dup 2))
+		      (match_dup 3)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*seqdi_zero"
-  [(set (match_operand:DI 0 "register_operand" "=&r")
-        (eq:DI (match_operand:DI 1 "register_operand" "r")
-               (const_int 0)))]
-  "TARGET_ARCH64"
+(define_insn_and_split "*plus_snedi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (ne:W (match_operand:DI 1 "register_operand" "r")
+		      (const_int 0))
+		(match_operand:W 2 "register_operand" "r")))
+   (clobber (reg:CCX CC_REG))]
+  "TARGET_ARCH64 && TARGET_VIS3"
   "#"
-  "&& ! reg_overlap_mentioned_p (operands[1], operands[0])"
-  [(set (match_dup 0) (const_int 0))
-   (set (match_dup 0) (if_then_else:DI (eq:DI (match_dup 1)
-                                              (const_int 0))
-                                       (const_int 1)
-                                       (match_dup 0)))]
+  ""
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (plus:W (ltu:W (reg:CCXC CC_REG) (const_int 0))
+			      (match_dup 2)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*neg_seqdi_zero"
-  [(set (match_operand:DI 0 "register_operand" "=&r")
-        (neg:DI (eq:DI (match_operand:DI 1 "register_operand" "r")
-                       (const_int 0))))]
-  "TARGET_ARCH64"
+(define_insn_and_split "*plus_plus_snedi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (plus:W (ne:W (match_operand:DI 1 "register_operand" "r")
+			      (const_int 0))
+			(match_operand:W 2 "register_operand" "r"))
+                 (match_operand:W 3 "register_operand" "r")))
+   (clobber (reg:CCX CC_REG))]
+  "TARGET_ARCH64 && TARGET_VIS3"
   "#"
-  "&& ! reg_overlap_mentioned_p (operands[1], operands[0])"
-  [(set (match_dup 0) (const_int 0))
-   (set (match_dup 0) (if_then_else:DI (eq:DI (match_dup 1)
-                                              (const_int 0))
-                                       (const_int -1)
-                                       (match_dup 0)))]
   ""
-  [(set_attr "length" "2")]) 
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (plus:W (plus:W (ltu:W (reg:CCXC CC_REG) (const_int 0))
+				      (match_dup 2))
+		      (match_dup 3)))]
+  ""
+  [(set_attr "length" "2")])
 
-(define_insn_and_split "*seqdi_zero_trunc"
-  [(set (match_operand:SI 0 "register_operand" "=&r")
-        (eq:SI (match_operand:DI 1 "register_operand" "r")
-               (const_int 0)))]
-  "TARGET_ARCH64"
+(define_insn_and_split "*minus_snesi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (match_operand:W 2 "register_operand" "r")
+		  (ne:W (match_operand:SI 1 "register_operand" "r")
+			(const_int 0))))
+   (clobber (reg:CC CC_REG))]
+  ""
   "#"
-  "&& ! reg_overlap_mentioned_p (operands[1], operands[0])"
-  [(set (match_dup 0) (const_int 0))
-   (set (match_dup 0) (if_then_else:SI (eq:DI (match_dup 1)
-                                              (const_int 0))
-                                       (const_int 1)
-                                       (match_dup 0)))]
+  ""
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (minus:W (match_dup 2)
+			       (ltu:W (reg:CCC CC_REG) (const_int 0))))]
   ""
   [(set_attr "length" "2")])
 
-;; We can also do (x + (i == 0)) and related, so put them in.
-;; ??? The addx/subx insns use the 32 bit carry flag so there are no DImode
-;; versions for v9.
-
-(define_insn_and_split "*x_plus_i_ne_0"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(plus:SI (ne:SI (match_operand:SI 1 "register_operand" "r")
-			(const_int 0))
-		 (match_operand:SI 2 "register_operand" "r")))
+(define_insn_and_split "*minus_minus_snesi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (minus:W (match_operand:W 2 "register_operand" "r")
+			  (ne:W (match_operand:SI 1 "register_operand" "r")
+				(const_int 0)))
+		 (match_operand:W 3 "register_operand" "r")))
    (clobber (reg:CC CC_REG))]
   ""
   "#"
   ""
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (plus:SI (ltu:SI (reg:CC CC_REG) (const_int 0))
-			       (match_dup 2)))]
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (minus:W (minus:W (match_dup 2)
+				        (ltu:W (reg:CCC CC_REG) (const_int 0)))
+			       (match_dup 3)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*x_minus_i_ne_0"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(minus:SI (match_operand:SI 2 "register_operand" "r")
-		  (ne:SI (match_operand:SI 1 "register_operand" "r")
-			 (const_int 0))))
-   (clobber (reg:CC CC_REG))]
+(define_insn_and_split "*minus_snedi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (match_operand:W 2 "register_operand" "r")
+		  (ne:W (match_operand:DI 1 "register_operand" "r")
+			(const_int 0))))
+   (clobber (reg:CCX CC_REG))]
+  "TARGET_ARCH64 && TARGET_SUBXC"
+  "#"
+  ""
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (minus:W (match_dup 2)
+			       (ltu:W (reg:CCXC CC_REG) (const_int 0))))]
   ""
+  [(set_attr "length" "2")])
+
+(define_insn_and_split "*minus_minus_snedi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (minus:W (match_operand:W 2 "register_operand" "r")
+			  (ne:W (match_operand:DI 1 "register_operand" "r")
+				(const_int 0)))
+		 (match_operand:W 3 "register_operand" "r")))
+   (clobber (reg:CCX CC_REG))]
+  "TARGET_ARCH64 && TARGET_SUBXC"
   "#"
   ""
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (minus:SI (match_dup 2)
-				(ltu:SI (reg:CC CC_REG) (const_int 0))))]
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (minus:W (minus:W (match_dup 2)
+				        (ltu:W (reg:CCXC CC_REG) (const_int 0)))
+			       (match_dup 3)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*x_plus_i_eq_0"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(plus:SI (eq:SI (match_operand:SI 1 "register_operand" "r")
-			(const_int 0))
-		 (match_operand:SI 2 "register_operand" "r")))
+(define_insn_and_split "*plus_seqsi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (eq:W (match_operand:SI 1 "register_operand" "r")
+		      (const_int 0))
+		(match_operand:W 2 "register_operand" "r")))
    (clobber (reg:CC CC_REG))]
   ""
   "#"
   ""
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (plus:SI (geu:SI (reg:CC CC_REG) (const_int 0))
-			       (match_dup 2)))]
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (plus:W (geu:W (reg:CCC CC_REG) (const_int 0))
+			      (match_dup 2)))]
   ""
   [(set_attr "length" "2")])
 
-(define_insn_and_split "*x_minus_i_eq_0"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(minus:SI (match_operand:SI 2 "register_operand" "r")
-		  (eq:SI (match_operand:SI 1 "register_operand" "r")
-			 (const_int 0))))
+(define_insn_and_split "*plus_seqdi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (eq:W (match_operand:DI 1 "register_operand" "r")
+		      (const_int 0))
+		(match_operand:W 2 "register_operand" "r")))
+   (clobber (reg:CCX CC_REG))]
+  "TARGET_ARCH64 && TARGET_SUBXC"
+  "#"
+  ""
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (plus:W (geu:W (reg:CCXC CC_REG) (const_int 0))
+			      (match_dup 2)))]
+  ""
+  [(set_attr "length" "2")])
+
+(define_insn_and_split "*minus_seqsi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (match_operand:W 2 "register_operand" "r")
+		 (eq:W (match_operand:SI 1 "register_operand" "r")
+		       (const_int 0))))
    (clobber (reg:CC CC_REG))]
   ""
   "#"
   ""
-  [(set (reg:CC_NOOV CC_REG) (compare:CC_NOOV (neg:SI (match_dup 1))
-					   (const_int 0)))
-   (set (match_dup 0) (minus:SI (match_dup 2)
-				(geu:SI (reg:CC CC_REG) (const_int 0))))]
+  [(set (reg:CCC CC_REG) (compare:CCC (not:SI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (minus:W (match_dup 2)
+			       (geu:W (reg:CCC CC_REG) (const_int 0))))]
+  ""
+  [(set_attr "length" "2")])
+
+(define_insn_and_split "*minus_seqdi<W:mode>_zero"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (match_operand:W 2 "register_operand" "r")
+		 (eq:W (match_operand:DI 1 "register_operand" "r")
+		       (const_int 0))))
+   (clobber (reg:CCX CC_REG))]
+  "TARGET_ARCH64 && TARGET_VIS3"
+  "#"
+  ""
+  [(set (reg:CCXC CC_REG) (compare:CCXC (not:DI (match_dup 1)) (const_int -1)))
+   (set (match_dup 0) (minus:W (match_dup 2)
+			       (geu:W (reg:CCXC CC_REG) (const_int 0))))]
   ""
   [(set_attr "length" "2")])
 
 ;; We can also do GEU and LTU directly, but these operate after a compare.
-;; ??? The addx/subx insns use the 32 bit carry flag so there are no DImode
-;; versions for v9.
 
-(define_insn "*sltu<P:mode>_insn"
-  [(set (match_operand:P 0 "register_operand" "=r")
-	(ltu:P (reg:CC CC_REG) (const_int 0)))]
-  ""
+(define_insn "*sltu<W:mode>_insn"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(ltu:W (match_operand 1 "icc_register_operand" "X") (const_int 0)))]
+  "GET_MODE (operands[1]) == CCmode || GET_MODE (operands[1]) == CCCmode"
   "addx\t%%g0, 0, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*sltu_insn_vis3"
+(define_insn "*plus_sltu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (ltu:W (match_operand 2 "icc_register_operand" "X")
+		       (const_int 0))
+		(match_operand:W 1 "arith_operand" "rI")))]
+  "GET_MODE (operands[2]) == CCmode || GET_MODE (operands[2]) == CCCmode"
+  "addx\t%%g0, %1, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*plus_plus_sltu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (plus:W (ltu:W (match_operand 3 "icc_register_operand" "X")
+			       (const_int 0))
+			(match_operand:W 1 "arith_operand" "%r"))
+		(match_operand:W 2 "arith_operand" "rI")))]
+  "GET_MODE (operands[3]) == CCmode || GET_MODE (operands[3]) == CCCmode"
+  "addx\t%1, %2, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*neg_sgeu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(neg:W (geu:W (match_operand 1 "icc_register_operand" "X")
+		      (const_int 0))))]
+  "GET_MODE (operands[1]) == CCmode || GET_MODE (operands[1]) == CCCmode"
+  "addx\t%%g0, -1, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*neg_sgeusidi"
   [(set (match_operand:DI 0 "register_operand" "=r")
-	(ltu:DI (reg:CCX CC_REG) (const_int 0)))]
-  "TARGET_ARCH64 && TARGET_VIS3"
-  "addxc\t%%g0, %%g0, %0"
+	(sign_extend:DI (neg:SI (geu:SI (match_operand 1 "icc_register_operand" "X")
+					(const_int 0)))))]
+  "TARGET_ARCH64
+   && (GET_MODE (operands[1]) == CCmode || GET_MODE (operands[1]) == CCCmode)"
+  "addx\t%%g0, -1, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*sltu_insn_vis3_trunc"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(ltu:SI (reg:CCX CC_REG) (const_int 0)))]
-  "TARGET_ARCH64 && TARGET_VIS3"
+(define_insn "*minus_sgeu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (match_operand:W 1 "register_operand" "r")
+		 (geu:W (match_operand 2 "icc_register_operand" "X")
+			(const_int 0))))]
+  "GET_MODE (operands[2]) == CCmode || GET_MODE (operands[2]) == CCCmode"
+  "addx\t%1, -1, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*addx<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (plus:W (match_operand:W 1 "arith_operand" "%r")
+			(match_operand:W 2 "arith_operand" "rI"))
+		(ltu:W (match_operand 3 "icc_register_operand" "X")
+		       (const_int 0))))]
+  "GET_MODE (operands[3]) == CCmode || GET_MODE (operands[3]) == CCCmode"
+  "addx\t%1, %2, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*sltu<W:mode>_insn_vis3"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(ltu:W (match_operand 1 "icc_register_operand" "X") (const_int 0)))]
+  "TARGET_ARCH64 && TARGET_VIS3
+   && (GET_MODE (operands[1]) == CCXmode || GET_MODE (operands[1]) == CCXCmode)"
   "addxc\t%%g0, %%g0, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*neg_sltusi_insn"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(neg:SI (ltu:SI (reg:CC CC_REG) (const_int 0))))]
-  ""
+(define_insn "*plus_sltu<W:mode>_vis3"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (ltu:W (match_operand 2 "icc_register_operand" "X")
+		       (const_int 0))
+		(match_operand:W 1 "arith_operand" "rI")))]
+  "TARGET_ARCH64 && TARGET_VIS3
+   && (GET_MODE (operands[2]) == CCXmode || GET_MODE (operands[2]) == CCXCmode)"
+  "addxc\t%%g0, %1, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*plus_plus_sltu<W:mode>_vis3"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (plus:W (ltu:W (match_operand 3 "icc_register_operand" "X")
+			       (const_int 0))
+			(match_operand:W 1 "arith_operand" "%r"))
+		(match_operand:W 2 "arith_operand" "rI")))]
+  "TARGET_ARCH64 && TARGET_VIS3
+   && (GET_MODE (operands[3]) == CCXmode || GET_MODE (operands[3]) == CCXCmode)"
+  "addxc\t%1, %2, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*neg_sgeu<W:mode>_vis3"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(neg:W (geu:W (match_operand 1 "icc_register_operand" "X")
+		      (const_int 0))))]
+  "TARGET_ARCH64 && TARGET_VIS3
+   && (GET_MODE (operands[1]) == CCXmode || GET_MODE (operands[1]) == CCXCmode)"
+  "addxc\t%%g0, -1, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*minus_sgeu<W:mode>_vis3"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (match_operand:W 1 "register_operand" "r")
+		 (geu:W (match_operand 2 "icc_register_operand" "X")
+		        (const_int 0))))]
+  "TARGET_ARCH64 && TARGET_VIS3
+   && (GET_MODE (operands[2]) == CCXmode || GET_MODE (operands[2]) == CCXCmode)"
+  "addxc\t%1, -1, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*addxc<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (plus:W (match_operand:W 1 "register_or_zero_operand" "%rJ")
+			(match_operand:W 2 "register_or_zero_operand" "rJ"))
+		(ltu:W (match_operand 3 "icc_register_operand" "X")
+		       (const_int 0))))]
+  "TARGET_ARCH64 && TARGET_VIS3
+   && (GET_MODE (operands[3]) == CCXmode || GET_MODE (operands[3]) == CCXCmode)"
+  "addxc\t%r1, %r2, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*neg_sltu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(neg:W (ltu:W (match_operand 1 "icc_register_operand" "X")
+		      (const_int 0))))]
+  "GET_MODE (operands[1]) == CCmode || GET_MODE (operands[1]) == CCCmode"
   "subx\t%%g0, 0, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*neg_sltudi_insn"
+(define_insn "*neg_sltusidi"
   [(set (match_operand:DI 0 "register_operand" "=r")
-	(sign_extend:DI (neg:SI (ltu:SI (reg:CC CC_REG) (const_int 0)))))]
-  "TARGET_ARCH64"
+	(sign_extend:DI (neg:SI (ltu:SI (match_operand 1 "icc_register_operand" "X")
+					(const_int 0)))))]
+  "TARGET_ARCH64
+   && (GET_MODE (operands[1]) == CCmode || GET_MODE (operands[1]) == CCCmode)"
   "subx\t%%g0, 0, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*neg_sltu_minus_x"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(minus:SI (neg:SI (ltu:SI (reg:CC CC_REG) (const_int 0)))
-		  (match_operand:SI 1 "arith_operand" "rI")))]
-  ""
+(define_insn "*minus_neg_sltu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (neg:W (ltu:W (match_operand 2 "icc_register_operand" "X")
+			       (const_int 0)))
+		 (match_operand:W 1 "arith_operand" "rI")))]
+  "GET_MODE (operands[2]) == CCmode || GET_MODE (operands[2]) == CCCmode"
   "subx\t%%g0, %1, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*neg_sltu_plus_x"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(neg:SI (plus:SI (ltu:SI (reg:CC CC_REG) (const_int 0))
-			 (match_operand:SI 1 "arith_operand" "rI"))))]
-  ""
+(define_insn "*neg_plus_sltu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(neg:W (plus:W (ltu:W (match_operand 2 "icc_register_operand" "X")
+			      (const_int 0))
+		       (match_operand:W 1 "arith_operand" "rI"))))]
+  "GET_MODE (operands[2]) == CCmode || GET_MODE (operands[2]) == CCCmode"
   "subx\t%%g0, %1, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*sgeu<P:mode>_insn"
-  [(set (match_operand:P 0 "register_operand" "=r")
-	(geu:P (reg:CC CC_REG) (const_int 0)))]
-  ""
+(define_insn "*minus_sltu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (match_operand:W 1 "register_operand" "r")
+		 (ltu:W (match_operand 2 "icc_register_operand" "X")
+			(const_int 0))))]
+  "GET_MODE (operands[2]) == CCmode || GET_MODE (operands[2]) == CCCmode"
+  "subx\t%1, 0, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*minus_minus_sltu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (minus:W (match_operand:W 1 "register_or_zero_operand" "rJ")
+			  (ltu:W (match_operand 3 "icc_register_operand" "X")
+				 (const_int 0)))
+		 (match_operand:W 2 "arith_operand" "rI")))]
+  "GET_MODE (operands[3]) == CCmode || GET_MODE (operands[3]) == CCCmode"
+  "subx\t%r1, %2, %0"
+  [(set_attr "type" "ialuX")])
+
+(define_insn "*sgeu<W:mode>_insn"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(geu:W (match_operand 1 "icc_register_operand" "X") (const_int 0)))]
+  "GET_MODE (operands[1]) == CCmode || GET_MODE (operands[1]) == CCCmode"
   "subx\t%%g0, -1, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*neg_sgeusi_insn"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(neg:SI (geu:SI (reg:CC CC_REG) (const_int 0))))]
-  ""
-  "addx\t%%g0, -1, %0"
+(define_insn "*plus_sgeu<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (geu:W (match_operand 2 "icc_register_operand" "X")
+		       (const_int 0))
+		(match_operand:W 1 "register_operand" "r")))]
+  "GET_MODE (operands[2]) == CCmode || GET_MODE (operands[2]) == CCCmode"
+  "subx\t%1, -1, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*neg_sgeudi_insn"
-  [(set (match_operand:DI 0 "register_operand" "=r")
-	(sign_extend:DI (neg:SI (geu:SI (reg:CC CC_REG) (const_int 0)))))]
-  "TARGET_ARCH64"
-  "addx\t%%g0, -1, %0"
+(define_insn "*subx<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (minus:W (match_operand:W 1 "register_or_zero_operand" "rJ")
+			  (match_operand:W 2 "arith_operand" "rI"))
+		 (ltu:W (match_operand 3 "icc_register_operand" "X")
+		        (const_int 0))))]
+  "GET_MODE (operands[3]) == CCmode || GET_MODE (operands[3]) == CCCmode"
+  "subx\t%r1, %2, %0"
   [(set_attr "type" "ialuX")])
 
-;; We can also do (x + ((unsigned) i >= 0)) and related, so put them in.
-;; ??? The addx/subx insns use the 32 bit carry flag so there are no DImode
-;; versions for v9.
+(define_insn "*neg_sltu<W:mode>_subxc"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(neg:W (ltu:W (match_operand 1 "icc_register_operand" "X")
+		      (const_int 0))))]
+  "TARGET_ARCH64 && TARGET_SUBXC
+   && (GET_MODE (operands[1]) == CCXmode || GET_MODE (operands[1]) == CCXCmode)"
+  "subxc\t%%g0, 0, %0"
+  [(set_attr "type" "ialuX")])
 
-(define_insn "*sltu_plus_x"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(plus:SI (ltu:SI (reg:CC CC_REG) (const_int 0))
-		 (match_operand:SI 1 "arith_operand" "rI")))]
-  ""
-  "addx\t%%g0, %1, %0"
+(define_insn "*minus_neg_sltu<W:mode>_subxc"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (neg:W (ltu:W (match_operand 2 "icc_register_operand" "X")
+			       (const_int 0)))
+		 (match_operand:W 1 "arith_operand" "rI")))]
+  "TARGET_ARCH64 && TARGET_SUBXC
+   && (GET_MODE (operands[2]) == CCXmode || GET_MODE (operands[2]) == CCXCmode)"
+  "subxc\t%%g0, %1, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*sltu_plus_x_plus_y"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(plus:SI (ltu:SI (reg:CC CC_REG) (const_int 0))
-		 (plus:SI (match_operand:SI 1 "arith_operand" "%r")
-			  (match_operand:SI 2 "arith_operand" "rI"))))]
-  ""
-  "addx\t%1, %2, %0"
+(define_insn "*neg_plus_sltu<W:mode>_subxc"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(neg:W (plus:W (ltu:W (match_operand 2 "icc_register_operand" "X")
+			      (const_int 0))
+		       (match_operand:W 1 "arith_operand" "rI"))))]
+  "TARGET_ARCH64 && TARGET_SUBXC
+   && (GET_MODE (operands[2]) == CCXmode || GET_MODE (operands[2]) == CCXCmode)"
+  "subxc\t%%g0, %1, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*x_minus_sltu"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(minus:SI (match_operand:SI 1 "register_operand" "r")
-		  (ltu:SI (reg:CC CC_REG) (const_int 0))))]
-  ""
-  "subx\t%1, 0, %0"
+(define_insn "*minus_sltu<W:mode>_subxc"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (match_operand:W 1 "register_operand" "r")
+		 (ltu:W (match_operand 2 "icc_register_operand" "X")
+			(const_int 0))))]
+  "TARGET_ARCH64 && TARGET_SUBXC
+   && (GET_MODE (operands[2]) == CCXmode || GET_MODE (operands[2]) == CCXCmode)"
+  "subxc\t%1, 0, %0"
   [(set_attr "type" "ialuX")])
 
-;; ??? Combine should canonicalize these next two to the same pattern.
-(define_insn "*x_minus_y_minus_sltu"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(minus:SI (minus:SI (match_operand:SI 1 "register_or_zero_operand" "rJ")
-			    (match_operand:SI 2 "arith_operand" "rI"))
-		  (ltu:SI (reg:CC CC_REG) (const_int 0))))]
-  ""
-  "subx\t%r1, %2, %0"
+(define_insn "*minus_minus_sltu<W:mode>_subxc"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (minus:W (match_operand:W 1 "register_or_zero_operand" "rJ")
+			  (ltu:W (match_operand 3 "icc_register_operand" "X")
+				 (const_int 0)))
+		 (match_operand:W 2 "arith_operand" "rI")))]
+  "TARGET_ARCH64 && TARGET_SUBXC
+   && (GET_MODE (operands[3]) == CCXmode || GET_MODE (operands[3]) == CCXCmode)"
+  "subxc\t%r1, %2, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*x_minus_sltu_plus_y"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(minus:SI (match_operand:SI 1 "register_or_zero_operand" "rJ")
-		  (plus:SI (ltu:SI (reg:CC CC_REG) (const_int 0))
-			   (match_operand:SI 2 "arith_operand" "rI"))))]
-  ""
-  "subx\t%r1, %2, %0"
+(define_insn "*sgeu<W:mode>_insn_subxc"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(geu:W (match_operand 1 "icc_register_operand" "X") (const_int 0)))]
+  "TARGET_ARCH64 && TARGET_SUBXC
+   && (GET_MODE (operands[1]) == CCXmode || GET_MODE (operands[1]) == CCXCmode)"
+  "subxc\t%%g0, -1, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*sgeu_plus_x"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(plus:SI (geu:SI (reg:CC CC_REG) (const_int 0))
-		 (match_operand:SI 1 "register_operand" "r")))]
-  ""
-  "subx\t%1, -1, %0"
+(define_insn "*plus_sgeu<W:mode>_subxc"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(plus:W (geu:W (match_operand 2 "icc_register_operand" "X")
+		       (const_int 0))
+		(match_operand:W 1 "register_operand" "r")))]
+  "TARGET_ARCH64 && TARGET_SUBXC
+   && (GET_MODE (operands[2]) == CCXmode || GET_MODE (operands[2]) == CCXCmode)"
+  "subxc\t%1, -1, %0"
   [(set_attr "type" "ialuX")])
 
-(define_insn "*x_minus_sgeu"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(minus:SI (match_operand:SI 1 "register_operand" "r")
-		  (geu:SI (reg:CC CC_REG) (const_int 0))))]
-  ""
-  "addx\t%1, -1, %0"
+(define_insn "*subxc<W:mode>"
+  [(set (match_operand:W 0 "register_operand" "=r")
+	(minus:W (minus:W (match_operand:W 1 "register_or_zero_operand" "rJ")
+			  (match_operand:W 2 "arith_operand" "rI"))
+		 (ltu:W (match_operand 3 "icc_register_operand" "X")
+			(const_int 0))))]
+  "TARGET_ARCH64 && TARGET_SUBXC
+   && (GET_MODE (operands[3]) == CCXmode || GET_MODE (operands[3]) == CCXCmode)"
+  "subxc\t%r1, %2, %0"
   [(set_attr "type" "ialuX")])
 
 (define_split
-  [(set (match_operand:SI 0 "register_operand" "")
-	(match_operator:SI 2 "noov_compare_operator"
-			   [(match_operand 1 "icc_or_fcc_register_operand" "")
-			    (const_int 0)]))]
+  [(set (match_operand:W 0 "register_operand" "")
+	(match_operator:W 1 "icc_comparison_operator"
+	  [(match_operand 2 "icc_register_operand" "") (const_int 0)]))]
   "TARGET_V9
-   && REGNO (operands[1]) == SPARC_ICC_REG
-   && (GET_MODE (operands[1]) == CCXmode
-       /* 32-bit LTU/GEU are better implemented using addx/subx.  */
-       || (GET_CODE (operands[2]) != LTU && GET_CODE (operands[2]) != GEU))"
+   /* 64-bit LTU is better implemented using addxc with VIS3.  */
+   && !(GET_CODE (operands[1]) == LTU
+	&& (GET_MODE (operands[2]) == CCXmode
+	    || GET_MODE (operands[2]) == CCXCmode)
+	&& TARGET_VIS3)
+   /* 64-bit GEU is better implemented using subxc with SUBXC.  */
+   && !(GET_CODE (operands[1]) == GEU
+	&& (GET_MODE (operands[2]) == CCXmode
+	    || GET_MODE (operands[2]) == CCXCmode)
+	&& TARGET_SUBXC)
+   /* 32-bit LTU/GEU are better implemented using addx/subx.  */
+   && !((GET_CODE (operands[1]) == LTU || GET_CODE (operands[1]) == GEU)
+	&& (GET_MODE (operands[2]) == CCmode
+	    || GET_MODE (operands[2]) == CCCmode))"
   [(set (match_dup 0) (const_int 0))
    (set (match_dup 0)
-	(if_then_else:SI (match_op_dup:SI 2 [(match_dup 1) (const_int 0)])
+	(if_then_else:SI (match_op_dup:W 1 [(match_dup 2) (const_int 0)])
 			 (const_int 1)
 			 (match_dup 0)))]
   "")
 
-
 ;; These control RTL generation for conditional jump insns
 
 (define_expand "cbranchcc4"
@@ -1195,7 +1384,7 @@
 ;; XXX fpcmp nop braindamage
 (define_insn "*normal_branch"
   [(set (pc)
-	(if_then_else (match_operator 0 "noov_compare_operator"
+	(if_then_else (match_operator 0 "icc_comparison_operator"
 				      [(reg CC_REG) (const_int 0)])
 		      (label_ref (match_operand 1 "" ""))
 		      (pc)))]
@@ -1211,7 +1400,7 @@
 ;; XXX fpcmp nop braindamage
 (define_insn "*inverted_branch"
   [(set (pc)
-	(if_then_else (match_operator 0 "noov_compare_operator"
+	(if_then_else (match_operator 0 "icc_comparison_operator"
 				      [(reg CC_REG) (const_int 0)])
 		      (pc)
 		      (label_ref (match_operand 1 "" ""))))]
@@ -1297,7 +1486,7 @@
 
 (define_insn "*cbcond_sp32"
   [(set (pc)
-        (if_then_else (match_operator 0 "noov_compare_operator"
+        (if_then_else (match_operator 0 "comparison_operator"
                        [(match_operand:SI 1 "register_operand" "r")
                         (match_operand:SI 2 "arith5_operand" "rA")])
                       (label_ref (match_operand 3 "" ""))
@@ -1310,7 +1499,7 @@
 
 (define_insn "*cbcond_sp64"
   [(set (pc)
-        (if_then_else (match_operator 0 "noov_compare_operator"
+        (if_then_else (match_operator 0 "comparison_operator"
                        [(match_operand:DI 1 "register_operand" "r")
                         (match_operand:DI 2 "arith5_operand" "rA")])
                       (label_ref (match_operand 3 "" ""))
@@ -1321,11 +1510,11 @@
 }
   [(set_attr "type" "cbcond")])
 
-;; There are no 32 bit brreg insns.
+;; There are no 32-bit brreg insns.
 
 (define_insn "*normal_int_branch_sp64"
   [(set (pc)
-	(if_then_else (match_operator 0 "v9_register_compare_operator"
+	(if_then_else (match_operator 0 "v9_register_comparison_operator"
 				      [(match_operand:DI 1 "register_operand" "r")
 				       (const_int 0)])
 		      (label_ref (match_operand 2 "" ""))
@@ -1341,7 +1530,7 @@
 
 (define_insn "*inverted_int_branch_sp64"
   [(set (pc)
-	(if_then_else (match_operator 0 "v9_register_compare_operator"
+	(if_then_else (match_operator 0 "v9_register_comparison_operator"
 				      [(match_operand:DI 1 "register_operand" "r")
 				       (const_int 0)])
 		      (pc)
@@ -2696,7 +2885,7 @@
 
 (define_insn "*mov<I:mode>_cc_v9"
   [(set (match_operand:I 0 "register_operand" "=r")
-	(if_then_else:I (match_operator 1 "comparison_operator"
+	(if_then_else:I (match_operator 1 "icc_or_fcc_comparison_operator"
 			       [(match_operand 2 "icc_or_fcc_register_operand" "X")
 				(const_int 0)])
 			(match_operand:I 3 "arith11_operand" "rL")
@@ -2707,7 +2896,7 @@
 
 (define_insn "*mov<I:mode>_cc_reg_sp64"
   [(set (match_operand:I 0 "register_operand" "=r")
-	(if_then_else:I (match_operator 1 "v9_register_compare_operator"
+	(if_then_else:I (match_operator 1 "v9_register_comparison_operator"
 				[(match_operand:DI 2 "register_operand" "r")
 				 (const_int 0)])
 			(match_operand:I 3 "arith10_operand" "rM")
@@ -2718,7 +2907,7 @@
 
 (define_insn "*movsf_cc_v9"
   [(set (match_operand:SF 0 "register_operand" "=f")
-	(if_then_else:SF (match_operator 1 "comparison_operator"
+	(if_then_else:SF (match_operator 1 "icc_or_fcc_comparison_operator"
 				[(match_operand 2 "icc_or_fcc_register_operand" "X")
 				 (const_int 0)])
 			 (match_operand:SF 3 "register_operand" "f")
@@ -2729,7 +2918,7 @@
 
 (define_insn "*movsf_cc_reg_sp64"
   [(set (match_operand:SF 0 "register_operand" "=f")
-	(if_then_else:SF (match_operator 1 "v9_register_compare_operator"
+	(if_then_else:SF (match_operator 1 "v9_register_comparison_operator"
 				[(match_operand:DI 2 "register_operand" "r")
 				 (const_int 0)])
 			 (match_operand:SF 3 "register_operand" "f")
@@ -2741,7 +2930,7 @@
 ;; Named because invoked by movtf_cc_v9
 (define_insn "movdf_cc_v9"
   [(set (match_operand:DF 0 "register_operand" "=e")
-	(if_then_else:DF (match_operator 1 "comparison_operator"
+	(if_then_else:DF (match_operator 1 "icc_or_fcc_comparison_operator"
 				[(match_operand 2 "icc_or_fcc_register_operand" "X")
 				 (const_int 0)])
 			 (match_operand:DF 3 "register_operand" "e")
@@ -2754,7 +2943,7 @@
 ;; Named because invoked by movtf_cc_reg_sp64
 (define_insn "movdf_cc_reg_sp64"
   [(set (match_operand:DF 0 "register_operand" "=e")
-	(if_then_else:DF (match_operator 1 "v9_register_compare_operator"
+	(if_then_else:DF (match_operator 1 "v9_register_comparison_operator"
 				[(match_operand:DI 2 "register_operand" "r")
 				 (const_int 0)])
 			 (match_operand:DF 3 "register_operand" "e")
@@ -2766,7 +2955,7 @@
 
 (define_insn "*movtf_cc_hq_v9"
   [(set (match_operand:TF 0 "register_operand" "=e")
-	(if_then_else:TF (match_operator 1 "comparison_operator"
+	(if_then_else:TF (match_operator 1 "icc_or_fcc_comparison_operator"
 				[(match_operand 2 "icc_or_fcc_register_operand" "X")
 				 (const_int 0)])
 			 (match_operand:TF 3 "register_operand" "e")
@@ -2777,7 +2966,7 @@
 
 (define_insn "*movtf_cc_reg_hq_sp64"
   [(set (match_operand:TF 0 "register_operand" "=e")
-	(if_then_else:TF (match_operator 1 "v9_register_compare_operator"
+	(if_then_else:TF (match_operator 1 "v9_register_comparison_operator"
 				[(match_operand:DI 2 "register_operand" "r")
 				 (const_int 0)])
 			 (match_operand:TF 3 "register_operand" "e")
@@ -2788,7 +2977,7 @@
 
 (define_insn_and_split "*movtf_cc_v9"
   [(set (match_operand:TF 0 "register_operand" "=e")
-	(if_then_else:TF (match_operator 1 "comparison_operator"
+	(if_then_else:TF (match_operator 1 "icc_or_fcc_comparison_operator"
 			    [(match_operand 2 "icc_or_fcc_register_operand" "X")
 			     (const_int 0)])
 			 (match_operand:TF 3 "register_operand" "e")
@@ -2824,7 +3013,7 @@
 
 (define_insn_and_split "*movtf_cc_reg_sp64"
   [(set (match_operand:TF 0 "register_operand" "=e")
-	(if_then_else:TF (match_operator 1 "v9_register_compare_operator"
+	(if_then_else:TF (match_operator 1 "v9_register_comparison_operator"
 				[(match_operand:DI 2 "register_operand" "r")
 				 (const_int 0)])
 			 (match_operand:TF 3 "register_operand" "e")
@@ -3616,36 +3805,29 @@
 		 (match_operand:DI 2 "arith_double_add_operand" "")))]
   ""
 {
-  if (! TARGET_ARCH64)
+  if (!TARGET_ARCH64)
     {
-      emit_insn (gen_rtx_PARALLEL (VOIDmode, gen_rtvec (2,
-			  gen_rtx_SET (operands[0],
-				   gen_rtx_PLUS (DImode, operands[1],
-						 operands[2])),
-			  gen_rtx_CLOBBER (VOIDmode,
-				   gen_rtx_REG (CCmode, SPARC_ICC_REG)))));
+      emit_insn (gen_adddi3_sp32 (operands[0], operands[1], operands[2]));
       DONE;
     }
 })
 
-(define_insn_and_split "*adddi3_insn_sp32"
+(define_insn_and_split "adddi3_sp32"
   [(set (match_operand:DI 0 "register_operand" "=&r")
-	(plus:DI (match_operand:DI 1 "arith_double_operand" "%r")
+	(plus:DI (match_operand:DI 1 "register_operand" "%r")
 		 (match_operand:DI 2 "arith_double_operand" "rHI")))
    (clobber (reg:CC CC_REG))]
-  "! TARGET_ARCH64"
+  "!TARGET_ARCH64"
   "#"
   "&& reload_completed"
-  [(parallel [(set (reg:CC_NOOV CC_REG)
-		   (compare:CC_NOOV (plus:SI (match_dup 4)
-					     (match_dup 5))
-				    (const_int 0)))
+  [(parallel [(set (reg:CCC CC_REG)
+		   (compare:CCC (plus:SI (match_dup 4) (match_dup 5))
+				(match_dup 4)))
 	      (set (match_dup 3)
 		   (plus:SI (match_dup 4) (match_dup 5)))])
    (set (match_dup 6)
-	(plus:SI (plus:SI (match_dup 7)
-			  (match_dup 8))
-		 (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0))))]
+	(plus:SI (plus:SI (match_dup 7) (match_dup 8))
+		 (ltu:SI (reg:CCC CC_REG) (const_int 0))))]
 {
   operands[3] = gen_lowpart (SImode, operands[0]);
   operands[4] = gen_lowpart (SImode, operands[1]);
@@ -3656,59 +3838,22 @@
 }
   [(set_attr "length" "2")])
 
-;; LTU here means "carry set"
-(define_insn "addx"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(plus:SI (plus:SI (match_operand:SI 1 "arith_operand" "%r")
-			  (match_operand:SI 2 "arith_operand" "rI"))
-		 (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0))))]
-  ""
-  "addx\t%1, %2, %0"
-  [(set_attr "type" "ialuX")])
-
-(define_insn "addxc"
-  [(set (match_operand:DI 0 "register_operand" "=r")
-	(plus:DI (plus:DI (match_operand:DI 1 "register_or_zero_operand" "%rJ")
-			  (match_operand:DI 2 "register_or_zero_operand" "rJ"))
-		 (ltu:DI (reg:CCX_NOOV CC_REG) (const_int 0))))]
-  "TARGET_ARCH64 && TARGET_VIS3"
-  "addxc\t%r1, %r2, %0"
-  [(set_attr "type" "ialuX")])
-
 (define_insn_and_split "*addx_extend_sp32"
   [(set (match_operand:DI 0 "register_operand" "=r")
 	(zero_extend:DI (plus:SI (plus:SI
-                                  (match_operand:SI 1 "register_or_zero_operand" "%rJ")
-                                  (match_operand:SI 2 "arith_operand" "rI"))
-                                 (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0)))))]
+                                   (match_operand:SI 1 "register_operand" "%r")
+                                   (match_operand:SI 2 "arith_operand" "rI"))
+                                 (ltu:SI (reg:CCC CC_REG) (const_int 0)))))]
   "! TARGET_ARCH64"
   "#"
   "&& reload_completed"
   [(set (match_dup 3) (plus:SI (plus:SI (match_dup 1) (match_dup 2))
-                               (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0))))
+                               (ltu:SI (reg:CCC CC_REG) (const_int 0))))
    (set (match_dup 4) (const_int 0))]
   "operands[3] = gen_lowpart (SImode, operands[0]);
    operands[4] = gen_highpart (SImode, operands[0]);"
   [(set_attr "length" "2")])
 
-(define_insn "*addx_extend_sp64"
-  [(set (match_operand:DI 0 "register_operand" "=r")
-	(zero_extend:DI (plus:SI (plus:SI (match_operand:SI 1 "register_or_zero_operand" "%rJ")
-                                          (match_operand:SI 2 "register_or_zero_operand" "rJ"))
-                                 (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0)))))]
-  "TARGET_ARCH64"
-  "addx\t%r1, %r2, %0"
-  [(set_attr "type" "ialuX")])
-
-(define_insn "*addxc_trunc_sp64_vis3"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(plus:SI (plus:SI (match_operand:SI 1 "register_or_zero_operand" "%rJ")
-                          (match_operand:SI 2 "register_or_zero_operand" "rJ"))
-                 (ltu:SI (reg:CCX_NOOV CC_REG) (const_int 0))))]
-  "TARGET_ARCH64 && TARGET_VIS3"
-  "addxc\t%r1, %r2, %0"
-  [(set_attr "type" "ialuX")])
-
 (define_insn_and_split "*adddi3_extend_sp32"
   [(set (match_operand:DI 0 "register_operand" "=&r")
         (plus:DI (zero_extend:DI (match_operand:SI 1 "register_operand" "r"))
@@ -3717,13 +3862,13 @@
   "! TARGET_ARCH64"
   "#"
   "&& reload_completed"
-  [(parallel [(set (reg:CC_NOOV CC_REG)
-                   (compare:CC_NOOV (plus:SI (match_dup 3) (match_dup 1))
-                                    (const_int 0)))
+  [(parallel [(set (reg:CCC CC_REG)
+                   (compare:CCC (plus:SI (match_dup 3) (match_dup 1))
+                                (match_dup 3)))
               (set (match_dup 5) (plus:SI (match_dup 3) (match_dup 1)))])
    (set (match_dup 6)
         (plus:SI (plus:SI (match_dup 4) (const_int 0))
-                 (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0))))]
+                 (ltu:SI (reg:CCC CC_REG) (const_int 0))))]
   "operands[3] = gen_lowpart (SImode, operands[2]);
    operands[4] = gen_highpart (SImode, operands[2]);
    operands[5] = gen_lowpart (SImode, operands[0]);
@@ -3750,40 +3895,80 @@
   [(set_attr "type" "*,*")
    (set_attr "fptype" "*,*")])
 
-(define_insn "*cmp_cc_plus"
-  [(set (reg:CC_NOOV CC_REG)
-	(compare:CC_NOOV (plus:SI (match_operand:SI 0 "arith_operand" "%r")
-				  (match_operand:SI 1 "arith_operand" "rI"))
-			 (const_int 0)))]
+(define_insn "*cmp_ccnz_plus"
+  [(set (reg:CCNZ CC_REG)
+	(compare:CCNZ (plus:SI (match_operand:SI 0 "register_operand" "%r")
+			       (match_operand:SI 1 "arith_operand" "rI"))
+		      (const_int 0)))]
+  ""
+  "addcc\t%0, %1, %%g0"
+  [(set_attr "type" "compare")])
+
+(define_insn "*cmp_ccxnz_plus"
+  [(set (reg:CCXNZ CC_REG)
+	(compare:CCXNZ (plus:DI (match_operand:DI 0 "register_operand" "%r")
+			        (match_operand:DI 1 "arith_operand" "rI"))
+		       (const_int 0)))]
+  "TARGET_ARCH64"
+  "addcc\t%0, %1, %%g0"
+  [(set_attr "type" "compare")])
+
+(define_insn "*cmp_ccnz_plus_set"
+  [(set (reg:CCNZ CC_REG)
+	(compare:CCNZ (plus:SI (match_operand:SI 1 "register_operand" "%r")
+			       (match_operand:SI 2 "arith_operand" "rI"))
+		      (const_int 0)))
+   (set (match_operand:SI 0 "register_operand" "=r")
+	(plus:SI (match_dup 1) (match_dup 2)))]
+  ""
+  "addcc\t%1, %2, %0"
+  [(set_attr "type" "compare")])
+
+(define_insn "*cmp_ccxnz_plus_set"
+  [(set (reg:CCXNZ CC_REG)
+	(compare:CCXNZ (plus:DI (match_operand:DI 1 "register_operand" "%r")
+			        (match_operand:DI 2 "arith_operand" "rI"))
+		       (const_int 0)))
+   (set (match_operand:DI 0 "register_operand" "=r")
+	(plus:DI (match_dup 1) (match_dup 2)))]
+  "TARGET_ARCH64"
+  "addcc\t%1, %2, %0"
+  [(set_attr "type" "compare")])
+
+(define_insn "*cmp_ccc_plus"
+  [(set (reg:CCC CC_REG)
+	(compare:CCC (plus:SI (match_operand:SI 0 "register_operand" "%r")
+			      (match_operand:SI 1 "arith_operand" "rI"))
+		     (match_dup 0)))]
   ""
   "addcc\t%0, %1, %%g0"
   [(set_attr "type" "compare")])
 
-(define_insn "*cmp_ccx_plus"
-  [(set (reg:CCX_NOOV CC_REG)
-	(compare:CCX_NOOV (plus:DI (match_operand:DI 0 "arith_operand" "%r")
-				   (match_operand:DI 1 "arith_operand" "rI"))
-			  (const_int 0)))]
+(define_insn "*cmp_ccxc_plus"
+  [(set (reg:CCXC CC_REG)
+	(compare:CCXC (plus:DI (match_operand:DI 0 "register_operand" "%r")
+			       (match_operand:DI 1 "arith_operand" "rI"))
+		      (match_dup 0)))]
   "TARGET_ARCH64"
   "addcc\t%0, %1, %%g0"
   [(set_attr "type" "compare")])
 
-(define_insn "*cmp_cc_plus_set"
-  [(set (reg:CC_NOOV CC_REG)
-	(compare:CC_NOOV (plus:SI (match_operand:SI 1 "arith_operand" "%r")
-				  (match_operand:SI 2 "arith_operand" "rI"))
-			 (const_int 0)))
+(define_insn "*cmp_ccc_plus_set"
+  [(set (reg:CCC CC_REG)
+	(compare:CCC (plus:SI (match_operand:SI 1 "register_operand" "%r")
+			      (match_operand:SI 2 "arith_operand" "rI"))
+		     (match_dup 1)))
    (set (match_operand:SI 0 "register_operand" "=r")
 	(plus:SI (match_dup 1) (match_dup 2)))]
   ""
   "addcc\t%1, %2, %0"
   [(set_attr "type" "compare")])
 
-(define_insn "*cmp_ccx_plus_set"
-  [(set (reg:CCX_NOOV CC_REG)
-	(compare:CCX_NOOV (plus:DI (match_operand:DI 1 "arith_operand" "%r")
-				   (match_operand:DI 2 "arith_operand" "rI"))
-			  (const_int 0)))
+(define_insn "*cmp_ccxc_plus_set"
+  [(set (reg:CCXC CC_REG)
+	(compare:CCXC (plus:DI (match_operand:DI 1 "register_operand" "%r")
+			       (match_operand:DI 2 "arith_operand" "rI"))
+		      (match_dup 1)))
    (set (match_operand:DI 0 "register_operand" "=r")
 	(plus:DI (match_dup 1) (match_dup 2)))]
   "TARGET_ARCH64"
@@ -3796,19 +3981,14 @@
 		  (match_operand:DI 2 "arith_double_add_operand" "")))]
   ""
 {
-  if (! TARGET_ARCH64)
+  if (!TARGET_ARCH64)
     {
-      emit_insn (gen_rtx_PARALLEL (VOIDmode, gen_rtvec (2,
-			  gen_rtx_SET (operands[0],
-				   gen_rtx_MINUS (DImode, operands[1],
-						  operands[2])),
-			  gen_rtx_CLOBBER (VOIDmode,
-				   gen_rtx_REG (CCmode, SPARC_ICC_REG)))));
+      emit_insn (gen_subdi3_sp32 (operands[0], operands[1], operands[2]));
       DONE;
     }
 })
 
-(define_insn_and_split "*subdi3_insn_sp32"
+(define_insn_and_split "subdi3_sp32"
   [(set (match_operand:DI 0 "register_operand" "=&r")
 	(minus:DI (match_operand:DI 1 "register_operand" "r")
 		  (match_operand:DI 2 "arith_double_operand" "rHI")))
@@ -3816,16 +3996,13 @@
   "! TARGET_ARCH64"
   "#"
   "&& reload_completed"
-  [(parallel [(set (reg:CC_NOOV CC_REG)
-		   (compare:CC_NOOV (minus:SI (match_dup 4)
-					      (match_dup 5))
-				    (const_int 0)))
+  [(parallel [(set (reg:CC CC_REG)
+		   (compare:CC (match_dup 4) (match_dup 5)))
 	      (set (match_dup 3)
 		   (minus:SI (match_dup 4) (match_dup 5)))])
    (set (match_dup 6)
-	(minus:SI (minus:SI (match_dup 7)
-			    (match_dup 8))
-		  (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0))))]
+	(minus:SI (minus:SI (match_dup 7) (match_dup 8))
+		  (ltu:SI (reg:CC CC_REG) (const_int 0))))]
 {
   operands[3] = gen_lowpart (SImode, operands[0]);
   operands[4] = gen_lowpart (SImode, operands[1]);
@@ -3836,35 +4013,17 @@
 }
   [(set_attr "length" "2")])
 
-;; LTU here means "carry set"
-(define_insn "subx"
-  [(set (match_operand:SI 0 "register_operand" "=r")
-	(minus:SI (minus:SI (match_operand:SI 1 "register_or_zero_operand" "rJ")
-			    (match_operand:SI 2 "arith_operand" "rI"))
-		  (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0))))]
-  ""
-  "subx\t%r1, %2, %0"
-  [(set_attr "type" "ialuX")])
-
-(define_insn "*subx_extend_sp64"
-  [(set (match_operand:DI 0 "register_operand" "=r")
-	(zero_extend:DI (minus:SI (minus:SI (match_operand:SI 1 "register_or_zero_operand" "rJ")
-                                            (match_operand:SI 2 "arith_operand" "rI"))
-                                  (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0)))))]
-  "TARGET_ARCH64"
-  "subx\t%r1, %2, %0"
-  [(set_attr "type" "ialuX")])
-
 (define_insn_and_split "*subx_extend_sp32"
   [(set (match_operand:DI 0 "register_operand" "=r")
-	(zero_extend:DI (minus:SI (minus:SI (match_operand:SI 1 "register_or_zero_operand" "rJ")
-                                            (match_operand:SI 2 "arith_operand" "rI"))
-                                  (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0)))))]
+	(zero_extend:DI (minus:SI (minus:SI
+				    (match_operand:SI 1 "register_or_zero_operand" "rJ")
+				    (match_operand:SI 2 "arith_operand" "rI"))
+                                  (ltu:SI (reg:CCC CC_REG) (const_int 0)))))]
   "! TARGET_ARCH64"
   "#"
   "&& reload_completed"
   [(set (match_dup 3) (minus:SI (minus:SI (match_dup 1) (match_dup 2))
-                                (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0))))
+                                (ltu:SI (reg:CCC CC_REG) (const_int 0))))
    (set (match_dup 4) (const_int 0))]
   "operands[3] = gen_lowpart (SImode, operands[0]);
    operands[4] = gen_highpart (SImode, operands[0]);"
@@ -3878,13 +4037,12 @@
   "! TARGET_ARCH64"
   "#"
   "&& reload_completed"
-  [(parallel [(set (reg:CC_NOOV CC_REG)
-                   (compare:CC_NOOV (minus:SI (match_dup 3) (match_dup 2))
-                                    (const_int 0)))
+  [(parallel [(set (reg:CC CC_REG)
+                   (compare:CC (match_dup 3) (match_dup 2)))
               (set (match_dup 5) (minus:SI (match_dup 3) (match_dup 2)))])
    (set (match_dup 6)
         (minus:SI (minus:SI (match_dup 4) (const_int 0))
-                  (ltu:SI (reg:CC_NOOV CC_REG) (const_int 0))))]
+                  (ltu:SI (reg:CC CC_REG) (const_int 0))))]
   "operands[3] = gen_lowpart (SImode, operands[1]);
    operands[4] = gen_highpart (SImode, operands[1]);
    operands[5] = gen_lowpart (SImode, operands[0]);
@@ -3911,44 +4069,64 @@
   [(set_attr "type" "*,*")
    (set_attr "fptype" "*,*")])
 
-(define_insn "*cmp_minus_cc"
-  [(set (reg:CC_NOOV CC_REG)
-	(compare:CC_NOOV (minus:SI (match_operand:SI 0 "register_or_zero_operand" "rJ")
-				   (match_operand:SI 1 "arith_operand" "rI"))
-			 (const_int 0)))]
+(define_insn "*cmp_ccnz_minus"
+  [(set (reg:CCNZ CC_REG)
+	(compare:CCNZ (minus:SI (match_operand:SI 0 "register_or_zero_operand" "rJ")
+				(match_operand:SI 1 "arith_operand" "rI"))
+		      (const_int 0)))]
   ""
   "subcc\t%r0, %1, %%g0"
   [(set_attr "type" "compare")])
 
-(define_insn "*cmp_minus_ccx"
-  [(set (reg:CCX_NOOV CC_REG)
-	(compare:CCX_NOOV (minus:DI (match_operand:DI 0 "register_operand" "r")
-				    (match_operand:DI 1 "arith_operand" "rI"))
-			  (const_int 0)))]
+(define_insn "*cmp_ccxnz_minus"
+  [(set (reg:CCXNZ CC_REG)
+	(compare:CCXNZ (minus:DI (match_operand:DI 0 "register_or_zero_operand" "rJ")
+				 (match_operand:DI 1 "arith_operand" "rI"))
+		       (const_int 0)))]
   "TARGET_ARCH64"
-  "subcc\t%0, %1, %%g0"
+  "subcc\t%r0, %1, %%g0"
   [(set_attr "type" "compare")])
 
-(define_insn "cmp_minus_cc_set"
-  [(set (reg:CC_NOOV CC_REG)
-	(compare:CC_NOOV (minus:SI (match_operand:SI 1 "register_or_zero_operand" "rJ")
-				   (match_operand:SI 2 "arith_operand" "rI"))
-			 (const_int 0)))
+(define_insn "*cmp_ccnz_minus_set"
+  [(set (reg:CCNZ CC_REG)
+	(compare:CCNZ (minus:SI (match_operand:SI 1 "register_or_zero_operand" "rJ")
+				(match_operand:SI 2 "arith_operand" "rI"))
+		      (const_int 0)))
    (set (match_operand:SI 0 "register_operand" "=r")
 	(minus:SI (match_dup 1) (match_dup 2)))]
   ""
   "subcc\t%r1, %2, %0"
   [(set_attr "type" "compare")])
 
-(define_insn "*cmp_minus_ccx_set"
-  [(set (reg:CCX_NOOV CC_REG)
-	(compare:CCX_NOOV (minus:DI (match_operand:DI 1 "register_operand" "r")
-				    (match_operand:DI 2 "arith_operand" "rI"))
-			  (const_int 0)))
+(define_insn "*cmp_ccxnz_minus_set"
+  [(set (reg:CCXNZ CC_REG)
+	(compare:CCXNZ (minus:DI (match_operand:DI 1 "register_or_zero_operand" "rJ")
+				 (match_operand:DI 2 "arith_operand" "rI"))
+		       (const_int 0)))
    (set (match_operand:DI 0 "register_operand" "=r")
 	(minus:DI (match_dup 1) (match_dup 2)))]
   "TARGET_ARCH64"
-  "subcc\t%1, %2, %0"
+  "subcc\t%r1, %2, %0"
+  [(set_attr "type" "compare")])
+
+(define_insn "*cmpsi_set"
+  [(set (reg:CC CC_REG)
+	(compare:CC (match_operand:SI 1 "register_or_zero_operand" "rJ")
+		    (match_operand:SI 2 "arith_operand" "rI")))
+   (set (match_operand:SI 0 "register_operand" "=r")
+	(minus:SI (match_dup 1) (match_dup 2)))]
+  ""
+  "subcc\t%r1, %2, %0"
+  [(set_attr "type" "compare")])
+
+(define_insn "*cmpdi_set"
+  [(set (reg:CCX CC_REG)
+	(compare:CCX (match_operand:DI 1 "register_or_zero_operand" "rJ")
+		     (match_operand:DI 2 "arith_operand" "rI")))
+   (set (match_operand:DI 0 "register_operand" "=r")
+	(minus:DI (match_dup 1) (match_dup 2)))]
+  "TARGET_ARCH64"
+  "subcc\t%r1, %2, %0"
   [(set_attr "type" "compare")])
 
 
@@ -5040,33 +5218,25 @@
 	(neg:DI (match_operand:DI 1 "register_operand" "r")))]
   ""
 {
-  if (! TARGET_ARCH64)
+  if (!TARGET_ARCH64)
     {
-      emit_insn (gen_rtx_PARALLEL
-		 (VOIDmode,
-		  gen_rtvec (2,
-			     gen_rtx_SET (operand0,
-					  gen_rtx_NEG (DImode, operand1)),
-			     gen_rtx_CLOBBER (VOIDmode,
-					      gen_rtx_REG (CCmode,
-							   SPARC_ICC_REG)))));
+      emit_insn (gen_negdi2_sp32 (operands[0], operands[1]));
       DONE;
     }
 })
 
-(define_insn_and_split "*negdi2_sp32"
+(define_insn_and_split "negdi2_sp32"
   [(set (match_operand:DI 0 "register_operand" "=&r")
 	(neg:DI (match_operand:DI 1 "register_operand" "r")))
    (clobber (reg:CC CC_REG))]
   "! TARGET_ARCH64"
   "#"
   "&& reload_completed"
-  [(parallel [(set (reg:CC_NOOV CC_REG)
-                   (compare:CC_NOOV (minus:SI (const_int 0) (match_dup 5))
-                                    (const_int 0)))
-              (set (match_dup 4) (minus:SI (const_int 0) (match_dup 5)))])
+  [(parallel [(set (reg:CCC CC_REG)
+                   (compare:CCC (not:SI (match_dup 5)) (const_int -1)))
+              (set (match_dup 4) (neg:SI (match_dup 5)))])
    (set (match_dup 2) (minus:SI (minus:SI (const_int 0) (match_dup 3))
-                                (ltu:SI (reg:CC CC_REG) (const_int 0))))]
+                                (ltu:SI (reg:CCC CC_REG) (const_int 0))))]
   "operands[2] = gen_highpart (SImode, operands[0]);
    operands[3] = gen_highpart (SImode, operands[1]);
    operands[4] = gen_lowpart (SImode, operands[0]);
@@ -5085,42 +5255,63 @@
   ""
   "sub\t%%g0, %1, %0")
 
-(define_insn "*cmp_cc_neg"
-  [(set (reg:CC_NOOV CC_REG)
-	(compare:CC_NOOV (neg:SI (match_operand:SI 0 "arith_operand" "rI"))
-			 (const_int 0)))]
+(define_insn "*cmp_ccnz_neg"
+  [(set (reg:CCNZ CC_REG)
+	(compare:CCNZ (neg:SI (match_operand:SI 0 "arith_operand" "rI"))
+		      (const_int 0)))]
   ""
   "subcc\t%%g0, %0, %%g0"
   [(set_attr "type" "compare")])
 
-(define_insn "*cmp_ccx_neg"
-  [(set (reg:CCX_NOOV CC_REG)
-	(compare:CCX_NOOV (neg:DI (match_operand:DI 0 "arith_operand" "rI"))
-			  (const_int 0)))]
+(define_insn "*cmp_ccxnz_neg"
+  [(set (reg:CCXNZ CC_REG)
+	(compare:CCXNZ (neg:DI (match_operand:DI 0 "arith_operand" "rI"))
+		       (const_int 0)))]
   "TARGET_ARCH64"
   "subcc\t%%g0, %0, %%g0"
   [(set_attr "type" "compare")])
 
-(define_insn "*cmp_cc_set_neg"
-  [(set (reg:CC_NOOV CC_REG)
-	(compare:CC_NOOV (neg:SI (match_operand:SI 1 "arith_operand" "rI"))
-			 (const_int 0)))
+(define_insn "*cmp_ccnz_neg_set"
+  [(set (reg:CCNZ CC_REG)
+	(compare:CCNZ (neg:SI (match_operand:SI 1 "arith_operand" "rI"))
+		      (const_int 0)))
    (set (match_operand:SI 0 "register_operand" "=r")
 	(neg:SI (match_dup 1)))]
   ""
   "subcc\t%%g0, %1, %0"
   [(set_attr "type" "compare")])
 
-(define_insn "*cmp_ccx_set_neg"
-  [(set (reg:CCX_NOOV CC_REG)
-	(compare:CCX_NOOV (neg:DI (match_operand:DI 1 "arith_operand" "rI"))
-			  (const_int 0)))
+(define_insn "*cmp_ccxnz_neg_set"
+  [(set (reg:CCXNZ CC_REG)
+	(compare:CCXNZ (neg:DI (match_operand:DI 1 "arith_operand" "rI"))
+		       (const_int 0)))
    (set (match_operand:DI 0 "register_operand" "=r")
 	(neg:DI (match_dup 1)))]
   "TARGET_ARCH64"
   "subcc\t%%g0, %1, %0"
   [(set_attr "type" "compare")])
 
+(define_insn "*cmp_ccc_neg_set"
+  [(set (reg:CCC CC_REG)
+	(compare:CCC (not:SI (match_operand:SI 1 "arith_operand" "rI"))
+		     (const_int -1)))
+   (set (match_operand:SI 0 "register_operand" "=r")
+	(neg:SI (match_dup 1)))]
+  ""
+  "subcc\t%%g0, %1, %0"
+  [(set_attr "type" "compare")])
+
+(define_insn "*cmp_ccxc_neg_set"
+  [(set (reg:CCXC CC_REG)
+	(compare:CCXC (not:DI (match_operand:DI 1 "arith_operand" "rI"))
+		      (const_int -1)))
+   (set (match_operand:DI 0 "register_operand" "=r")
+	(neg:DI (match_dup 1)))]
+  "TARGET_ARCH64"
+  "subcc\t%%g0, %1, %0"
+  [(set_attr "type" "compare")])
+
+
 (define_insn "one_cmpldi2"
   [(set (match_operand:DI 0 "register_operand" "=r")
 	(not:DI (match_operand:DI 1 "arith_operand" "rI")))]
@@ -5173,8 +5364,7 @@
   [(set (match_operand:SI 0 "register_operand" "=r")
 	(match_operand:SI 1 "register_operand" "r"))
    (set (reg:CC CC_REG)
-	(compare:CC (match_dup 1)
-		    (const_int 0)))]
+	(compare:CC (match_dup 1) (const_int 0)))]
   ""
   "orcc\t%1, 0, %0"
   [(set_attr "type" "compare")])
@@ -5183,8 +5373,7 @@
   [(set (match_operand:DI 0 "register_operand" "=r")
 	(match_operand:DI 1 "register_operand" "r"))
    (set (reg:CCX CC_REG)
-	(compare:CCX (match_dup 1)
-		     (const_int 0)))]
+	(compare:CCX (match_dup 1) (const_int 0)))]
   "TARGET_ARCH64"
   "orcc\t%1, 0, %0"
    [(set_attr "type" "compare")])
@@ -5730,37 +5919,20 @@
   [(set_attr "type" "multi")
    (set_attr "length" "5,5,6")])
 
-;; Optimize (1LL<<x)-1
-;; XXX this also needs to be fixed to handle equal subregs
-;; XXX first before we could re-enable it.
-;(define_insn ""
-;  [(set (match_operand:DI 0 "register_operand" "=h")
-;	(plus:DI (ashift:DI (const_int 1)
-;			    (match_operand:SI 1 "arith_operand" "rI"))
-;		 (const_int -1)))]
-;  "0 && TARGET_V8PLUS"
-;{
-;  if (GET_CODE (operands[1]) == REG && REGNO (operands[1]) == REGNO (operands[0]))
-;    return "mov\t1, %L0\;sllx\t%L0, %1, %L0\;sub\t%L0, 1, %L0\;srlx\t%L0, 32, %H0";
-;  return "mov\t1, %H0\;sllx\t%H0, %1, %L0\;sub\t%L0, 1, %L0\;srlx\t%L0, 32, %H0";
-;}
-;  [(set_attr "type" "multi")
-;   (set_attr "length" "4")])
-
-(define_insn "*cmp_cc_ashift_1"
-  [(set (reg:CC_NOOV CC_REG)
-	(compare:CC_NOOV (ashift:SI (match_operand:SI 0 "register_operand" "r")
-				    (const_int 1))
-			 (const_int 0)))]
+(define_insn "*cmp_ccnz_ashift_1"
+  [(set (reg:CCNZ CC_REG)
+	(compare:CCNZ (ashift:SI (match_operand:SI 0 "register_operand" "r")
+				 (const_int 1))
+		      (const_int 0)))]
   ""
   "addcc\t%0, %0, %%g0"
   [(set_attr "type" "compare")])
 
-(define_insn "*cmp_cc_set_ashift_1"
-  [(set (reg:CC_NOOV CC_REG)
-	(compare:CC_NOOV (ashift:SI (match_operand:SI 1 "register_operand" "r")
-				    (const_int 1))
-			 (const_int 0)))
+(define_insn "*cmp_ccnz_set_ashift_1"
+  [(set (reg:CCNZ CC_REG)
+	(compare:CCNZ (ashift:SI (match_operand:SI 1 "register_operand" "r")
+				 (const_int 1))
+		      (const_int 0)))
    (set (match_operand:SI 0 "register_operand" "=r")
 	(ashift:SI (match_dup 1) (const_int 1)))]
   ""
@@ -7092,29 +7264,34 @@
   [(set_attr "type" "trap")])
 
 (define_expand "ctrapsi4"
-  [(trap_if (match_operator 0 "noov_compare_operator"
+  [(trap_if (match_operator 0 "comparison_operator"
 	     [(match_operand:SI 1 "compare_operand" "")
 	      (match_operand:SI 2 "arith_operand" "")])
 	   (match_operand 3 "arith_operand"))]
   ""
-  "operands[1] = gen_compare_reg (operands[0]);
-   if (GET_MODE (operands[1]) != CCmode && GET_MODE (operands[1]) != CCXmode)
-     FAIL;
-   operands[2] = const0_rtx;")
+{
+  operands[1] = gen_compare_reg (operands[0]);
+  if (GET_MODE (operands[1]) != CCmode && GET_MODE (operands[1]) != CCXmode)
+    FAIL;
+  operands[2] = const0_rtx;
+})
 
 (define_expand "ctrapdi4"
-  [(trap_if (match_operator 0 "noov_compare_operator"
+  [(trap_if (match_operator 0 "comparison_operator"
 	     [(match_operand:DI 1 "compare_operand" "")
 	      (match_operand:DI 2 "arith_operand" "")])
 	   (match_operand 3 "arith_operand"))]
   "TARGET_ARCH64"
-  "operands[1] = gen_compare_reg (operands[0]);
-   if (GET_MODE (operands[1]) != CCmode && GET_MODE (operands[1]) != CCXmode)
-     FAIL;
-   operands[2] = const0_rtx;")
+{
+  operands[1] = gen_compare_reg (operands[0]);
+  if (GET_MODE (operands[1]) != CCmode && GET_MODE (operands[1]) != CCXmode)
+    FAIL;
+  operands[2] = const0_rtx;
+})
 
-(define_insn ""
-  [(trap_if (match_operator 0 "noov_compare_operator" [(reg:CC CC_REG) (const_int 0)])
+(define_insn "*trapsi_insn"
+  [(trap_if (match_operator 0 "icc_comparison_operator"
+	      [(reg:CC CC_REG) (const_int 0)])
 	    (match_operand:SI 1 "arith_operand" "rM"))]
   ""
 {
@@ -7125,8 +7302,9 @@
 }
   [(set_attr "type" "trap")])
 
-(define_insn ""
-  [(trap_if (match_operator 0 "noov_compare_operator" [(reg:CCX CC_REG) (const_int 0)])
+(define_insn "*trapdi_insn"
+  [(trap_if (match_operator 0 "icc_comparison_operator"
+	      [(reg:CCX CC_REG) (const_int 0)])
 	    (match_operand:SI 1 "arith_operand" "rM"))]
   "TARGET_V9"
   "t%C0\t%%xcc, %1"
@@ -8315,10 +8493,10 @@
 ;; Edge instructions produce condition codes equivalent to a 'subcc'
 ;; with the same operands.
 (define_insn "edge8<P:mode>_vis"
-  [(set (reg:CC_NOOV CC_REG)
-        (compare:CC_NOOV (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
-			  	  (match_operand:P 2 "register_or_zero_operand" "rJ"))
-			 (const_int 0)))
+  [(set (reg:CCNZ CC_REG)
+        (compare:CCNZ (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
+			       (match_operand:P 2 "register_or_zero_operand" "rJ"))
+		      (const_int 0)))
    (set (match_operand:P 0 "register_operand" "=r")
         (unspec:P [(match_dup 1) (match_dup 2)] UNSPEC_EDGE8))]
   "TARGET_VIS"
@@ -8326,10 +8504,10 @@
   [(set_attr "type" "edge")])
 
 (define_insn "edge8l<P:mode>_vis"
-  [(set (reg:CC_NOOV CC_REG)
-        (compare:CC_NOOV (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
-			  	  (match_operand:P 2 "register_or_zero_operand" "rJ"))
-			 (const_int 0)))
+  [(set (reg:CCNZ CC_REG)
+        (compare:CCNZ (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
+			       (match_operand:P 2 "register_or_zero_operand" "rJ"))
+		      (const_int 0)))
    (set (match_operand:P 0 "register_operand" "=r")
         (unspec:P [(match_dup 1) (match_dup 2)] UNSPEC_EDGE8L))]
   "TARGET_VIS"
@@ -8337,10 +8515,10 @@
   [(set_attr "type" "edge")])
 
 (define_insn "edge16<P:mode>_vis"
-  [(set (reg:CC_NOOV CC_REG)
-        (compare:CC_NOOV (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
-			  	  (match_operand:P 2 "register_or_zero_operand" "rJ"))
-			 (const_int 0)))
+  [(set (reg:CCNZ CC_REG)
+        (compare:CCNZ (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
+			       (match_operand:P 2 "register_or_zero_operand" "rJ"))
+		      (const_int 0)))
    (set (match_operand:P 0 "register_operand" "=r")
         (unspec:P [(match_dup 1) (match_dup 2)] UNSPEC_EDGE16))]
   "TARGET_VIS"
@@ -8348,10 +8526,10 @@
   [(set_attr "type" "edge")])
 
 (define_insn "edge16l<P:mode>_vis"
-  [(set (reg:CC_NOOV CC_REG)
-        (compare:CC_NOOV (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
-			  	  (match_operand:P 2 "register_or_zero_operand" "rJ"))
-			 (const_int 0)))
+  [(set (reg:CCNZ CC_REG)
+        (compare:CCNZ (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
+			       (match_operand:P 2 "register_or_zero_operand" "rJ"))
+		      (const_int 0)))
    (set (match_operand:P 0 "register_operand" "=r")
         (unspec:P [(match_dup 1) (match_dup 2)] UNSPEC_EDGE16L))]
   "TARGET_VIS"
@@ -8359,10 +8537,10 @@
   [(set_attr "type" "edge")])
 
 (define_insn "edge32<P:mode>_vis"
-  [(set (reg:CC_NOOV CC_REG)
-        (compare:CC_NOOV (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
-			  	  (match_operand:P 2 "register_or_zero_operand" "rJ"))
-			 (const_int 0)))
+  [(set (reg:CCNZ CC_REG)
+        (compare:CCNZ (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
+			       (match_operand:P 2 "register_or_zero_operand" "rJ"))
+		      (const_int 0)))
    (set (match_operand:P 0 "register_operand" "=r")
         (unspec:P [(match_dup 1) (match_dup 2)] UNSPEC_EDGE32))]
   "TARGET_VIS"
@@ -8370,10 +8548,10 @@
   [(set_attr "type" "edge")])
 
 (define_insn "edge32l<P:mode>_vis"
-  [(set (reg:CC_NOOV CC_REG)
-        (compare:CC_NOOV (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
-			  	  (match_operand:P 2 "register_or_zero_operand" "rJ"))
-			 (const_int 0)))
+  [(set (reg:CCNZ CC_REG)
+        (compare:CCNZ (minus:P (match_operand:P 1 "register_or_zero_operand" "rJ")
+			       (match_operand:P 2 "register_or_zero_operand" "rJ"))
+		      (const_int 0)))
    (set (match_operand:P 0 "register_operand" "=r")
         (unspec:P [(match_dup 1) (match_dup 2)] UNSPEC_EDGE32L))]
   "TARGET_VIS"
diff --git a/gcc/config/sparc/sparc.opt b/gcc/config/sparc/sparc.opt
index 13d4151..1be7800 100644
--- a/gcc/config/sparc/sparc.opt
+++ b/gcc/config/sparc/sparc.opt
@@ -89,6 +89,10 @@ mpopc
 Target Report Mask(POPC)
 Use UltraSPARC Population-Count instruction.
 
+msubxc
+Target Report Mask(SUBXC)
+Use UltraSPARC Subtract-Extended-with-Carry instruction.
+
 mptr64
 Target Report RejectNegative Mask(PTR64)
 Pointers are 64-bit.
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 3f9c0a0..fa693ba 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -1092,8 +1092,8 @@ See RS/6000 and PowerPC Options.
 -muser-mode  -mno-user-mode @gol
 -mv8plus  -mno-v8plus  -mvis  -mno-vis @gol
 -mvis2  -mno-vis2  -mvis3  -mno-vis3 @gol
--mcbcond -mno-cbcond @gol
--mfmaf  -mno-fmaf  -mpopc  -mno-popc @gol
+-mcbcond -mno-cbcond  -mfmaf  -mno-fmaf  @gol
+-mpopc  -mno-popc  -msubxc  -mno-subxc@gol
 -mfix-at697f -mfix-ut699}
 
 @emph{SPU Options}
@@ -22941,27 +22941,36 @@ also sets @option{-mvis3}, @option{-mvis2} and @option{-mvis}.
 @itemx -mno-cbcond
 @opindex mcbcond
 @opindex mno-cbcond
-With @option{-mcbcond}, GCC generates code that takes advantage of
-compare-and-branch instructions, as defined in the Sparc Architecture 2011.
-The default is @option{-mcbcond} when targeting a cpu that supports such
-instructions, such as niagara-4 and later.
+With @option{-mcbcond}, GCC generates code that takes advantage of the UltraSPARC
+Compare-and-Branch-on-Condition instructions.  The default is @option{-mcbcond}
+when targeting a CPU that supports such instructions, such as Niagara-4 and
+later.
+
+@item -mfmaf
+@itemx -mno-fmaf
+@opindex mfmaf
+@opindex mno-fmaf
+With @option{-mfmaf}, GCC generates code that takes advantage of the UltraSPARC
+Fused Multiply-Add Floating-point instructions.  The default is @option{-mfmaf}
+when targeting a CPU that supports such instructions, such as Niagara-3 and
+later.
 
 @item -mpopc
 @itemx -mno-popc
 @opindex mpopc
 @opindex mno-popc
 With @option{-mpopc}, GCC generates code that takes advantage of the UltraSPARC
-population count instruction.  The default is @option{-mpopc}
-when targeting a cpu that supports such instructions, such as Niagara-2 and
+Population Count instruction.  The default is @option{-mpopc}
+when targeting a CPU that supports such an instruction, such as Niagara-2 and
 later.
 
-@item -mfmaf
-@itemx -mno-fmaf
-@opindex mfmaf
-@opindex mno-fmaf
-With @option{-mfmaf}, GCC generates code that takes advantage of the UltraSPARC
-Fused Multiply-Add Floating-point extensions.  The default is @option{-mfmaf}
-when targeting a cpu that supports such instructions, such as Niagara-3 and
+@item -msubxc
+@itemx -mno-subxc
+@opindex msubxc
+@opindex mno-subxc
+With @option{-msubxc}, GCC generates code that takes advantage of the UltraSPARC
+Subtract-Extended-with-Carry instruction.  The default is @option{-msubxc}
+when targeting a CPU that supports such an instruction, such as Niagara-7 and
 later.
 
 @item -mfix-at697f
diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
index 634f701..d54f96c 100644
--- a/gcc/doc/tm.texi
+++ b/gcc/doc/tm.texi
@@ -6021,8 +6021,8 @@ the case of the add on the SPARC discussed above, we have the pattern
 
 @smallexample
 (define_insn ""
-  [(set (reg:CC_NOOV 0)
-        (compare:CC_NOOV
+  [(set (reg:CCNZ 0)
+        (compare:CCNZ
           (plus:SI (match_operand:SI 0 "register_operand" "%r")
                    (match_operand:SI 1 "arith_operand" "rI"))
           (const_int 0)))]
@@ -6031,7 +6031,7 @@ the case of the add on the SPARC discussed above, we have the pattern
 @end smallexample
 
 @noindent
-together with a @code{SELECT_CC_MODE} that returns @code{CC_NOOVmode}
+together with a @code{SELECT_CC_MODE} that returns @code{CCNZmode}
 for comparisons whose argument is a @code{plus}:
 
 @smallexample
@@ -6041,7 +6041,7 @@ for comparisons whose argument is a @code{plus}:
       ? CCFPEmode : CCFPmode)                            \
    : ((GET_CODE (X) == PLUS || GET_CODE (X) == MINUS     \
        || GET_CODE (X) == NEG || GET_CODE (x) == ASHIFT) \
-      ? CC_NOOVmode : CCmode))
+      ? CCNZmode : CCmode))
 @end smallexample
 
 Another reason to use modes is to retain information on which operands
diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
index 7b6fa87..00699e3 100644
--- a/gcc/doc/tm.texi.in
+++ b/gcc/doc/tm.texi.in
@@ -4443,8 +4443,8 @@ the case of the add on the SPARC discussed above, we have the pattern
 
 @smallexample
 (define_insn ""
-  [(set (reg:CC_NOOV 0)
-        (compare:CC_NOOV
+  [(set (reg:CCNZ 0)
+        (compare:CCNZ
           (plus:SI (match_operand:SI 0 "register_operand" "%r")
                    (match_operand:SI 1 "arith_operand" "rI"))
           (const_int 0)))]
@@ -4453,7 +4453,7 @@ the case of the add on the SPARC discussed above, we have the pattern
 @end smallexample
 
 @noindent
-together with a @code{SELECT_CC_MODE} that returns @code{CC_NOOVmode}
+together with a @code{SELECT_CC_MODE} that returns @code{CCNZmode}
 for comparisons whose argument is a @code{plus}:
 
 @smallexample
@@ -4463,7 +4463,7 @@ for comparisons whose argument is a @code{plus}:
       ? CCFPEmode : CCFPmode)                            \
    : ((GET_CODE (X) == PLUS || GET_CODE (X) == MINUS     \
        || GET_CODE (X) == NEG || GET_CODE (x) == ASHIFT) \
-      ? CC_NOOVmode : CCmode))
+      ? CCNZmode : CCmode))
 @end smallexample
 
 Another reason to use modes is to retain information on which operands
diff --git a/gcc/testsuite/ChangeLog b/gcc/testsuite/ChangeLog
index e31bcc6..126342d 100644
--- a/gcc/testsuite/ChangeLog
+++ b/gcc/testsuite/ChangeLog
@@ -1,3 +1,20 @@
+2016-10-11  Eric Botcazou  <ebotcazou@adacore.com>
+
+	* gcc.target/sparc/cbcond-1.c: New test.
+	* gcc.target/sparc/cbcond-2.c: Likewise.
+	* gcc.target/sparc/movcc-1.c: Likewise.
+	* gcc.target/sparc/movcc-2.c: Likewise.
+	* gcc.target/sparc/setcc-1.c: Adjust.
+	* gcc.target/sparc/setcc-2.c: Likewise.
+	* gcc.target/sparc/setcc-3.c: Likewise.
+	* gcc.target/sparc/setcc-4.c: Likewise.
+	* gcc.target/sparc/setcc-5.c: Likewise.
+	* gcc.target/sparc/setcc-6.c: New test.
+	* gcc.target/sparc/setcc-7.c: Likewise.
+	* gcc.target/sparc/setcc-8.c: Likewise.
+	* gcc.target/sparc/setcc-9.c: Likewise.
+	* gcc.target/sparc/setcc-10.c: Likewise.
+
 2016-10-10  Jeff Law  <law@redhat.com>
 
 	PR tree-optimization/71947
diff --git a/gcc/testsuite/gcc.target/sparc/cbcond-1.c b/gcc/testsuite/gcc.target/sparc/cbcond-1.c
new file mode 100644
index 0000000..74fe475
--- /dev/null
+++ b/gcc/testsuite/gcc.target/sparc/cbcond-1.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-O -mcbcond" } */
+
+extern void foo (void);
+extern void bar (void);
+
+void cbcondne (int a)
+{
+  if (a != 0)
+    foo ();
+  bar ();
+}
+
+void cbconde (int a)
+{
+  if (a == 0)
+    foo ();
+  bar ();
+}
+
+void cbcondl (int a)
+{
+  if (a < 0)
+    foo ();
+  bar ();
+}
+
+void cbcondle (int a)
+{
+  if (a <= 0)
+    foo ();
+  bar ();
+}
+
+/* { dg-final { scan-assembler "cwbe\t%"  { target ilp32 } } } */
+/* { dg-final { scan-assembler "cwbne\t%" { target ilp32 } } } */
+/* { dg-final { scan-assembler "cwbl\t%"  } } */
+/* { dg-final { scan-assembler "cwble\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/cbcond-2.c b/gcc/testsuite/gcc.target/sparc/cbcond-2.c
new file mode 100644
index 0000000..3c03d44
--- /dev/null
+++ b/gcc/testsuite/gcc.target/sparc/cbcond-2.c
@@ -0,0 +1,39 @@
+/* { dg-do compile } */
+/* { dg-options "-O -mcbcond" } */
+/* { dg-require-effective-target lp64 } */
+
+extern void foo (void);
+extern void bar (void);
+
+void cbcondne (long a)
+{
+  if (a != 0)
+    foo ();
+  bar ();
+}
+
+void cbconde (long a)
+{
+  if (a == 0)
+    foo ();
+  bar ();
+}
+
+void cbcondl (long a)
+{
+  if (a < 0)
+    foo ();
+  bar ();
+}
+
+void cbcondle (long a)
+{
+  if (a <= 0)
+    foo ();
+  bar ();
+}
+
+/* { dg-final { scan-assembler "cxbe\t%"  } } */
+/* { dg-final { scan-assembler "cxbne\t%" } } */
+/* { dg-final { scan-assembler "cxbl\t%"  } } */
+/* { dg-final { scan-assembler "cxble\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/movcc-1.c b/gcc/testsuite/gcc.target/sparc/movcc-1.c
new file mode 100644
index 0000000..57ba0f9
--- /dev/null
+++ b/gcc/testsuite/gcc.target/sparc/movcc-1.c
@@ -0,0 +1,30 @@
+/* { dg-do compile } */
+/* { dg-options "-O2" } */
+
+int foo1 (int a)
+{
+  int b = a + 1;
+  if (b != 0)
+    return b;
+  return 1;
+}
+
+int foo2 (int a)
+{
+  int b = a + 1;
+  if (b < 0)
+    return b;
+  return 1;
+}
+
+int foo3 (int a)
+{
+  int b = a + 1;
+  if (b >= 0)
+    return b;
+  return 1;
+}
+
+/* { dg-final { scan-assembler "move\t%"  } } */
+/* { dg-final { scan-assembler "movpos\t%"  } } */
+/* { dg-final { scan-assembler "movneg\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/movcc-2.c b/gcc/testsuite/gcc.target/sparc/movcc-2.c
new file mode 100644
index 0000000..3af7876
--- /dev/null
+++ b/gcc/testsuite/gcc.target/sparc/movcc-2.c
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-O2" } */
+/* { dg-require-effective-target lp64 } */
+
+long foo1 (long a)
+{
+  long b = a + 1;
+  if (b != 0)
+    return b;
+  return 1;
+}
+
+long foo2 (long a)
+{
+  long b = a + 1;
+  if (b < 0)
+    return b;
+  return 1;
+}
+
+long foo3 (long a)
+{
+  long b = a + 1;
+  if (b >= 0)
+    return b;
+  return 1;
+}
+
+/* { dg-final { scan-assembler "movre\t%"  } } */
+/* { dg-final { scan-assembler "movrgez\t%"  } } */
+/* { dg-final { scan-assembler "movrlz\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/setcc-1.c b/gcc/testsuite/gcc.target/sparc/setcc-1.c
index 6065bbb..a4ff6ae 100644
--- a/gcc/testsuite/gcc.target/sparc/setcc-1.c
+++ b/gcc/testsuite/gcc.target/sparc/setcc-1.c
@@ -32,8 +32,7 @@ int gt (unsigned int a, unsigned int b)
 }
 
 /* { dg-final { scan-assembler-times "xor\t%" 2 } } */
-/* { dg-final { scan-assembler-times "subcc\t%" 2 } } */
 /* { dg-final { scan-assembler-times "addx\t%" 3 } } */
 /* { dg-final { scan-assembler-times "subx\t%" 3 } } */
-/* { dg-final { scan-assembler-times "cmp\t%" 4 } } */
+/* { dg-final { scan-assembler-times "cmp\t%" 6 } } */
 /* { dg-final { scan-assembler-not "sra\t%" { target lp64 } } } */
diff --git a/gcc/testsuite/gcc.target/sparc/setcc-2.c b/gcc/testsuite/gcc.target/sparc/setcc-2.c
index cc17c65..6d19a0a 100644
--- a/gcc/testsuite/gcc.target/sparc/setcc-2.c
+++ b/gcc/testsuite/gcc.target/sparc/setcc-2.c
@@ -32,8 +32,7 @@ int gt (unsigned int a, unsigned int b)
 }
 
 /* { dg-final { scan-assembler-times "xor\t%" 2 } } */
-/* { dg-final { scan-assembler-times "subcc\t%" 2 } } */
 /* { dg-final { scan-assembler-times "addx\t%" 3 } } */
 /* { dg-final { scan-assembler-times "subx\t%" 3 } } */
-/* { dg-final { scan-assembler-times "cmp\t%" 4 } } */
+/* { dg-final { scan-assembler-times "cmp\t%" 6 } } */
 /* { dg-final { scan-assembler-not "sra\t%" { target lp64 } } } */
diff --git a/gcc/testsuite/gcc.target/sparc/setcc-3.c b/gcc/testsuite/gcc.target/sparc/setcc-3.c
index 8a26b67..58542c1 100644
--- a/gcc/testsuite/gcc.target/sparc/setcc-3.c
+++ b/gcc/testsuite/gcc.target/sparc/setcc-3.c
@@ -18,7 +18,6 @@ int gt (unsigned long a, unsigned long b)
 }
 
 /* { dg-final { scan-assembler "xor\t%" } } */
-/* { dg-final { scan-assembler "subcc\t%" } } */
 /* { dg-final { scan-assembler-times "addxc\t%" 3 } } */
-/* { dg-final { scan-assembler-times "cmp\t%" 2 } } */
+/* { dg-final { scan-assembler-times "cmp\t%" 3 } } */
 /* { dg-final { scan-assembler-not "sra\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/setcc-4.c b/gcc/testsuite/gcc.target/sparc/setcc-4.c
index ffa4ee0..a519557 100644
--- a/gcc/testsuite/gcc.target/sparc/setcc-4.c
+++ b/gcc/testsuite/gcc.target/sparc/setcc-4.c
@@ -1,44 +1,23 @@
 /* { dg-do compile } */
 /* { dg-require-effective-target lp64 } */
-/* { dg-options "-O1 -mno-vis3" } */
+/* { dg-options "-O1 -msubxc" } */
 
-long neq (long a, long b)
-{
-  return a != b;
-}
-
-long eq (long a, long b)
+int eq (long a, long b)
 {
   return a == b;
 }
 
-long lt (unsigned long a, unsigned long b)
-{
-  return a < b;
-}
-
-long leq (unsigned long a, unsigned long b)
-{
-  return a <= b;
-}
-
-long geq (unsigned long a, unsigned long b)
+int ge (unsigned long a, unsigned long b)
 {
   return a >= b;
 }
 
-long gt (unsigned long a, unsigned long b)
+int le (unsigned long a, unsigned long b)
 {
-  return a > b;
+  return a <= b;
 }
 
-/* { dg-final { scan-assembler-times "xor\t%" 2 } } */
-/* { dg-final { scan-assembler-times "cmp\t%" 4 } } */
-/* { dg-final { scan-assembler-times "movrne\t%" 1 } } */
-/* { dg-final { scan-assembler-times "movre\t%" 1 } } */
-/* { dg-final { scan-assembler-times "movlu\t%" 1 } } */
-/* { dg-final { scan-assembler-times "movleu\t%" 1 } } */
-/* { dg-final { scan-assembler-times "movgeu\t%" 1 } } */
-/* { dg-final { scan-assembler-times "movgu\t%" 1 } } */
+/* { dg-final { scan-assembler "xor\t%" } } */
+/* { dg-final { scan-assembler-times "subxc\t%" 3 } } */
+/* { dg-final { scan-assembler-times "cmp\t%" 3 } } */
 /* { dg-final { scan-assembler-not "sra\t%" } } */
-/* { dg-final { scan-assembler-not "and\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/setcc-5.c b/gcc/testsuite/gcc.target/sparc/setcc-5.c
index 58f1ee3..1c32d41 100644
--- a/gcc/testsuite/gcc.target/sparc/setcc-5.c
+++ b/gcc/testsuite/gcc.target/sparc/setcc-5.c
@@ -1,6 +1,6 @@
 /* { dg-do compile } */
 /* { dg-require-effective-target lp64 } */
-/* { dg-options "-O1 -mvis3" } */
+/* { dg-options "-O1 -mno-vis3 -mno-subxc" } */
 
 long neq (long a, long b)
 {
@@ -34,9 +34,11 @@ long gt (unsigned long a, unsigned long b)
 
 /* { dg-final { scan-assembler-times "xor\t%" 2 } } */
 /* { dg-final { scan-assembler-times "cmp\t%" 4 } } */
-/* { dg-final { scan-assembler-times "addxc\t%" 3 } } */
+/* { dg-final { scan-assembler-times "movrne\t%" 1 } } */
 /* { dg-final { scan-assembler-times "movre\t%" 1 } } */
+/* { dg-final { scan-assembler-times "movlu\t%" 1 } } */
 /* { dg-final { scan-assembler-times "movleu\t%" 1 } } */
 /* { dg-final { scan-assembler-times "movgeu\t%" 1 } } */
+/* { dg-final { scan-assembler-times "movgu\t%" 1 } } */
 /* { dg-final { scan-assembler-not "sra\t%" } } */
 /* { dg-final { scan-assembler-not "and\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/setcc-6.c b/gcc/testsuite/gcc.target/sparc/setcc-6.c
new file mode 100644
index 0000000..7cd5894
--- /dev/null
+++ b/gcc/testsuite/gcc.target/sparc/setcc-6.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target lp64 } */
+/* { dg-options "-O1 -mvis3 -msubxc" } */
+
+long neq (long a, long b)
+{
+  return a != b;
+}
+
+long eq (long a, long b)
+{
+  return a == b;
+}
+
+long lt (unsigned long a, unsigned long b)
+{
+  return a < b;
+}
+
+long leq (unsigned long a, unsigned long b)
+{
+  return a <= b;
+}
+
+long geq (unsigned long a, unsigned long b)
+{
+  return a >= b;
+}
+
+long gt (unsigned long a, unsigned long b)
+{
+  return a > b;
+}
+
+/* { dg-final { scan-assembler-times "xor\t%" 2 } } */
+/* { dg-final { scan-assembler-times "cmp\t%" 6 } } */
+/* { dg-final { scan-assembler-times "addxc\t%" 3 } } */
+/* { dg-final { scan-assembler-times "subxc\t%" 3 } } */
+/* { dg-final { scan-assembler-not "sra\t%" } } */
+/* { dg-final { scan-assembler-not "and\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/setcc-7.c b/gcc/testsuite/gcc.target/sparc/setcc-7.c
new file mode 100644
index 0000000..fa658763
--- /dev/null
+++ b/gcc/testsuite/gcc.target/sparc/setcc-7.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-O1" } */
+
+int foo1 (int a, int i)
+{
+  return a + (i != 0);
+}
+
+int foo2 (int a, int i)
+{
+  return a - (i != 0);
+}
+
+int foo3 (int a, int b, int i)
+{
+  return a + b + (i != 0);
+}
+
+int foo4 (int a, int b, int i)
+{
+  return a - b - (i != 0);
+}
+
+int foo5 (int a, int i)
+{
+  return a + (i == 0);
+}
+
+int foo6 (int a, int i)
+{
+  return a - (i == 0);
+}
+
+/* { dg-final { scan-assembler-times "addx\t%" 3 } } */
+/* { dg-final { scan-assembler-times "subx\t%" 3 } } */
+/* { dg-final { scan-assembler-times "cmp\t%" 6 } } */
+/* { dg-final { scan-assembler-not "add\t%" } } */
+/* { dg-final { scan-assembler-not "sub\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/setcc-8.c b/gcc/testsuite/gcc.target/sparc/setcc-8.c
new file mode 100644
index 0000000..3eb9c3d
--- /dev/null
+++ b/gcc/testsuite/gcc.target/sparc/setcc-8.c
@@ -0,0 +1,39 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target lp64 } */
+/* { dg-options "-O1 -mno-vis3 -mno-subxc" } */
+
+long foo1 (long a, int i)
+{
+  return a + (i != 0);
+}
+
+long foo2 (long a, int i)
+{
+  return a - (i != 0);
+}
+
+long foo3 (long a, long b, int i)
+{
+  return a + b + (i != 0);
+}
+
+long foo4 (long a, long b, int i)
+{
+  return a - b - (i != 0);
+}
+
+long foo5 (long a, int i)
+{
+  return a + (i == 0);
+}
+
+long foo6 (long a, int i)
+{
+  return a - (i == 0);
+}
+
+/* { dg-final { scan-assembler-times "addx\t%" 3 } } */
+/* { dg-final { scan-assembler-times "subx\t%" 3 } } */
+/* { dg-final { scan-assembler-times "cmp\t%" 6 } } */
+/* { dg-final { scan-assembler-not "add\t%" } } */
+/* { dg-final { scan-assembler-not "sub\t%" } } */
diff --git a/gcc/testsuite/gcc.target/sparc/setcc-9.c b/gcc/testsuite/gcc.target/sparc/setcc-9.c
new file mode 100644
index 0000000..9da7c23
--- /dev/null
+++ b/gcc/testsuite/gcc.target/sparc/setcc-9.c
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target lp64 } */
+/* { dg-options "-O1 -mvis3" } */
+
+long foo1 (long a, long i)
+{
+  return a + (i != 0);
+}
+
+long foo3 (long a, long b, long i)
+{
+  return a + b + (i != 0);
+}
+
+long foo6 (long a, long i)
+{
+  return a - (i == 0);
+}
+
+/* { dg-final { scan-assembler-times "addxc\t%" 3 } } */
+/* { dg-final { scan-assembler-times "cmp\t%" 3 } } */
+/* { dg-final { scan-assembler-not "add\t%" } } */
+/* { dg-final { scan-assembler-not "sub\t%" } } */