Depend on D157628, which set the cost of extends 0 because they will fold into
the s/urhadd.

Reviewed By: kmclaughlin
Differential Revision: https://reviews.llvm.org/D159273

This patch extends MaskedVectorizeOp so that it can be used for
"regular" (as opposed to "masked") vectorization as well. While we can
already use VectorizeOp for "regular" vectorization, that Op will also
apply various patterns on top of vectorization. That means that at the
moment, when testing the vectorizer with VectorizeOp, we are effectively
testing "vectorization + patterns", i.e. 2 things at a time.

With these updates, you can trigger "regular" vectorization with
MaskedVectorizeOp by simply skipping the vector sizes:

  transform.structured.masked_vectorize %target : !transform.any_op

Following this change we should probably also rename this Op.

Differential Revision: https://reviews.llvm.org/D157774

Add support for syntax highlighting assembly. The patch introduces new
RAII helper called WithMarkup that takes care of both emitting colors
and markup annotations. It makes adding markup easier and ensures colors
and annotations remain consistent.

This patch adopts the new helper in the AArch64 backend. If your backend
already uses markup annotations, adoption is as easy as using the new
MCInstPrinter::markup overload.

Differential revision: https://reviews.llvm.org/D159162

The revision adds the dbg label intrinsic to the allow list of
operations that are legal to inline.

Reviewed By: zero9178

Differential Revision: https://reviews.llvm.org/D159359

The RUN here was mising ":" and there was no check file passed
to FileCheck. This has been the case since this was originally added.

UBSAN is the only sanitizer that is available for OpenBSD,
but it does not add a simple "-fsanitize=undefined" instead
it adds a bunch of smaller options. So check for those like
the existing tests do.

These were missing the ":" on the end.

The isRV64 field contains the same information, and we can derive XLen
from that.

Differential Revision: https://reviews.llvm.org/D159306

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D159343

This work adds CTA Cluster barrier intrinsics for sm_90 in NVVM dialect. They are already supported in LLVM core, so this work uses the existing intrinsics.

Differential Revision: https://reviews.llvm.org/D158720

This work introduces special registers such as cluster ID, dimensions, and more for managing CTA clusters, which are groups of CTAsthat can synchronize and communicate through shared memory. This is for Nvidia's sm_90 capability.

Differential Revision: https://reviews.llvm.org/D158588

This is intended to be a non-functional change. This patch removes
OBSCURE_COPY in favour of using `forceDisableTriviallyReMaterializable`.

Reviewed By: paulwalker-arm

Differential Revision: https://reviews.llvm.org/D159194

This patch adds a test that uses a target region to set a scalar value. It also
adds rules in lit.cfg to handle fortran testing.

Reviewed By: jdoerfert

Differential Revision: https://reviews.llvm.org/D159216

https://github.com/llvm/llvm-project/issues/65015 shows a case where
tryCombineMULLWithUZP1 could attempt to look at the wrong operand of another
user instruction. This adds an extra else as if we don't find the right opcode,
we don't need to check the operands.

Differential Revision: https://reviews.llvm.org/D159282

In the process of splitting out the liveness tracking, I ran into
these cases which should have been caught. There are still missing
errors for some cases in the entry block.

https://reviews.llvm.org/D127104

Size-type inconsistency (signedness) causes confusion and even bugs.
For example when signed compared to unsigned the result might not
be expected. Summary of this commit:

Related APIs changes:
1. getDynamicExtent() returns signed version of extent;
2. Add getDynamicElementCountWithOffset() for offset version of element count;
3. getElementExtent() could be 0, add defensive checking for
   getDynamicElementCount(), if element is of zero-length, try
   ConstantArrayType::getSize() as element count;

Related checker changes:
1. ArrayBoundCheckerV2: add testcase for signed <-> unsigned comparison from
   type-inconsistency results by getDynamicExtent()
2. ExprInspection: use more general API to report more results

Fixes https://github.com/llvm/llvm-project/issues/64920

Reviewed By: donat.nagy, steakhal

Differential Revision: https://reviews.llvm.org/D158499

Add tests for `hasher hash_function() const` and `key_equal key_eq() const`
observers in unordered containers.

Differential Revision: https://reviews.llvm.org/D119703

If llvm-reduce is going to unconditionally verify functions with
LiveIntervals, it needs to be tolerant of generic vregs.

https://reviews.llvm.org/D133813

https://reviews.llvm.org/D158999

https://reviews.llvm.org/D158998

These codegen correctly but f64 doesn't. This prevents losing fast
math flags on the way to the underlying intrinsic.

https://reviews.llvm.org/D158997

https://reviews.llvm.org/D158996

https://reviews.llvm.org/D159006

powr is just pow with the assumption that x >= 0, otherwise nan. This
fires at least 6 times in luxmark

https://reviews.llvm.org/D158908

Also fixes not handling the partially undef case.

https://reviews.llvm.org/D158905

This was requiring all fast math flags, which is practically
useless. This wouldn't fire using all the standard OpenCL fast math
flags. This only needs afn nnan and ninf.

https://reviews.llvm.org/D158904

This is a way to prevent the register allocator from inserting instructions
which behave differently for different runtime vector-lengths, inside a
call-sequence which changes the streaming-SVE mode before/after the call.

I've considered using BUNDLEs in Machine IR, but found that using this is
not possible for a few reasons:
* Most passes don't look inside BUNDLEs, but some passes would need to
  look inside these call-sequence bundles, for example the PrologEpilog
  pass (to remove the CALLSEQSTART/END), a PostRA pass to remove COPY
  instructions, or the AArch64PseudoExpand pass.
* Within the streaming-mode-changing call sequence, one of the instructions
  is a CALLSEQEND. The corresponding CALLSEQBEGIN (AArch64::ADJCALLSTACKUP)
  is outside this sequence. This means we'd end up with a BUNDLE that has
  [SMSTART, COPY, BL, ADJCALLSTACKUP, COPY, SMSTOP]. The MachineVerifier
  doesn't accept this, and we also can't move the CALLSEQSTART into the
  call sequence.

Maybe in the future we could model this differently by modelling
the runtime vector-length as a value that's used by certain operations
(similar to e.g. NCZV flags) and clobbered by SMSTART/MMSTOP, such that the
register allocator can consider these as actual dependences and avoid
rematerialization. For now we just want to address the immediate problem.

Reviewed By: paulwalker-arm, aemerson

Differential Revision: https://reviews.llvm.org/D159193

When a function is compiled to be in Streaming(-compatible) mode, the full
set of SVE instructions may not be available. This patch adds an interface
to query that and changes the codegen for FADDA (not legal in Streaming-SVE
mode) to instead be expanded for fixed-length vectors, or otherwise not to
code-generate for scalable vectors.

Reviewed By: david-arm

Differential Revision: https://reviews.llvm.org/D156109

Fold select (fcmp oeq x, 0), (fmul x, y), x => x

This cleans up a pattern left behind by denormal range checks under
denormals are zero.

The pattern starts out as something like:
  x = x < smallest_normal ? x * K : x;

The comparison folds to an == 0 when the denormal mode treats input
denormals as zero. This makes library denormal checks free after
linked into DAZ enabled code.

alive2 is mostly happy with this, but there are some issues. First,
there are many reported failures in some of the negative tests that
happen to trigger some preexisting canonicalize introducing
combine. Second, alive2 is incorrectly asserting that denormals must
be flushed with the DAZ modes. It's allowed to drop a canonicalize.

https://reviews.llvm.org/D157030

Followup to D158364

Also, final fix for Issue #59902 which noted that the snippet should just return 1

Followup to D158364

HLFIR lowering always adds hlfir.declare when symbols are bound to their
address allocated on the stack. Ensure that the declare is placed along
with the alloca if it is hoisted. And always return the mlir value that
is bound to the symbol (i.e the alloca in FIR lowering and the declare
in HLFIR lowering).

Context: Loop index variables in OpenMP parallel regions should be
privatised to work correctly.

Reviewed By: tblah

Differential Revision: https://reviews.llvm.org/D158594

This reverts commit e698695f. The commit caused
invalid AddressSanitizer: stack-use-after-scope errors.

See https://reviews.llvm.org/D74094#4633785 for details.

Differential Revision: https://reviews.llvm.org/D159346

There are some slow/congested bots, that can't go idle in 10 secs, see https://github.com/llvm/llvm-project/issues/64964

Differential Revision: https://reviews.llvm.org/D159338