Reviewed By: #libc, Mordante, philnik

Differential Revision: https://reviews.llvm.org/D156783

The starting indices of all vector dimensions are allowed to be out-of-bounds.

E.g.:
```
// %j is allowed to be out-of-bounds (but not %i).
%0 = vector.transfer_read %m[%i, %j] ... {in_bounds = [false]} : memref<?x?xf32>, vector<5xf32>
```

This revision just updates the op documentation and adds extra test cases. Out-of-bounds starting points are already supported by the respective lowerings:
* 2D and higher-dimensional transfers are lowered to 1D transfers by `VectorToScf`. These patterns generate an `scf.if` check for every (potentially unrolled) loop iteration if the dimension is `in_bounds = false`, including the first loop iteration.
- 1D out-of-bounds transfers are lowered to in-bounds transfers by `MaterializeTransferMask`, which adds a mask to the op. The mask is defined by `vector.create_mask (dim-size) - (index)`. In case of an out-of-bounds starting point, the operand of the `vector.create_mask` op is 0 or negative. Negative operands are treated like 0 according to the documentation of `vector.create_mask`.

Differential Revision: https://reviews.llvm.org/D155719

In sorting elements can compare with themselves and sometimes assert
further down the line was triggered.

The changes are somewhat NFC, which explains the lack of test coverage.
libc++ has a debug mode that enables extra precondition checking. When
Clang is built with libc++ in that special mode, a few of Clang's tests
would fail with the libc++ assertion because Clang was not honoring the
preconditions for std::stable_sort. However, Clang would not hit the
precondition failure with any release mode STL, so the changes have no
impact on users beyond ones in this very special circumstance.

Differential Revision: https://reviews.llvm.org/D155809

Temporaries created to store worksharing loop index values were
using different types than that of the original index variables.
This caused invalid IR to be produced when an index variable was
used in binary operations which expected its original type.

Fix this by creating temporaries with the types of their original
variables and converting the loop values, that continue to use
the types that OpenMP runtime expects, to them.

Fixes https://github.com/llvm/llvm-project/issues/60870

Reviewed By: kiranchandramohan

Differential Revision: https://reviews.llvm.org/D156803

Differential Revision: https://reviews.llvm.org/D156876

In 32-bit systems, sizeof(size_t) is 4, so we fail to build an 128-bit
integer in mul_shift_mod_1e9, which ends up ignoring the top bits in the
mantissa.

This patch fixes the issue by calling the Uint constructor directly. If
it's a system that supports 128-bit integers, the constructor that takes
a value will be called, if the system doesn't support 128-bit integers
(like rv32), mantissa is already a UInt.

Reviewed By: lntue, michaelrj

Differential Revision: https://reviews.llvm.org/D156813

When generating snippets for AArch64 with --opcode-index=-1, the code
generator asserts on opcodes that are not supported according to CPU
features.

The same assertion can be triggered even when generating a serial
snippet for a supported opcode if SERIAL_VIA_NON_MEMORY_INSTR execution
mode is used and an unsupported instruction is chosen as the "other
instruction". Unlike the first case, this one may result in flaky
failures because the other instruction is randomly chosen from the
instructions suitable for serializing execution.

This patch adjusts TableGen emitter for *GenInstrInfo.inc to make
possible to query for opcode availability instead of just asserting on
unsupported ones.

~~

Huawei RRI, OS Lab

Reviewed By: courbet

Differential Revision: https://reviews.llvm.org/D146303

As noted on D154130, this was preventing path matching between normalized/unnormalized paths on some windows builds.

This patch lazily initializes queues/streams/events since their initialization
might come at a cost even if we do not use them.

To further benefit from this, AMDGPU/HSA queue management is moved into the
AMDGPUStreamManager of an AMDGPUDevice. Streams may now use different HSA queues
during their lifetime and identify busy queues.

When a Stream is requested from the resource manager, it will search for and
try to assign an idle queue. During the search for an idle queue the manager
may initialize more queues, up to the set maximum (default: 4).
When no idle queue could be found: resort to round robin selection.

With contributions from Johannes Doerfert <johannes@jdoerfert.de>

Depends on D156245

Reviewed By: kevinsala

Differential Revision: https://reviews.llvm.org/D154523

This works aims to address the issue related to larger shared memory usage in the MLIR CUDA runtime. Currently, when the shared memory usage exceeds 48KB, we need to set the CU_FUNC_ATTRIBUTE_MAX_DYNAMIC_SHARED_SIZE_BYTES attribute of the CUDA kernel appropriately. This work takes care of that by setting the attribute as required. Additionally, it includes some debug prints for better visibility and troubleshooting.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D156874

This patch permits map operands to be not specified for the target
data operation. Also emit an error if none of the map, use_device_addr,
or use_device_ptr operands are specified.

Reviewed By: TIFitis

Differential Revision: https://reviews.llvm.org/D156170

This addresses issues found by:
https://lab.llvm.org/buildbot/#/builders/30/builds/38316

as well as issues found when building locally.

Currently we use a combined metric TargetTransformInfo::TCK_SizeAndLatency
when estimating the specialization bonus. This is suboptimal, and in some
cases erroneous. For example we shouldn't be weighting the codesize decrease
attributed to constant propagation by the block frequency of the dead code.
Instead only the latency savings should be weighted by block frequency. The
total codesize savings from all the specialization arguments should be
deducted from the specialization cost.

Differential Revision: https://reviews.llvm.org/D155103

This moves packScalarIntoVectorValue from ILV to the more approriate
VPTransformState.

This new function to replace a Value with another Value saves us from re-running
the entire alias analysis when an operation has to be re-build because
additional result values have to be added (e.g., when adding more iter_args to
an scf.for).

Reviewed By: springerm

Differential Revision: https://reviews.llvm.org/D156665

Depends on D156122

Differential Revision: https://reviews.llvm.org/D156123

Differential Revision: https://reviews.llvm.org/D156122

For G_ABS with type v2s16 and sgpr inputs break down into two s32 G_ABS
instructions.

Patch by: Acim Maravic

Differential Revision: https://reviews.llvm.org/D155867

There is no need to increase the size of odd sized vectors if they are
going to be scalarized by a different rule.

Patch by: Acim Maravic

Differential Revision: https://reviews.llvm.org/D155865

Differential Revision: https://reviews.llvm.org/D156739

Differential Revision: https://reviews.llvm.org/D156736

Since we no longer support typed LLVM IR pointer types, the code can
be simplified into for example using PointerType::get directly instead
of using Type::getInt8PtrTy and Type::getInt32PtrTy etc.

Differential Revision: https://reviews.llvm.org/D156733

Added a series of optimization to the Subtract & Union function of PresburgerRelation, referring to the ISL implementation.
Add isPlainEqual to Subtract & union,also some basic check to union.
Tested it on a simple Benchmark implemented by myself to see that it can speed up the Subtract operation and Union operation, also decrease the result size.

The Benchmark can be found here: [[ https://github.com/gilsaia/llvm-project-test-fpl/blob/develop_benchmark/mlir/benchmark/presburger/Benchmark.cpp | benchmark]]

The overall results for Union & Subtract are as follows (previous benchmark has a bug,after fix that,the figure below is new)
{F28455229}

The results for each case are as follows
{F28455234}
{F28455239}

{F28455245}
{F28455246}

Reviewed By: Groverkss

Differential Revision: https://reviews.llvm.org/D156241

This work introduces `MLIR_CUDA_DEBUG` environment value and `debug_print` function to be able to debug runtimes.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D156232

Differential Revision: https://reviews.llvm.org/D156228

Fixes https://github.com/llvm/llvm-project/issues/57504.

Differential Revision: https://reviews.llvm.org/D138263

This was an omission in the spec that has now been addressed
https://github.com/riscv/riscv-code-size-reduction/pull/224.

Differential Revision: https://reviews.llvm.org/D156314

This change follows from https://reviews.llvm.org/D156416 We include
`llvm-remark-size-diff` as a part of `llvm-remarkutil` under a
subcommand `size-diff`.

Differential Revision: https://reviews.llvm.org/D156515

This reverts commit 3b34d69a.

As noted in <https://github.com/llvm/llvm-project/issues/64090>, it's
more efficient to lower a partword 'atomicrmw xchg a, 0` to and amoand
with appropriate mask. There are a range of possible ways to go about
this - e.g. writing a combine based on the
`llvm.riscv.masked.atomicrmw.xchg` intrinsic, or introducing a new
interface to AtomicExpandPass to allow target-specific atomics
conversions, or trying to lift the conversion into AtomicExpandPass
itself based on querying some target hook. Ultimately I've gone with
what appears to be the simplest approach - just covering this case in
emitMaskedAtomicRMWIntrinsic. I perhaps should have given that hook a
different name way back when it was introduced.

This also handles the `atomicrmw xchg a, -1` case suggested by Craig
during review.

Fixes https://github.com/llvm/llvm-project/issues/64090

Differential Revision: https://reviews.llvm.org/D156801

Differential Revision: https://reviews.llvm.org/D155581

Add a method to the CallOpInterface to get a mutable operand range over
the function arguments.  This allows to add, remove, or change the type
of call arguments in a generic manner without having to assume that the
argument operand range is at the end of the operand list, or having to
type switch on all supported concrete operation kinds.

Alternatively, a new OpInterface could be added which inherits from
CallOpInterface and appends it with the mutable variants of the base
interface.

There will be two users of this new function in the beginning:
(1) A few passes in the Arc dialect in CIRCT already use a downstream
implementation of the alternative case mentioned above: https://github.com/llvm/circt/blob/main/include/circt/Dialect/Arc/ArcInterfaces.td#L15
(2) The BufferDeallocation pass will be modified to be able to pass
ownership of memrefs to called private functions if the caller does not
need the memref anymore by appending the function argument list with a
boolean value per memref, thus enabling earlier deallocation of the
memref which can lead to lower peak memory usage.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D156675

GFX9 subtargets from GFX90A onwards lack GDS but still have GWS.

Differential Revision: https://reviews.llvm.org/D156713

MachineVerifier does not check that DBG_VALUE, DBG_VALUE_LIST and
DBG_INSTR_REF have the expected number of operands, so printing them
(e.g. with -print-after-all) should not crash.

Differential Revision: https://reviews.llvm.org/D156226

This revision removes the createIndexConstant method, which implicitly creates constants of the
getIndexType type and updates all uses to the more explicit createIndexAttrConstant which requires
an explicit Type parameter.

This is an NFC step towards entangling index type conversion in LLVM lowering.

The selection of which index type to use requires finer granularity than the existing
implementations which all rely on pass level flags and end up in mismatches, especially on GPUs
with multiple address spaces of different capacities.

This revision also includes an NFC fix to MemRefToLLVM.cpp that prevents a crash in cases where
an integer memory space cannot be derived for a MemRef.

Differential Revision: https://reviews.llvm.org/D156854