Update adjustRecipesForReductions to directly use the VPlan def-use
chains for in-loop reductions to collect the reduction operations that
need adjusting.

This allows the removal of
 * ReductionChainMap
 * recording of recipes for instruction in the reduction chain
 * removes late uses of getVPValue
 * removes to need for removeVPValueFor.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D155845

Some places in the format library were identified to benefit from
basic_string's from_range constructor. At that time that constructor was
not implemented. It's implemented now so adjust the code to use this new
constructor.

Reviewed By: #libc, var-const

Differential Revision: https://reviews.llvm.org/D156022

The virtual functions getDefaultNumBlocks and getDefaultNumThreads from the kernels are
only forwarding the call to the generic device's ones. This patch removes those two
functions from the kernels (and their derived ones). Now calls are made to the device's
functions directly.

Differential Revision: https://reviews.llvm.org/D156905

Restructure three code sites that are now eliciting new warnings
from the latest GCC compiler.  One of these looks like a
legitimate problem with a reference to an expression temporary.

Fixes https://github.com/llvm/llvm-project/issues/64200.

Differential Revision: https://reviews.llvm.org/D156750

Differential Revision: https://reviews.llvm.org/D156445

The check was repeated for the fmul and fdiv case, and the caller was
already checking anyway.

The above combine matching m_FNeg to produce a new fneg always would
hide this.

Both LLVM and SPIR-V have some form of "is this float a NaN/Inf"
operation (though LLVM's uses the rather opaque "is.fpclass"
intrinsic), which is not exposed in MLIR.

This has lead to awkward workarounds in -arith-expands-ops where a NaN
test was performed by comparing an operation to itself. This commit
resolves that issue.

Reviewed By: dcaballe, kuhar

Differential Revision: https://reviews.llvm.org/D156169

5ccfa156 removed normalization from abs_path_preserve_drive but I missed adding this case back to the caller

This is failing and/or causing time outs (hard to tell which ) on
our Arm Linux bot:
https://lab.llvm.org/buildbot/#/builders/17/builds/41108

Reviewed By: #libc, Mordante, philnik

Differential Revision: https://reviews.llvm.org/D156783

The starting indices of all vector dimensions are allowed to be out-of-bounds.

E.g.:
```
// %j is allowed to be out-of-bounds (but not %i).
%0 = vector.transfer_read %m[%i, %j] ... {in_bounds = [false]} : memref<?x?xf32>, vector<5xf32>
```

This revision just updates the op documentation and adds extra test cases. Out-of-bounds starting points are already supported by the respective lowerings:
* 2D and higher-dimensional transfers are lowered to 1D transfers by `VectorToScf`. These patterns generate an `scf.if` check for every (potentially unrolled) loop iteration if the dimension is `in_bounds = false`, including the first loop iteration.
- 1D out-of-bounds transfers are lowered to in-bounds transfers by `MaterializeTransferMask`, which adds a mask to the op. The mask is defined by `vector.create_mask (dim-size) - (index)`. In case of an out-of-bounds starting point, the operand of the `vector.create_mask` op is 0 or negative. Negative operands are treated like 0 according to the documentation of `vector.create_mask`.

Differential Revision: https://reviews.llvm.org/D155719

In sorting elements can compare with themselves and sometimes assert
further down the line was triggered.

The changes are somewhat NFC, which explains the lack of test coverage.
libc++ has a debug mode that enables extra precondition checking. When
Clang is built with libc++ in that special mode, a few of Clang's tests
would fail with the libc++ assertion because Clang was not honoring the
preconditions for std::stable_sort. However, Clang would not hit the
precondition failure with any release mode STL, so the changes have no
impact on users beyond ones in this very special circumstance.

Differential Revision: https://reviews.llvm.org/D155809

Temporaries created to store worksharing loop index values were
using different types than that of the original index variables.
This caused invalid IR to be produced when an index variable was
used in binary operations which expected its original type.

Fix this by creating temporaries with the types of their original
variables and converting the loop values, that continue to use
the types that OpenMP runtime expects, to them.

Fixes https://github.com/llvm/llvm-project/issues/60870

Reviewed By: kiranchandramohan

Differential Revision: https://reviews.llvm.org/D156803

Differential Revision: https://reviews.llvm.org/D156876

In 32-bit systems, sizeof(size_t) is 4, so we fail to build an 128-bit
integer in mul_shift_mod_1e9, which ends up ignoring the top bits in the
mantissa.

This patch fixes the issue by calling the Uint constructor directly. If
it's a system that supports 128-bit integers, the constructor that takes
a value will be called, if the system doesn't support 128-bit integers
(like rv32), mantissa is already a UInt.

Reviewed By: lntue, michaelrj

Differential Revision: https://reviews.llvm.org/D156813

When generating snippets for AArch64 with --opcode-index=-1, the code
generator asserts on opcodes that are not supported according to CPU
features.

The same assertion can be triggered even when generating a serial
snippet for a supported opcode if SERIAL_VIA_NON_MEMORY_INSTR execution
mode is used and an unsupported instruction is chosen as the "other
instruction". Unlike the first case, this one may result in flaky
failures because the other instruction is randomly chosen from the
instructions suitable for serializing execution.

This patch adjusts TableGen emitter for *GenInstrInfo.inc to make
possible to query for opcode availability instead of just asserting on
unsupported ones.

~~

Huawei RRI, OS Lab

Reviewed By: courbet

Differential Revision: https://reviews.llvm.org/D146303

As noted on D154130, this was preventing path matching between normalized/unnormalized paths on some windows builds.

This patch lazily initializes queues/streams/events since their initialization
might come at a cost even if we do not use them.

To further benefit from this, AMDGPU/HSA queue management is moved into the
AMDGPUStreamManager of an AMDGPUDevice. Streams may now use different HSA queues
during their lifetime and identify busy queues.

When a Stream is requested from the resource manager, it will search for and
try to assign an idle queue. During the search for an idle queue the manager
may initialize more queues, up to the set maximum (default: 4).
When no idle queue could be found: resort to round robin selection.

With contributions from Johannes Doerfert <johannes@jdoerfert.de>

Depends on D156245

Reviewed By: kevinsala

Differential Revision: https://reviews.llvm.org/D154523

This works aims to address the issue related to larger shared memory usage in the MLIR CUDA runtime. Currently, when the shared memory usage exceeds 48KB, we need to set the CU_FUNC_ATTRIBUTE_MAX_DYNAMIC_SHARED_SIZE_BYTES attribute of the CUDA kernel appropriately. This work takes care of that by setting the attribute as required. Additionally, it includes some debug prints for better visibility and troubleshooting.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D156874

This patch permits map operands to be not specified for the target
data operation. Also emit an error if none of the map, use_device_addr,
or use_device_ptr operands are specified.

Reviewed By: TIFitis

Differential Revision: https://reviews.llvm.org/D156170

This addresses issues found by:
https://lab.llvm.org/buildbot/#/builders/30/builds/38316

as well as issues found when building locally.

Currently we use a combined metric TargetTransformInfo::TCK_SizeAndLatency
when estimating the specialization bonus. This is suboptimal, and in some
cases erroneous. For example we shouldn't be weighting the codesize decrease
attributed to constant propagation by the block frequency of the dead code.
Instead only the latency savings should be weighted by block frequency. The
total codesize savings from all the specialization arguments should be
deducted from the specialization cost.

Differential Revision: https://reviews.llvm.org/D155103

This moves packScalarIntoVectorValue from ILV to the more approriate
VPTransformState.

This new function to replace a Value with another Value saves us from re-running
the entire alias analysis when an operation has to be re-build because
additional result values have to be added (e.g., when adding more iter_args to
an scf.for).

Reviewed By: springerm

Differential Revision: https://reviews.llvm.org/D156665

Depends on D156122

Differential Revision: https://reviews.llvm.org/D156123

Differential Revision: https://reviews.llvm.org/D156122

For G_ABS with type v2s16 and sgpr inputs break down into two s32 G_ABS
instructions.

Patch by: Acim Maravic

Differential Revision: https://reviews.llvm.org/D155867

There is no need to increase the size of odd sized vectors if they are
going to be scalarized by a different rule.

Patch by: Acim Maravic

Differential Revision: https://reviews.llvm.org/D155865

Differential Revision: https://reviews.llvm.org/D156739

Differential Revision: https://reviews.llvm.org/D156736

Since we no longer support typed LLVM IR pointer types, the code can
be simplified into for example using PointerType::get directly instead
of using Type::getInt8PtrTy and Type::getInt32PtrTy etc.

Differential Revision: https://reviews.llvm.org/D156733

Added a series of optimization to the Subtract & Union function of PresburgerRelation, referring to the ISL implementation.
Add isPlainEqual to Subtract & union,also some basic check to union.
Tested it on a simple Benchmark implemented by myself to see that it can speed up the Subtract operation and Union operation, also decrease the result size.

The Benchmark can be found here: [[ https://github.com/gilsaia/llvm-project-test-fpl/blob/develop_benchmark/mlir/benchmark/presburger/Benchmark.cpp | benchmark]]

The overall results for Union & Subtract are as follows (previous benchmark has a bug,after fix that,the figure below is new)
{F28455229}

The results for each case are as follows
{F28455234}
{F28455239}

{F28455245}
{F28455246}

Reviewed By: Groverkss

Differential Revision: https://reviews.llvm.org/D156241

This work introduces `MLIR_CUDA_DEBUG` environment value and `debug_print` function to be able to debug runtimes.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D156232