| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
Internal functions should use a lowerCaseName, thus renamed.
|
|
|
|
Ensure intrinsics and auto-upgrades support i16, i32, and i64 for for
`nvvm.{min,max,mulhi,sad}`
- `nvvm.min` and `nvvm.max`: These are auto-upgraded to `select`
instructions but it is still nice to support the 16 bit variants just in
case any generators of IR are still trying to use these intrinsics.
- `nvvm.sad` added both the 16 and 64 bit variants, also marked this
instruction as speculateble. These directly correspond to the PTX
`sad.{u16,s16,u64,s64}` instructions.
- `nvvm.mulhi` added the 16 bit variants. These directly correspond to
the PTX `mul.hi.{s,u}16` instructions.
|
|
|
|
Break out and refactor AArch64 & ARM intrinsic updating. There's a fair amount of comonality, but let's avoid continually checking the same prefixes.
|
|
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
|
|
This is an experimental address space for strided buffers. These buffers
can have structs as elements and
a stride > 1.
These pointers allow the indexed access in units of stride, i.e., they
point at `buffer[index * stride]`.
Thus, we can use the `idxen` modifier for buffer loads.
We assign address space 9 to 192-bit buffer pointers which contain a
128-bit descriptor, a 32-bit offset and a 32-bit index. Essentially,
they are fat buffer pointers with an additional 32-bit index.
|
|
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
|
|
This code should only be run for intrinsics known to LLVM (otherwise
it will crash), not for everything that starts with "llvm.".
|
|
We can use Intrinsic::getDeclaration() here, we just have to pass
the correct arguments. This function accepts only the mangled types,
not all argument types.
|
|
Use consume_front to swallow the 'llvm.' prefix, and 'empty' to check
there's at least one character left.
|
|
Fix nvptx autoupdater's intrinsic matcher's typo'd names that used `_` (underbar), rather than '.' (dot), as a separator.
|
|
Check atomic prefix before looking for atomic instructions
|
|
With these three intrinsics it's probable faster to check the number of
arguments first and then check the names. We can also handle ctlz and
cttz in the same block.
|
|
Sort x86 intrinsics and use prefix checking.
|
|
Sort x86 intrinsics for better readability and use common prefixes to reduce number of comparisons.
|
|
startswith/endswith. NFC.
startswith/endswith wrap starts_with/ends_with and will eventually go away (to more closely match string_view)
|
|
This is an attempt at rebooting https://reviews.llvm.org/D28990
I've included AutoUpgrade changes to modify the data layout to satisfy the compatible layout check. But this does mean alloca, loads, stores, etc in old IR will automatically get this new alignment.
This should fix PR46320.
Reviewed By: echristo, rnk, tmgross
Differential Revision: https://reviews.llvm.org/D86310
|
|
Partial progress towards removing in-tree uses of `getPointerTo()`,
by employing the following options:
* Drop the call entirely if the sole purpose of it is to support a no-op
bitcast (remove the no-op bitcast as well).
* Replace with `PointerType::get()`/`PointerType::getUnqual()`
This is a NFC cleanup effort.
Reviewed By: barannikov88
Differential Revision: https://reviews.llvm.org/D155232
|
|
result (#66333)
One of the main user of these kind of coroutines is swift. There yield-once (`retcon.once`) coroutines are used to temporary "expose" pointers to internal fields of various objects creating borrow scopes.
However, in some cases it might be useful also to allow these coroutines to produce a normal result, but there is no convenient way to represent this (as compared to switched-resume kind of coroutines where C++ `co_return`
is transformed to a member / callback call on promise object).
The extension is simple: we allow continuation function to have a non-void result and accept optional extra arguments via a special `llvm.coro.end.result` intrinsic that would essentially forward them as normal results.
|
|
This reverts commit d9333e360a7c52587ab6e4328e7493b357fb2cf3.
|
|
The NVPTX intrinsics are under 'n'. Use the consume_front API, so fix
that. Refactor the helper function to group matchers on the first
component and check that first. Do similarly with the final set of
intrinsics, which have a lot of commonality in the matching. Finally
reorder the argument/return type checking wrt name checking -- the
former is going to be cheaper, so do that first before checking the
name.#
Reviewed By: tra
Differential Revision: https://reviews.llvm.org/D158445
|
|
Check for 'wasm.' prefix before proceeding, and a bit of common handling
for some of the intrinsics therein.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D158370
|
|
Consume the dbg. prefix before looking further. Also merge the matched
path.
Differential Revision: https://reviews.llvm.org/D158341
Reviewed By: nikic
|
|
It is more efficient to gate on the prefix before checking for the
individual cases. Also, the regexps weren't consistent about escaping '.'.
Differential Revision: https://reviews.llvm.org/D157923
Reviewed By: arsenm
|
|
Drop unneeded 'case'. Clean up bracing & whitespace.
Approved By: nikic
Differential Revision: https://reviews.llvm.org/D158126
|
|
There's a large amount of commonality in the riscv upgrader, make that
clearer. And check for a riscv prefix before diving in.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D157924
|
|
If the only purpose of the upgrade is a mangling change, this is
handled generically by intrinsic remangling, and does not require
any hand-written code.
|
|
The address-space intrinsics need detecting, and all have very similar
processing. Refactor to do just that.
Differential Revision: https://reviews.llvm.org/D157763
Reviewed By: jroelofs
|
|
The old syncscope parameter never really worked correctly, but
effectively gave "workgroup" scope. Use something faster than system
but more correct than before.
https://reviews.llvm.org/D157389
|
|
Differential Revision: https://reviews.llvm.org/D157551
|
|
AutoUpgrade detection of memcpy and memmove is almost
identical. Commonize the code path.
Reviewed By: jroelofs
|
|
|
|
Previously we returned i32 on RV32 and i64 on RV64. The instructions
only consume 32 bits and only produce 32 bits. For RV64, the result
is sign extended to 64 bits like *W instructions.
This patch removes this detail from the interface to improve
portability and consistency. This matches the proposal for scalar
intrinsics here https://github.com/riscv-non-isa/riscv-c-api-doc/pull/44
I've included IR autoupgrade support as well.
I'll be doing this for other builtins/intrinsics that currently use
'long' in other patches.
Reviewed By: VincentWu
Differential Revision: https://reviews.llvm.org/D154647
|
|
Currently, bf16 has been scatteredly added to the PTX codegen. This patch aims to complete the set of instructions and code path required to support bf16 data type.
Reviewed By: tra
Differential Revision: https://reviews.llvm.org/D144911
Co-authored-by: Artem Belevich <tra@google.com>
|
|
Move `AttributeMask` out of `llvm/IR/Attributes.h` to a new file
`llvm/IR/AttributeMask.h`. After doing this we can remove the
`#include <bitset>` and `#include <set>` directives from `Attributes.h`.
Since there are many headers including `Attributes.h`, but not needing
the definition of `AttributeMask`, this causes unnecessary bloating of
the translation units and slows down compilation.
This commit adds in the include directive for `llvm/IR/AttributeMask.h`
to the handful of source files that need to see the definition.
This reduces the total number of preprocessing tokens across the LLVM
source files in lib from (roughly) 1,917,509,187 to 1,902,982,273 - a
reduction of ~0.76%. This should result in a small improvement in
compilation time.
Differential Revision: https://reviews.llvm.org/D153728
|
|
|
|
This reverts commit 1159c670d40e3ef302264c681fe7e0268a550874.
Accidentally pushed wrong patch
|
|
|
|
This matches gcc. It also lets us fix a bug that the byteselect
predicate was not being evaluated in tablegen. We can't have i8
TImmLeaf in tablegen because i8 isn't a type for any register class.
I've added AutoUpgrade support for the IR intrinsics.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D152627
|
|
Produce a verifier error instead.
|
|
|
|
Re-land D145441 with data layout upgrade code fixed to not break OpenMP.
This reverts commit 3f2fbe92d0f40bcb46db7636db9ec3f7e7899b27.
Differential Revision: https://reviews.llvm.org/D149776
|
|
This reverts commit f9c1ede2543b37fabe9f2d8f8fed5073c475d850.
Differential Revision: https://reviews.llvm.org/D149758
|
|
Per discussion at
https://discourse.llvm.org/t/representing-buffer-descriptors-in-the-amdgpu-target-call-for-suggestions/68798,
we define two new address spaces for AMDGCN targets.
The first is address space 7, a non-integral address space (which was
already in the data layout) that has 160-bit pointers (which are
256-bit aligned) and uses a 32-bit offset. These pointers combine a
128-bit buffer descriptor and a 32-bit offset, and will be usable with
normal LLVM operations (load, store, GEP). However, they will be
rewritten out of existence before code generation.
The second of these is address space 8, the address space for "buffer
resources". These will be used to represent the resource arguments to
buffer instructions, and new buffer intrinsics will be defined that
take them instead of <4 x i32> as resource arguments. ptr
addrspace(8). These pointers are 128-bits long (with the same
alignment). They must not be used as the arguments to getelementptr or
otherwise used in address computations, since they can have
arbitrarily complex inherent addressing semantics that can't be
represented in LLVM. Even though, like their address space 7 cousins,
these pointers have deterministic ptrtoint/inttoptr semantics, they
are defined to be non-integral in order to prevent optimizations that
rely on pointers being a [0, [addr_max]] value from applying to them.
Future work includes:
- Defining new buffer intrinsics that take ptr addrspace(8) resources.
- A late rewrite to turn address space 7 operations into buffer
intrinsics and offset computations.
This commit also updates the "fallback address space" for buffer
intrinsics to the buffer resource, and updates the alias analysis
table.
Depends on D143437
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D145441
|
|
D138249 renamed a number of wasm intrinsics without implementing
auto-upgrade support.
Differential Revision: https://reviews.llvm.org/D146424
|
|
Auto-upgrade can be expensive, especially UpgradeDebugInfo() since it runs the verifier.
With this patch, we can specify that the imported bitcode is built with the same revision of LLVM, meaning there's no need to run any auto-upgrading. For now, limit this to just debug info since projects like Rust want to support multiple versions of LLVM at the same time (at compiler build time, not at Rust source code build time) and run a subset of the autoupgrade functionality for simplicity.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D143229
|
|
As part of removing `dbg.addr`, this upgrades any calls to `dbg.value` with
`DW_OP_deref` prepended onto the value expression.
Part of `dbg.addr` removal
Discussed in https://discourse.llvm.org/t/what-is-the-status-of-dbg-addr/62898
Differential Revision: https://reviews.llvm.org/D144793
|
|
I also ran `git clang-format` to get the headers in the right order for
the new location, which has changed the order of other headers in two
files.
|
