aboutsummaryrefslogtreecommitdiff
path: root/llvm/lib/CodeGen/TargetLoweringBase.cpp
AgeCommit message (Collapse)AuthorFilesLines
2023-06-23Darwin: Use the GOT to reference ___stack_chk_guard.Amara Emerson1-1/+2
e018cbf7208b changed the default behaviour for Darwin, and this breaks some existing software. rdar://110350601
2023-06-13[Intrinsic] Introduce reduction intrinsics for minimum/maximumAnna Thomas1-1/+2
This patch introduces the reduction intrinsic for floating point minimum and maximum which has the same semantics (for NaN and signed zero) as llvm.minimum and llvm.maximum. Reviewed-By: nikic Differential Revision: https://reviews.llvm.org/D152370
2023-06-06IR: Add llvm.ldexp and llvm.experimental.constrained.ldexp intrinsicsMatt Arsenault1-1/+14
AMDGPU has native instructions and target intrinsics for this, but these really should be subject to legalization and generic optimizations. This will enable legalization of f16->f32 on targets without f16 support. Implement a somewhat horrible inline expansion for targets without libcall support. This could be better if we could introduce control flow (GlobalISel version not yet implemented). Support for strictfp legalization is less complete but works for the simple cases.
2023-06-05[FPEnv] Intrinsics for access to FP environmentSerge Pavlov1-0/+8
The change implements intrinsics 'get_fpenv', 'set_fpenv' and 'reset_fpenv'. They are used to read floating-point environment, set it or reset to some default state. They do the same actions as C library functions 'fegetenv' and 'fesetenv'. By default these intrinsics are lowered to calls to these functions. The new intrinsics specify FP environment as a value of integer type, it is convenient of most targets where the FP state is a content of some register. Some targets however use long representations. On X86 the size of FP environment is 256 bits, and even half of this size is not a legal ibteger type. To facilitate legalization in such cases, two sets of DAG nodes is used. Nodes GET_FPENV and SET_FPENV are used when FP environment may be represented by a legal integer type. Nodes GET_FPENV_MEM and SET_FPENV_MEM consider FP environment as a region in memory, much like `fesetenv` and `fegetenv` do. They are used when target has long representation for floationg-point state. Differential Revision: https://reviews.llvm.org/D71742
2023-05-23[IR] Make stack protector symbol dso_local according to ↵Fangrui Song1-1/+1
-f[no-]direct-access-external-data There are two motivations. `-fno-pic -fstack-protector -mstack-protector-guard=global` created `__stack_chk_guard` is referenced directly on all ELF OSes except FreeBSD. This patch allows referencing the symbol indirectly with -fno-direct-access-external-data. Some Linux kernel folks want `-fno-pic -fstack-protector -mstack-protector-guard-reg=gs -mstack-protector-guard-symbol=__stack_chk_guard` created `__stack_chk_guard` to be referenced directly, avoiding R_X86_64_REX_GOTPCRELX (even if the relocation may be optimized out by the linker). https://github.com/llvm/llvm-project/issues/60116 Why they need this isn't so clear to me. --- Add module flag "direct-access-external-data" and set the dso_local property of the stack protector symbol. The module flag can benefit other LLVMCodeGen synthesized symbols that are not represented in LLVM IR. Nowadays, with `-fno-pic` being uncommon, ideally we should set "direct-access-external-data" when it is true. However, doing so would require ~90 clang/test tests to be updated, which are too much. As a compromise, we set "direct-access-external-data" only when it's different from the implied default value. Reviewed By: nickdesaulniers Differential Revision: https://reviews.llvm.org/D150841
2023-05-03Restore CodeGen/MachineValueType.h from `Support`NAKAMURA Takumi1-1/+1
This is rework of; - rG13e77db2df94 (r328395; MVT) Since `LowLevelType.h` has been restored to `CodeGen`, `MachinveValueType.h` can be restored as well. Depends on D148767 Differential Revision: https://reviews.llvm.org/D149024
2023-04-29[SelectionDAG] Rename ADDCARRY/SUBCARRY to UADDO_CARRY/USUBO_CARRY (NFC)Sergei Barannikov1-2/+2
This will make them consistent with other overflow-aware nodes. Reviewed By: RKSimon Differential Revision: https://reviews.llvm.org/D148196
2023-04-05[SelectionDAG] Expand VP SDNodes by default.Craig Topper1-4/+4
Differential Revision: https://reviews.llvm.org/D147643
2023-03-01[CodeGen] Always expand division larger than i128Nikita Popov1-1/+3
Default MaxDivRemBitWidthSupported to 128, so that divisions larger than 128 bits are always expanded, without requiring additional configuration from the target. Note that this may still emit calls to __udivti3 on 32-bit targets, which likely don't have an implementation of that builtin. However, I believe this is sufficient to fix https://github.com/llvm/llvm-project/issues/60531, because Zig must already be defining those builtins. Differential Revision: https://reviews.llvm.org/D144871
2023-02-15Use llvm::has_single_bit<uint32_t> (NFC)Kazu Hirata1-1/+1
This patch replaces isPowerOf2_32 with llvm::has_single_bit<uint32_t> where the argument is wider than uint32_t.
2023-02-14Revert "[CGP] Add generic TargetLowering::shouldAlignPointerArgs() ↵Jake Egan1-37/+0
implementation" These commits are causing a test-suite build failure on AIX. Revert for now for time to investigate. https://lab.llvm.org/buildbot/#/builders/214/builds/5779/steps/9/logs/stdio This reverts commit bd87a2449da0c82e63cebdf9c131c54a5472e3a7 and 4c72266830ffa332ebb7cf1d3bbd6c56d001fa0f.
2023-02-14[CodeGen] Trivial simplification of some getRegisterType calls. NFC.Jay Foad1-1/+1
2023-02-09Fix call to deprecated API in bd87a2449da0c82e63cebdf9c131c54a5472e3a7Alex Richardson1-1/+1
2023-02-09[CGP] Add generic TargetLowering::shouldAlignPointerArgs() implementationAlex Richardson1-0/+37
This function was added for ARM targets, but aligning global/stack pointer arguments passed to memcpy/memmove/memset can improve code size and performance for all targets that don't have fast unaligned accesses. This adds a generic implementation that adjusts the alignment to pointer size if unaligned accesses are slow. Review D134168 suggests that this significantly improves performance on synthetic benchmarks such as Dhrystone on RV32 as it avoids memcpy() calls. Reviewed By: efriedma Differential Revision: https://reviews.llvm.org/D134282
2023-02-07[NFC][TargetParser] Remove llvm/ADT/Triple.hArchibald Elliott1-1/+1
I also ran `git clang-format` to get the headers in the right order for the new location, which has changed the order of other headers in two files.
2023-01-28Use llvm::bit_ceil (NFC)Kazu Hirata1-2/+1
Note that: std::has_single_bit(X) ? X : llvm::NextPowerOf2(X); is equivalent to: std::bit_ceil(X) even for input 0.
2023-01-13DAG/GlobalISel: Fix broken/redundant setting of MODereferenceableMatt Arsenault1-4/+6
This was incorrectly setting dereferenceable on unaligned operands. getLoadMemOperandFlags does the alignment dereferenceabilty check without alignment, and then both paths went on to check isDereferenceableAndAlignedPointer. Make getLoadMemOperandFlags check isDereferenceableAndAlignedPointer, and remove the second call.
2023-01-11[NFC] Use TypeSize::getKnownMinValue() instead of TypeSize::getKnownMinSize()Guillaume Chatelet1-1/+1
This change is one of a series to implement the discussion from https://reviews.llvm.org/D141134.
2022-12-31[DAGCombiner][TLI] Do not fuse bitcast to <1 x ?> into a load/store of a vectorRoman Lebedev1-0/+6
Single-element vectors are legalized by splitting, so the the memory operations would also get scalarized. While we do have some support to reconstruct scalarized loads, we clearly don't catch everything. The comment for the affected AArch64 store suggests that having two stores was the desired outcome in the first place. This was showing as a source of *many* regressions with more aggressive ZERO_EXTEND_VECTOR_INREG recognition.
2022-12-31[NFC][TLI] Move `isLoadBitCastBeneficial()` implementation into source fileRoman Lebedev1-0/+22
... so any change to it does not cause 700 source files to be recompiled.
2022-12-01[X86] Add ExpandLargeFpConvert Pass and enable for X86Freddy Ye1-0/+2
As stated in https://discourse.llvm.org/t/rfc-llc-add-expandlargeintfpconvert-pass-for-fp-int-conversion-of-large-bitint/65528, this implementation is very similar to ExpandLargeDivRem, which expands ‘fptoui .. to’, ‘fptosi .. to’, ‘uitofp .. to’, ‘sitofp .. to’ instructions with a bitwidth above a threshold into auto-generated functions. This is useful for targets like x86_64 that cannot lower fp convertions with more than 128 bits. The expanded nodes are referring from the IR generated by `compiler-rt/lib/builtins/floattidf.c`, `compiler-rt/lib/builtins/fixdfti.c`, and etc. Corner cases: 1. For fp16: as there is no related builtins added in compliler-rt. So I mainly utilized the fp32 <-> fp16 lib calls to implement. 2. For fp80: as this pass is soft fp emulation and no fp80 instructions can help in this problem. I recommend users to deprecate this usage. For now, the implementation uses fp128 as the temporary conversion type and inserts fptrunc/ext at top/end of the function. 3. For bf16: as clang FE currently doesn't support bf16 algorithm operations (convert to int, float, +, -, *, ...), this patch doesn't consider bf16 for now. 4. For unsigned FPToI: since both default hardware behaviors and libgcc are ignoring "returns 0 for negative input" spec. This pass follows this old way to ignore unsigned FPToI. See this example: https://gcc.godbolt.org/z/bnv3jqW1M The end-to-end tests are uploaded at https://reviews.llvm.org/D138261 Reviewed By: LuoYuanke, mgehre-amd Differential Revision: https://reviews.llvm.org/D137241
2022-11-22[X86] Allow no X87 on 32-bitPhoebe Wang1-0/+9
This patch is an alternative of D100091. It solved the problems in `f80` type lowering. Reviewed By: LuoYuanke Differential Revision: https://reviews.llvm.org/D137946
2022-11-17[AMDGPU] Allow finer grain control of an unaligned access speedStanislav Mekhanoshin1-6/+6
A target can return if a misaligned access is 'fast' as defined by the target or not. In reality there can be different levels of 'fast' and 'slow'. This patch changes the boolean 'Fast' argument of the allowsMisalignedMemoryAccesses family of functions to an unsigned representing its speed. A target can still define it as it wants and the direct translation of the current code uses 0 and 1 for current false and true. This makes the change an NFC. Subsequent patch will start using an actual value of speed in the load/store vectorizer to compare if a vectorized access going to be not just fast, but not slower than before. Differential Revision: https://reviews.llvm.org/D124217
2022-09-16[VP][VE] Default VP_SREM/UREM to Expand and add generic expansion using ↵Craig Topper1-0/+5
VP_SDIV/UDIV+VP_MUL+VP_SUB. I want to default all VP operations to Expand. These 2 were blocking because VE doesn't support them and the tests were expecting them to fail a specific way. Using Expand caused them to fail differently. Seemed better to emulate them using operations that are supported. @simoll mentioned on Discord that VE has some expansion downstream. Not sure if its done like this or in the VE target. Reviewed By: frasercrmck, efocht Differential Revision: https://reviews.llvm.org/D133514
2022-09-12Move TargetTransformInfo::maxLegalDivRemBitWidth -> ↵Matthias Gehre1-0/+2
TargetLowering::maxSupportedDivRemBitWidth Also remove new-pass-manager version of ExpandLargeDivRem because there is no way yet to access TargetLowering in the new pass manager. Differential Revision: https://reviews.llvm.org/D133691
2022-09-08[llvm] Use std::size instead of llvm::array_lengthofJoe Loser1-1/+1
LLVM contains a helpful function for getting the size of a C-style array: `llvm::array_lengthof`. This is useful prior to C++17, but not as helpful for C++17 or later: `std::size` already has support for C-style arrays. Change call sites to use `std::size` instead. Differential Revision: https://reviews.llvm.org/D133429
2022-09-06[SelectionDAG] Rewrite bfloat16 softening to use the "half promotion" pathBenjamin Kramer1-1/+1
The main difference is that this preserves intermediate rounding steps, which the other route doesn't. This aligns bfloat16 more with half floats, which use this path on most targets. I didn't understand what the difference was between these softening approaches when I first added bfloat lowerings, would be nice if we only had one of them. Based on @pengfei 's D131502 Differential Revision: https://reviews.llvm.org/D133207
2022-08-18[NFCI] Move cost estimation from TargetLowering to TargetTransformInfo.Daniil Fukalov1-35/+0
TragetLowering had two last InstructionCost related `getTypeLegalizationCost()` and `getScalingFactorCost()` members, but all other costs are processed in TTI. E.g. it is not comfortable to use other TTI members in these two functions overrided in a target. Minor refactoring: `getTypeLegalizationCost()` now doesn't need DataLayout parameter - it was always passed from TTI. Reviewed By: RKSimon Differential Revision: https://reviews.llvm.org/D117723
2022-08-08[llvm] LLVM_FALLTHROUGH => [[fallthrough]]. NFCFangrui Song1-2/+2
With C++17 there is no Clang pedantic warning or MSVC C5051.
2022-07-27Use hasNItemsOrLess() in MRI::hasAtMostUserInstrs().Amara Emerson1-1/+1
2022-07-27[AArch64][GlobalISel] Add heuristics for localizing G_CONSTANT.Amara Emerson1-14/+1
This adds similar heuristics to G_GLOBAL_VALUE, querying the cost of materializing a specific constant in code size. Doing so prevents us from sinking constants which require multiple instructions to generate into use blocks. Code size savings on CTMark -Os: Program size.__text before after diff ClamAV/clamscan 381940.00 382052.00 0.0% lencod/lencod 428408.00 428428.00 0.0% SPASS/SPASS 411868.00 411876.00 0.0% kimwitu++/kc 449944.00 449944.00 0.0% Bullet/bullet 463588.00 463556.00 -0.0% sqlite3/sqlite3 284696.00 284668.00 -0.0% consumer-typeset/consumer-typeset 414492.00 414424.00 -0.0% 7zip/7zip-benchmark 595244.00 594972.00 -0.0% mafft/pairlocalalign 247512.00 247368.00 -0.1% tramp3d-v4/tramp3d-v4 372884.00 372044.00 -0.2% Geomean difference -0.0% Differential Revision: https://reviews.llvm.org/D130554
2022-07-17[CodeGen] Qualify auto variables in for loops (NFC)Kazu Hirata1-1/+1
2022-06-15[PS5] Support sin+cos->sincos optimizationPaul Robinson1-1/+1
2022-06-15Add a conversion from double to bf16Benjamin Kramer1-0/+2
This introduces a new compiler-rt function `__truncdfbf2`.
2022-06-15Promote bf16 to f32 when the target doesn't support itBenjamin Kramer1-0/+13
This is modeled after the half-precision fp support. Two new nodes are introduced for casting from and to bf16. Since casting from bf16 is a simple operation I opted to always directly lower it to integer arithmetic. The other way round is more complicated if you want to preserve IEEE semantics, so it's handled by a new __truncsfbf2 compiler-rt builtin. This is of course very bare bones, but sufficient to get a semi-softened fadd on x86. Possible future improvements: - Targets with bf16 conversion instructions can now make fp_to_bf16 legal - The software conversion to bf16 can be replaced by a trivial implementation under fast math. Differential Revision: https://reviews.llvm.org/D126953
2022-06-02[ValueTypes] Define MVTs for v128i2/v64i4 as well as i2 and i4.Hendrik Greving1-0/+24
Adds MVT::v128i2, MVT::v64i4, and implied MVT::i2, MVT::i4. Keeps MVT::i2, MVT::i4 lowering actions as expand, which should be removed once targets set this explicitly. Adjusts 11 lit tests to reflect slightly different behavior during DAG combine. Differential Revision: https://reviews.llvm.org/D125247
2022-06-01Revert "[ValueTypes] Define MVTs for v128i2/v64i4 as well as i2 and i4."Hendrik Greving1-25/+0
This reverts commit 430ac5c3029c52e391e584c6d4447e6e361fae99. Due to failures in Clang tests. Differential Revision: https://reviews.llvm.org/D125247
2022-06-01[ValueTypes] Define MVTs for v128i2/v64i4 as well as i2 and i4.Hendrik Greving1-0/+25
Adds MVT::v128i2, MVT::v64i4, and implied MVT::i2, MVT::i4. Keeps MVT::i2, MVT::i4 lowering actions as `expand`, which should be removed once targets set this explicitly. Adjusts 11 lit tests to reflect slightly different behavior during DAG combine. Differential Revision: https://reviews.llvm.org/D125247
2022-05-10Avoid 8 and 16bit switch conditions on x86Matthias Braun1-0/+5
This adds a `TargetLoweringBase::getSwitchConditionType` callback to give targets a chance to control the type used in `CodeGenPrepare::optimizeSwitchInst`. Implement callback for X86 to avoid i8 and i16 types where possible as they often incur extra zero-extensions. This is NFC for non-X86 targets. Differential Revision: https://reviews.llvm.org/D124894
2022-04-26Intrinsic for checking floating point classSerge Pavlov1-1/+2
This change introduces a new intrinsic, `llvm.is.fpclass`, which checks if the provided floating-point number belongs to any of the the specified value classes. The intrinsic implements the checks made by C standard library functions `isnan`, `isinf`, `isfinite`, `isnormal`, `issubnormal`, `issignaling` and corresponding IEEE-754 operations. The primary motivation for this intrinsic is the support of strict FP mode. In this mode using compare instructions or other FP operations is not possible, because if the value is a signaling NaN, floating-point exception `Invalid` is raised, but the aforementioned functions must never raise exceptions. Currently there are two solutions for this problem, both are implemented partially. One of them is using integer operations to implement the check. It was implemented in https://reviews.llvm.org/D95948 for `isnan`. It solves the problem of exceptions, but offers one solution for all targets, although some can do the check in more efficient way. The other, implemented in https://reviews.llvm.org/D96568, introduced a hook 'clang::TargetCodeGenInfo::testFPKind', which injects a target specific code into IR to implement `isnan` and some other functions. It is convenient for targets that have dedicated instruction to determine FP data class. However using target-specific intrinsic complicates analysis and can prevent some optimizations. A special intrinsic for value class checks allows representing data class tests with enough flexibility. During IR transformations it represents the check in target-independent way and saves it from undesired transformations. In the instruction selector it allows efficient lowering depending on the used target and mode. This implementation is an extended variant of `llvm.isnan` introduced in https://reviews.llvm.org/D104854. It is limited to minimal intrinsic support. Target-specific treatment will be implemented in separate patches. Differential Revision: https://reviews.llvm.org/D112025
2022-04-13[NFC][CodeGen] Use ArrayRef in TargetLowering functionsShao-Ce SUN1-105/+54
This patch is similar to D122557, adding an `ArrayRef` version for `setOperationAction`, `setLoadExtAction`, `setCondCodeAction`, `setLibcallName`. Reviewed By: craig.topper Differential Revision: https://reviews.llvm.org/D123467
2022-03-16Cleanup codegen includesserge-sans-paille1-1/+0
This is a (fixed) recommit of https://reviews.llvm.org/D121169 after: 1061034926 before: 1063332844 Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup Differential Revision: https://reviews.llvm.org/D121681
2022-03-10Revert "Cleanup codegen includes"Nico Weber1-0/+1
This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20. Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang, and many LLVM tests, see comments on https://reviews.llvm.org/D121169
2022-03-10Cleanup codegen includesserge-sans-paille1-1/+0
after: 1061034926 before: 1063332844 Differential Revision: https://reviews.llvm.org/D121169
2022-03-03[PS4] isPS4 and isPS4CPU are not meaningfully differentPaul Robinson1-1/+1
2022-02-18[SelectionDAG][X86] Support f16 in getReciprocalOpName.Craig Topper1-1/+3
If the "reciprocal-estimates" attribute is present and it doesn't contain "all", "none", or "default", we previously crashed on f16 operations. This patch addes an 'h' suffix' to prevent the crash. I've added simple tests that just enable the estimate for all vec-sqrt and one test case that explicitly tests the new 'h' suffix to override the default steps. There may be some frontend change needed to, but I haven't checked that yet. Reviewed By: pengfei Differential Revision: https://reviews.llvm.org/D120158
2022-02-18Revert "[SelectionDAG][X86] Support f16 in getReciprocalOpName."Craig Topper1-3/+1
This reverts commit 86b5e256628ae49193ad9962626a73bafeda2883. This wasn't supposed to be commited yet
2022-02-18[SelectionDAG][X86] Support f16 in getReciprocalOpName.Craig Topper1-1/+3
If the "reciprocal-estimates" attribute is present and it doesn't contain "all", "none", or "default", we previously crashed on f16 operations. This patch addes an 'h' suffix' to prevent the crash. I've added simple tests that just enable the estimate for all vec-sqrt and one test case that explicitly tests the new 'h' suffix to override the default steps. There may be some frontend change needed to, but I haven't checked that yet. Differential Revision: https://reviews.llvm.org/D120158
2022-02-11[ISel] Port AArch64 HADD and RHADD to ISelDavid Green1-0/+6
This ports the aarch64 combines for HADD and RHADD over to DAG combine, so that they can be used in more architectures (notably MVE in a followup patch). They are renamed to AVGFLOOR and AVGCEIL in the process, to avoid confusion with instructions such as X86 hadd. The code was also rewritten slightly to remove the AArch64 idiosyncrasies. The general pattern for a AVGFLOORS is %xe = sext i8 %x to i32 %ye = sext i8 %y to i32 %a = add i32 %xe, %ye %r = lshr i32 %a, 1 %t = trunc i32 %r to i8 An AVGFLOORU is equivalent with zext. Because of the truncate lshr==ashr, as the top bits are not demanded. An AVGCEIL also includes an extra rounding, so includes an extra add of 1. Differential Revision: https://reviews.llvm.org/D106237
2022-01-26Revert "Rename llvm::array_lengthof into llvm::size to match std::size from ↵Benjamin Kramer1-1/+1
C++17" This reverts commit ef8206320769ad31422a803a0d6de6077fd231d2. - It conflicts with the existing llvm::size in STLExtras, which will now never be called. - Calling it without llvm:: breaks C++17 compat
generated by cgit v1.1 at 2025-09-30 17:00:25 +0000