aboutsummaryrefslogtreecommitdiff
path: root/llvm/lib/CodeGen/MachineSink.cpp
AgeCommit message (Collapse)AuthorFilesLines
2024-04-24[CodeGen] Make the parameter TRI required in some functions. (#85968)Xu Zhang1-1/+1
Fixes #82659 There are some functions, such as `findRegisterDefOperandIdx` and `findRegisterDefOperand`, that have too many default parameters. As a result, we have encountered some issues due to the lack of TRI parameters, as shown in issue #82411. Following @RKSimon 's suggestion, this patch refactors 9 functions, including `{reads, kills, defines, modifies}Register`, `registerDefIsDead`, and `findRegister{UseOperandIdx, UseOperand, DefOperandIdx, DefOperand}`, adjusting the order of the TRI parameter and making it required. In addition, all the places that call these functions have also been updated correctly to ensure no additional impact. After this, the caller of these functions should explicitly know whether to pass the `TargetRegisterInfo` or just a `nullptr`.
2024-02-15[CodeGen] Simplify updateLiveIn in MachineSink (#79831)Jay Foad1-7/+2
When a whole register is added a basic block's liveins, use LaneBitmask::getAll for the live lanes instead of trying to calculate an accurate mask of the lanes that comprise the register. This simplifies the code and matches other places where a whole register is marked as livein. This also avoids problems when regunits that are synthesized by TableGen to represent ad hoc aliasing have a lane mask of 0. Fixes #78942
2023-12-13[MachineSink] Clear kill flags of sunk addressing mode registers (#75072)Momchil Velikov1-5/+16
When doing sink-and-fold, the MachineSink clears the "killed" flags of the operands of the sunk (and deleted) instruction. However, this is not always sufficient. In some cases we can create the new load/store instruction with operands other than the ones present in the deleted instruction. One such example is folding a zero word extend into a memory load on AArch64. The zero-extend is represented by a pair of instructions - `MOV` (i.e. `ORRwrs`) followed by a `SUBREG_TO_REG`. The `SUBREG_TO_REG` is deleted (it is the sunk instruction), but the new load instruction mentions operands "killed" in the `MOV`, which is no longer correct. To fix this, clear the "killed" flags of the registers participating in the addressing mode.
2023-11-24 [MachineSink] Some more preserving of debug location when rematerialising ↵Momchil Velikov1-1/+3
an instruction to replace a COPY (#73155) Somewhat similar to ef9bcace834e63f25bbbc5e8e2b615f89d85fb2f ([MachineSink][AArch64] Preserve debug location when rematerialising an instruction to replace a COPY (#72685)) reuse the debug location of the COPY, iff the rematerialised instruction did not have a location. Fixes a regression in `DebugInfo/AArch64/constant-dbgloc.ll` after enabling sink-and-fold.
2023-11-21[MachineSink][AArch64] Preserve debug location when rematerialising an ↵Momchil Velikov1-3/+1
instruction to replace a COPY (#72685) Fixes a regression in `tools/lldb-dap/optimized/TestDAP_optimized.py` caused by enabling "sink-and-fold" in MachineSink.
2023-11-11[MachineSink] Drop debug info for instructions deleted by sink-and-fold (#71443)Momchil Velikov1-19/+12
After performing sink-and-fold over a COPY, the original instruction is replaced with one that produces its output in the destination of the copy. Its value is still available (in a hard register), so if there are debug instructions which refer to the (now deleted) virtual register they could be updated to refer to the hard register, in principle. However, it's not clear how to do that, moreover in some cases the debug instructions may need to be replicated proportionally to the number of the COPY instructions replaced and in some extreme cases we can end up with quadratic increase in the number of debug instructions, e.g: int f(int); void g(int x) { int y = x + 1; int t0 = y; f(t0); int t1 = y; f(t1); }
2023-10-12[MachineSink] Reduce the number of unnecessary invalidations of ↵Momchil Velikov1-2/+3
StoreInstrCache (NFC) (#68676) Don't invalidate the cache when erasing instructions which cannot ever appear in the cache.
2023-10-12[MachineSink] Use LLVM ADTs (NFC) (#68677)Momchil Velikov1-10/+10
Replace a few uses of `std::map` with `llvm::DenseMap`.
2023-10-06[MachineSink] Fix crash due to use-after-free in a MachineInstr* cache.Amara Emerson1-0/+2
After the SinkAndFold optimization was enabled, we saw some crashes with GISel due to SinkAndFold erasing an MI while a reference was being held in a cache.
2023-10-06AMDGPU: Fix temporal divergence introduced by machine-sink (#67456)Petar Avramovic1-0/+4
Temporal divergence that was present in input or introduced in IR transforms, like code-sinking or LICM, is handled in SIFixSGPRCopies by changing sgpr source instr to vgpr instr. After 5b657f5, that moved LICM after AMDGPUCodeGenPrepare, machine-sinking can introduce temporal divergence by sinking instructions outside of the cycle. Add isSafeToSink callback in TargetInstrInfo.
2023-10-06Revert "MachineSink: Fix sinking VGPR def out of a divergent loop"Petar Avramovic1-11/+4
This reverts commit 3f8ef57bede94445b1a1042c987cc914a886e7ff.
2023-10-04[AArch64] Fix an incorrect handling of debug values in MachineSink (#68107)Momchil Velikov1-1/+4
2023-09-29[AArch64] Fix a compiler crash in MachineSink (#67705)Momchil Velikov1-17/+19
There were a couple of issues with maintaining register def/uses held in `MachineRegisterInfo`: * when an operand is changed from one register to another, the corresponding instruction must already be inserted into the function, or MRI won't be updated * when traversing the set of all uses of a register, that set must not change
2023-09-25[MachineSink][AArch64] Sink instruction copies when they can replace copy ↵Momchil Velikov1-29/+268
into hard register or folded into addressing mode This patch adds a new code transformation to the `MachineSink` pass, that tries to sink copies of an instruction, when the copies can be folded into the addressing modes of load/store instructions, or replace another instruction (currently, copies into a hard register). The criteria for performing the transformation is that: * the register pressure at the sink destination block must not exceed the register pressure limits * the latency and throughput of the load/store or the copy must not deteriorate * the original instruction must be deleted Reviewed By: dmgreen Differential Revision: https://reviews.llvm.org/D152828
2023-08-14[CodeGen] Set regunitmasks for leaf regs to all instead of noneJay Foad1-3/+2
This simplifies every use of MCRegUnitMaskIterator. Differential Revision: https://reviews.llvm.org/D157864
2023-08-09Remove a reference to rdar://problem/8030636Jon Roelofs1-1/+1
The surrounding comment has more than enough context to describe the problem.
2023-08-02MachineSink: Fix strict weak ordering in GetAllSortedSuccessorsDanila Kutenin1-1/+1
CodeGen/X86/pseudo_cmov_lower2.ll fails using libc++ debug mode (D150264) without this change. Reviewed By: MaskRay, aeubanks Differential Revision: https://reviews.llvm.org/D155811
2023-07-18MachineSink: Fix sinking VGPR def out of a divergent loopMatt Arsenault1-4/+11
This fixes sinking a VGPR def out of a loop past the reconvergence point at the SI_END_CF. There was a prior fix which introduced blockPrologueInterferes (D121277) to fix the same basic problem for the post RA sink. This also had the special case isIgnorableUse case which was incorrect, because in some contexts the exec use is not ignorable. I'm thinking about a new way to represent this which will avoid needing hasIgnorableUse and isBasicBlockPrologue, which would function more like the exception handling. Fixes: SWDEV-407790 https://reviews.llvm.org/D155343
2023-07-14MachineSink: Remove unnecessary empty block checkMatt Arsenault1-2/+0
2023-07-14MachineSink: Move helper function and use more constMatt Arsenault1-39/+41
2023-06-16[MC] Add MCRegisterInfo::regunits for iteration over register unitsSergei Barannikov1-5/+4
Reviewed By: foad Differential Revision: https://reviews.llvm.org/D152098
2023-06-01[CodeGen] Make use of MachineInstr::all_defs and all_uses. NFCI.Jay Foad1-18/+9
Differential Revision: https://reviews.llvm.org/D151424
2023-05-16[MachineSink] Don't reject sinking because of dead def in ↵Jonas Paulsson1-6/+4
isProfitableToSinkTo(). An instruction should be sunk (if otherwise legal and profitable) regardless of if it has a dead def of a physreg or not. Physreg defs are checked in other places and sinking is only done with dead defs of regs that are not live into the target MBB. Differential Revision: https://reviews.llvm.org/D150447 Reviewed By: sebastian-ne, arsenm
2023-04-18[MC] Use subregs/superregs instead of MCSubRegIterator/MCSuperRegIterator. NFC.Jay Foad1-2/+2
Differential Revision: https://reviews.llvm.org/D148613
2023-04-17Fix uninitialized pointer members in CodeGenAkshay Khadse1-9/+9
This change initializes the members TSI, LI, DT, PSI, and ORE pointer feilds of the SelectOptimize class to nullptr. Reviewed By: LuoYuanke Differential Revision: https://reviews.llvm.org/D148303
2023-02-07[CodeGen] Define and use MachineOperand::getOperandNoJay Foad1-1/+1
This is a helper function to very slightly simplify many calls to MachineInstruction::getOperandNo. Differential Revision: https://reviews.llvm.org/D143250
2023-01-13[CodeGen] Remove uses of Register::isPhysicalRegister/isVirtualRegister. NFCCraig Topper1-9/+9
Use isPhysical/isVirtual methods. Reviewed By: foad Differential Revision: https://reviews.llvm.org/D141715
2023-01-06[DebugInfo][NFC] Add new MachineOperand type and change DBG_INSTR_REF syntaxStephen Tozer1-1/+1
This patch makes two notable changes to the MIR debug info representation, which result in different MIR output but identical final DWARF output (NFC w.r.t. the full compilation). The two changes are: * The introduction of a new MachineOperand type, MO_DbgInstrRef, which consists of two unsigned numbers that are used to index an instruction and an output operand within that instruction, having a meaning identical to first two operands of the current DBG_INSTR_REF instruction. This operand is only used in DBG_INSTR_REF (see below). * A change in syntax for the DBG_INSTR_REF instruction, shuffling the operands to make it resemble DBG_VALUE_LIST instead of DBG_VALUE, and replacing the first two operands with a single MO_DbgInstrRef-type operand. This patch is the first of a set that will allow DBG_INSTR_REF instructions to refer to multiple machine locations in the same manner as DBG_VALUE_LIST. Reviewed By: jmorse Differential Revision: https://reviews.llvm.org/D129372
2022-07-17[CodeGen] Qualify auto variables in for loops (NFC)Kazu Hirata1-1/+1
2022-06-24[MachineSink] Clear kill flags on operands outside loopCarl Ritson1-0/+6
If an instruction is sunk into a loop then any kill flags on operands declared outside the loop must be cleared as these will be live for all loop iterations. Fixes #46827 Reviewed By: MatzeB Differential Revision: https://reviews.llvm.org/D126754
2022-06-21[machinesink] fix debug invariance issueMarkus Lavin1-2/+2
Do not include debug instructions when comparing block sizes with thresholds. Differential Revision: https://reviews.llvm.org/D127208
2022-06-15[CodeGen] Fix the bug of machine sinkLuo, Yuanke1-0/+2
The use operand may be undefined. In that case we can just continue to check the next operand since it won't increase register pressure. Differential Revision: https://reviews.llvm.org/D127848
2022-05-26[MachineSink] replace MachineLoop with MachineCycleChen Zheng1-94/+96
reapply 62a9b36fcf728b104ea87e6eb84c0be69b779df7 and fix module build failue: 1: remove MachineCycleInfoWrapperPass in MachinePassRegistry.def MachineCycleInfoWrapperPass is a anylysis pass, should not be there. 2: move the definition for MachineCycleInfoPrinterPass to cpp file. Otherwise, there are module conflicit for MachineCycleInfoWrapperPass in MachinePassRegistry.def and MachineCycleAnalysis.h after 62a9b36fcf728b104ea87e6eb84c0be69b779df7. MachineCycle can handle irreducible loop. Natural loop analysis (MachineLoop) can not return correct loop depth if the loop is irreducible loop. And MachineSink is sensitive to the loop depth, see MachineSinking::isProfitableToSinkTo(). This patch tries to use MachineCycle so that we can handle irreducible loop better. Reviewed By: sameerds, MatzeB Differential Revision: https://reviews.llvm.org/D123995
2022-05-24Revert "[MachineSink] replace MachineLoop with MachineCycle"Chen Zheng1-96/+94
This reverts commit 62a9b36fcf728b104ea87e6eb84c0be69b779df7. Cause build failure on lldb incremental buildbot: https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/43994/changes
2022-05-24[MachineSink] replace MachineLoop with MachineCycleChen Zheng1-94/+96
MachineCycle can handle irreducible loop. Natural loop analysis (MachineLoop) can not return correct loop depth if the loop is irreducible loop. And MachineSink is sensitive to the loop depth, see MachineSinking::isProfitableToSinkTo(). This patch tries to use MachineCycle so that we can handle irreducible loop better. Reviewed By: sameerds, MatzeB Differential Revision: https://reviews.llvm.org/D123995
2022-03-22[MachineSink] Check block prologue interferenceCarl Ritson1-2/+53
Sinking must check for interference between the block prologue and the instruction being sunk. Specifically check for clobbering of uses by the prologue, and overwrites to prologue defined registers by the sunk instruction. Reviewed By: rampitec, ruiling Differential Revision: https://reviews.llvm.org/D121277
2022-03-16Cleanup codegen includesserge-sans-paille1-1/+1
This is a (fixed) recommit of https://reviews.llvm.org/D121169 after: 1061034926 before: 1063332844 Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup Differential Revision: https://reviews.llvm.org/D121681
2022-03-10Revert "Cleanup codegen includes"Nico Weber1-1/+1
This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20. Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang, and many LLVM tests, see comments on https://reviews.llvm.org/D121169
2022-03-10Cleanup codegen includesserge-sans-paille1-1/+1
after: 1061034926 before: 1063332844 Differential Revision: https://reviews.llvm.org/D121169
2022-03-02[MachineSink] Disable if there are any irreducible cyclesNikita Popov1-0/+12
This is an alternative to D120330, which disables MachineSink for functions with irreducible cycles entirely. This avoids both the correctness problem, and ensures we don't perform non-profitable sinks into cycles. At the same time, it may also disable profitable sinks in the same function. This can be made more precise by using MachineCycleInfo in the future. Fixes https://github.com/llvm/llvm-project/issues/53990. Differential Revision: https://reviews.llvm.org/D120800
2022-02-16[MachineSink] Use SkipPHIsAndLabels for sink insertion pointsCarl Ritson1-5/+6
For AMDGPU the insertion point for a block may not be the first non-PHI instruction. This happens when a block contains EXEC mask manipulation related to control flow (converging lanes). Use SkipPHIsAndLabels to determine the block insertion point so that the target can skip any block prologue instructions. Reviewed By: rampitec, ruiling Differential Revision: https://reviews.llvm.org/D119399
2022-02-12[MachineSink] Inline getRegUnitsBenjamin Kramer1-21/+11
Reg unit sets are uniqued, so no need to wrap it in a set.
2022-01-18[MachineSink] Allow sinking of constant or ignorable physreg usesVang Thao1-3/+8
For AMDGPU, any use of the physical register EXEC prevents sinking even if it is not a real physical register read. Add check to see if a physical register use can be ignored for sinking. Also perform same constant and ignorable physical register check when considering sinking in loops. https://reviews.llvm.org/D116053
2021-11-25[llvm] Use range-based for loops (NFC)Kazu Hirata1-8/+4
2021-11-12Fix minor deficiency in machine-sink.Markus Lavin1-1/+2
Register uses that are MRI->isConstantPhysReg() should not inhibit sinking transformation. Reviewed By: StephenTozer Differential Revision: https://reviews.llvm.org/D111531
2021-11-09[llvm] Use MachineBasicBlock::{successors,predecessors} (NFC)Kazu Hirata1-6/+2
2021-11-06[llvm] Use llvm::reverse (NFC)Kazu Hirata1-2/+1
2021-10-31[CodeGen] Use make_early_inc_range (NFC)Kazu Hirata1-19/+16
2021-10-18[MachineSink] Compile time improvement for large testcases which has many ↵Bing1 Yu1-2/+2
kill flags We did a experiment and observed dramatic decrease on compilation time which spent on clearing kill flags. Before: Number of BasicBlocks:33357 Number of Instructions:162067 Number of Cleared Kill Flags:32869 Time of handling kill flags(ms):1.607509e+05 After: Number of BasicBlocks:33357 Number of Instructions:162067 Number of Cleared Kill Flags:32869 Time of handling kill flags:3.987371e+03 Reviewed By: MatzeB Differential Revision: https://reviews.llvm.org/D111688
2021-07-08[DebugInfo][InstrRef][4/4] Support DBG_INSTR_REF through all backend passesJeremy Morse1-1/+1
This is a cleanup patch -- we're now able to support all flavours of variable location in instruction referencing mode. This patch updates various tests for debug instructions to be broader: numerous code paths try to ignore debug isntructions, and they now have to ignore the additional DBG_PHI and DBG_INSTR_REFs that we can generate. A small amount of rework happens for LiveDebugVariables: as we don't need to track live intervals through regalloc any more, we can get away with unlinking debug instructions before regalloc, then re-inserting them after. Note that this isn't (yet) true of DBG_VALUE_LISTs, they still have to go through live interval tracking. In SelectionDAG, add a helper lambda that emits half-formed DBG_INSTR_REFs for arguments in instr-ref mode, DBG_VALUE otherwise. This is one of the final locations where DBG_VALUEs are emitted for vreg arguments. X86InstrInfo now un-sets the debug instr number on SUB instructions that get mutated into CMP instructions. As the instruction no longer computes a subtraction, we can't use it for variable locations. Differential Revision: https://reviews.llvm.org/D88898
generated by cgit v1.1 at 2025-10-04 00:18:08 +0000