aboutsummaryrefslogtreecommitdiff
path: root/llvm/lib/Transforms/Instrumentation/ControlHeightReduction.cpp
AgeCommit message (Collapse)AuthorFilesLines
6 daysReapply "[ControlHeightReduction] Drop lifetime annotations where necessary" ↵Aiden Grossman1-8/+37
(#160640) Reapplies #159686 This reverts commit 4f33d7b7a9f39d733b7572f9afbf178bca8da127. The original landing of this patch had an issue where it would try and hoist allocas into the entry block that were in the entry block. This would end up actually moving them lower in the block potentially after users, resulting in invalid IR. This update fixes this by ensuring that we are only hoisting static allocas that have been sunk into a split basic block. A regression test has been added. Integration tested using a three stage build of clang with IRPGO enabled.
9 daysRevert "[ControlHeightReduction] Drop lifetime annotations where necessary ↵Aiden Grossman1-37/+8
(#159686)" This reverts commit a00450944d2a91aba302954556c1c23ae049dfc7. Looks like this one is actually breaking the buildbots. Reverting the switch back to IRPGO did not fix things.
9 days[ControlHeightReduction] Drop lifetime annotations where necessary (#159686)Aiden Grossman1-8/+37
ControlHeightReduction will duplicate some blocks and insert phi nodes in exit blocks of regions that it operates on for any live values. This includes allocas. Having a lifetime annotation refer to a phi node was made illegal in 92c55a315eab455d5fed2625fe0f61f88cb25499, which causes the verifier to fail after CHR. There are some cases where we might not need to drop lifetime annotations (usually because we do not need the phi to begin with), but drop all annotations for now to be conservative. Fixes #159621.
2025-03-23[Transforms] Use *Set::insert_range (NFC) (#132652)Kazu Hirata1-11/+4
We can use *Set::insert_range to collapse: for (auto Elem : Range) Set.insert(E); down to: Set.insert_range(Range); In some cases, we can further fold that into the set declaration.
2025-03-19[Transforms] Use *Set::insert_range (NFC) (#132056)Kazu Hirata1-4/+3
DenseSet, SmallPtrSet, SmallSet, SetVector, and StringSet recently gained C++23-style insert_range. This patch replaces: Dest.insert(Src.begin(), Src.end()); with: Dest.insert_range(Src); This patch does not touch custom begin like succ_begin for now.
2025-01-24[NFC][DebugInfo] Use iterator moveBefore at many call-sites (#123583)Jeremy Morse1-1/+1
As part of the "RemoveDIs" project, BasicBlock::iterator now carries a debug-info bit that's needed when getFirstNonPHI and similar feed into instruction insertion positions. Call-sites where that's necessary were updated a year ago; but to ensure some type safety however, we'd like to have all calls to moveBefore use iterators. This patch adds a (guaranteed dereferenceable) iterator-taking moveBefore, and changes a bunch of call-sites where it's obviously safe to change to use it by just calling getIterator() on an instruction pointer. A follow-up patch will contain less-obviously-safe changes. We'll eventually deprecate and remove the instruction-pointer insertBefore, but not before adding concise documentation of what considerations are needed (very few).
2024-11-21[ControlHeightReduction] Add assert to avoid underflow (#116339)Wu Yingcong1-0/+1
`NumCHRedBranches - 1` is used later, we should add an assertion to make sure it will not underflow.
2024-11-06[Instrumentation] Remove unused includes (NFC) (#115117)Kazu Hirata1-1/+0
Identified with misc-include-cleaner.
2024-08-03[Transforms] Construct SmallVector with ArrayRef (NFC) (#101851)Kazu Hirata1-2/+1
2024-06-29[IRBuilder] Don't include Module.h (NFC) (#97159)Nikita Popov1-0/+1
This used to be necessary to fetch the DataLayout, but isn't anymore.
2024-06-12Reapply "[llvm][IR] Extend BranchWeightMetadata to track provenance o… ↵Paul Kirth1-1/+1
(#95281) …f weights" #95136 Reverts #95060, and relands #86609, with the unintended code generation changes addressed. This patch implements the changes to LLVM IR discussed in https://discourse.llvm.org/t/rfc-update-branch-weights-metadata-to-allow-tracking-branch-weight-origins/75032 In this patch, we add an optional field to MD_prof meatdata nodes for branch weights, which can be used to distinguish weights added from llvm.expect* intrinsics from those added via other methods, e.g. from profiles or inserted by the compiler. One of the major motivations, is for use with MisExpect diagnostics, which need to know if branch_weight metadata originates from an llvm.expect intrinsic. Without that information, we end up checking branch weights multiple times in the case if ThinLTO + SampleProfiling, leading to some inaccuracy in how we report MisExpect related diagnostics to users. Since we change the format of MD_prof metadata in a fundamental way, we need to update code handling branch weights in a number of places. We also update the lang ref for branch weights to reflect the change.
2024-06-11Revert "[llvm][IR] Extend BranchWeightMetadata to track provenance of ↵Paul Kirth1-1/+1
weights" (#95060) Reverts llvm/llvm-project#86609 This change causes compile-time regressions for stage2 builds (https://llvm-compile-time-tracker.com/compare.php?from=3254f31a66263ea9647c9547f1531c3123444fcd&to=c5978f1eb5eeca8610b9dfce1fcbf1f473911cd8&stat=instructions:u). It also introduced unintended changes to `.text` which should be addressed before relanding.
2024-06-10[llvm][IR] Extend BranchWeightMetadata to track provenance of weights (#86609)Paul Kirth1-1/+1
This patch implements the changes to LLVM IR discussed in https://discourse.llvm.org/t/rfc-update-branch-weights-metadata-to-allow-tracking-branch-weight-origins/75032 In this patch, we add an optional field to MD_prof metadata nodes for branch weights, which can be used to distinguish weights added from `llvm.expect*` intrinsics from those added via other methods, e.g. from profiles or inserted by the compiler. One of the major motivations, is for use with MisExpect diagnostics, which need to know if branch_weight metadata originates from an llvm.expect intrinsic. Without that information, we end up checking branch weights multiple times in the case if ThinLTO + SampleProfiling, leading to some inaccuracy in how we report MisExpect related diagnostics to users. Since we change the format of MD_prof metadata in a fundamental way, we need to update code handling branch weights in a number of places. We also update the lang ref for branch weights to reflect the change.
2023-11-16Add setBranchWeigths convenience function. NFC (#72446)Matthias Braun1-2/+1
Add `setBranchWeights` convenience function to ProfDataUtils.h and use it where appropriate.
2023-09-11[NFC][RemoveDIs] Prefer iterator-insertion over instructionsJeremy Morse1-2/+2
Continuing the patch series to get rid of debug intrinsics [0], instruction insertion needs to be done with iterators rather than instruction pointers, so that we can communicate information in the iterator class. This patch adds an iterator-taking insertBefore method and converts various call sites to take iterators. These are all sites where such debug-info needs to be preserved so that a stage2 clang can be built identically; it's likely that many more will need to be changed in the future. At this stage, this is just changing the spelling of a few operations, which will eventually become signifiant once the debug-info bearing iterator is used. [0] https://discourse.llvm.org/t/rfc-instruction-api-changes-needed-to-eliminate-debug-intrinsics-from-ir/68939 Differential Revision: https://reviews.llvm.org/D152537
2023-08-17[IR] Add PHINode::removeIncomingValueIf() (NFC)Nikita Popov1-11/+4
Add an API that allows removing multiple incoming phi values based on a predicate callback, as suggested on D157621. This makes sure that the removal is linear time rather than quadratic, and avoids subtleties around iterator invalidation. I have replaced some of the more straightforward users with the new API, though there's a couple more places that should be able to use it. Differential Revision: https://reviews.llvm.org/D158064
2023-08-16Reapply [CHR] Fix up phi nodes with unreachable predecessors (PR64594)Nikita Popov1-0/+14
Relative to the previous attempt, this also adjusts RegionInfo verification to allow unreachable predecessors. ----- If a block in the CHR region has an unreachable predecessor, then there will be no edge from that predecessor to the newly cloned block. However, a phi node entry for it will be left behind. Make sure that these incoming blocks get dropped as well. Fixes https://github.com/llvm/llvm-project/issues/64594. Differential Revision: https://reviews.llvm.org/D157621
2023-08-11Revert "[CHR] Fix up phi nodes with unreachable predecessors (PR64594)"Nikita Popov1-15/+0
This reverts commit cfc9298dc588c626a52193260da28a630e103916. The newly added test fails under EXPENSIVE_CHECKS, revert while I investigate.
2023-08-11[CHR] Fix up phi nodes with unreachable predecessors (PR64594)Nikita Popov1-0/+15
If a block in the CHR region has an unreachable predecessor, then there will be no edge from that predecessor to the newly cloned block. However, a phi node entry for it will be left behind. Make sure that these incoming blocks get dropped as well. Fixes https://github.com/llvm/llvm-project/issues/64594. Differential Revision: https://reviews.llvm.org/D157621
2023-04-17Remove several no longer needed includes. NFCIBjorn Pettersson1-1/+0
Mostly removing includes of InitializePasses.h and Pass.h in passes that no longer has support for the legacy PM.
2023-03-15[llvm] Use *{Map,Set}::contains (NFC)Kazu Hirata1-4/+2
2023-03-07[ControlHeightReduction] Freeze potentially poisonous conditionsKazu Hirata1-5/+2
This patch freezes potentially poisonous conditions in conditional branches so that we do not "move up" conditional branches "br i1 poison". Differential Revision: https://reviews.llvm.org/D145008
2023-02-28Revert "[ControlHeightReduction] Don't combine a "poison" branch"Kazu Hirata1-1/+1
This reverts commit 38a64aab4a3fbaaeb383638ff654247902796556. llvm-clang-x86_64-expensive-checks-debian is failing: https://lab.llvm.org/buildbot/#/builders/16/builds/44249
2023-02-28[ControlHeightReduction] Don't combine a "poison" branchKazu Hirata1-1/+1
Without this patch, the control height reduction pass would combine a "poison" branch with an earlier well-defined branch, turning the earlier branch into a "poison" branch also. This patch fixes the problem by rejecting "poison" conditional branches. Differential Revision: https://reviews.llvm.org/D145008
2023-02-28[CHR] Do not fetch BFI without profile summary (NFCI)Nikita Popov1-2/+2
Do not compute BFI if PGO is not used. This addresses the compile-time regression from https://reviews.llvm.org/D144769.
2023-02-27[Pass][CHR] Move ControlHeightReduction to module optimization pipelineRong Xu1-1/+5
This is a modified version of commit b374423304a8 by Arthur (https://reviews.llvm.org/D143424). Here we invoke to the pass independent of PGOOPT. We now check if the profile is available through the program summary. This ensures CHR is called in distributed ThinLTO BE compilation (where PGOOPT might not be created). Differential Revision: https://reviews.llvm.org/D144769
2023-01-19[llvm][ir] Purge MD_prof custom accessorsChristian Ulmann1-23/+16
This commit purges direct accesses to MD_prof metadata and replaces them with the accessors provided from the utility file wherever possible. This commit can be seen as the first step towards switching the branch weights to 64 bits. See post here: https://discourse.llvm.org/t/extend-md-prof-branch-weights-metadata-from-32-to-64-bits/67492 Reviewed By: davidxl, paulkirth Differential Revision: https://reviews.llvm.org/D141393
2022-12-15[NFC] Rename Instruction::insertAt() to Instruction::insertInto(), to be ↵Vasileios Porpodas1-1/+1
consistent with BasicBlock::insertInto() Differential Revision: https://reviews.llvm.org/D140085
2022-12-14[NFC] Cleanup: Replace Function::getBasicBlockList().splice() with ↵Vasileios Porpodas1-3/+2
Function::splice() This is part of a series of patches that aim at making Function::getBasicBlockList() private. Differential Revision: https://reviews.llvm.org/D139984
2022-12-12[NFC] Cleanup: Replaces BB->getInstList().insert() with I->insertAt().Vasileios Porpodas1-1/+1
This is part of a series of cleanup patches towards making BasicBlock::getInstList() private. Differential Revision: https://reviews.llvm.org/D138877
2022-12-06[YAML] Convert Optional to std::optionalKrzysztof Parzyszek1-1/+2
2022-12-01[NFC] Fix dbgs() statement in ControlHeightReductionMircea Trofin1-1/+2
2022-11-22[CHR] Add a threshold for the code duplicationRong Xu1-15/+51
ControlHeightReduction (CHR) clones the code region to reduce the branches in the hot code path. The number of clones is linear to the depth of the region. Currently it does not have control over the code size increase. We are seeing one ~9000 BB functions get expanded to ~250000 BBs, an 25x increase. This creates a big compile time issue for the downstream optimizations. This patch adds a cap for number of clones for one region. Differential Revision: https://reviews.llvm.org/D138333
2022-10-17ControlHeightReduction: Remove assert check in shouldApplyMatthias Braun1-1/+0
Remove assertion checking for non-empty `ProfileSummaryInfo`. Differential Revision: https://reviews.llvm.org/D133706
2022-09-19Analysis: Add AssumptionCache to isSafeToSpeculativelyExecuteMatt Arsenault1-1/+1
Does not update any of the uses.
2022-08-27[Transform] Use range-based for loops (NFC)Kazu Hirata1-2/+2
2022-08-07[llvm] Fix comment typos (NFC)Kazu Hirata1-1/+1
2022-07-16[LegacyPM] Remove ControlHeightReductionLegacyPassFangrui Song1-53/+0
This pass tries to reduce the number of conditional branches in the hot path based on profile. It's mostly a no-op after legacy PGO passes are moved.
2022-06-18[llvm] Use value_or instead of getValueOr (NFC)Kazu Hirata1-1/+1
2022-06-12[Transforms] Use default member initialization (NFC)Kazu Hirata1-4/+4
Identified with modernize-use-default-member-init.
2022-05-26[ControlHeightReduction] Use logical andNikita Popov1-1/+2
Use logical instead of bitwise and to combine conditions, to avoid propagating poison from a later condition if an earlier one is already false. This avoids introducing branch on poison. Differential Revision: https://reviews.llvm.org/D125898
2022-05-16[ControlHeightReduction] Freeze condition when converting select to branchNikita Popov1-0/+7
While select conditions can be poison, branch on poison is immediate UB. As such, we need to freeze the condition when converting a select into a branch. Differential Revision: https://reviews.llvm.org/D125398
2022-05-13[ControlHeightReduction] Simplify addToMergedCondition() (NFC)Nikita Popov1-18/+9
2022-04-27[CHR] Skip region containing llvm.coro.idWei Wang1-1/+14
When a block containing llvm.coro.id is cloned during CHR, it inserts an invalid PHI node with token type to the beginning of the block containing llvm.coro.begin. To avoid such case, we exclude regions with llvm.coro.id. Reviewed By: ChuanqiXu Differential Revision: https://reviews.llvm.org/D124418
2022-03-20[Transform] Apply clang-tidy fixes for readability-redundant-smartptr-get (NFC)Kazu Hirata1-1/+1
2022-02-13[Transforms] Use default member initialization in CHRStats (NFC)Kazu Hirata1-8/+8
2021-10-31[Transforms] Use {DenseSet,SetVector,SmallPtrSet}::contains (NFC)Kazu Hirata1-3/+3
2021-10-30[clang, llvm] Use Optional::getValueOr (NFC)Kazu Hirata1-1/+1
2021-06-12[CHR] Don't run ControlHeightReduction if any BB has address takenXun Li1-0/+5
This patch is to address https://bugs.llvm.org/show_bug.cgi?id=50610. In computed goto pattern, there are usually a list of basic blocks that are all targets of indirectbr instruction, and each basic block also has address taken and stored in a variable. CHR pass could potentially clone these basic blocks, which would generate a cloned version of the indirectbr and clonved version of all basic blocks in the list. However these basic blocks will not have their addresses taken and stored anywhere. So latter SimplifyCFG pass will simply remove all tehse cloned basic blocks, resulting in incorrect code. To fix this, when searching for scopes, we skip scopes that contains BBs with addresses taken. Added a few test cases. Reviewed By: aeubanks, wenlei, hoy Differential Revision: https://reviews.llvm.org/D103867
2021-05-18[NewPM] Don't mark AA analyses as preservedArthur Eubanks1-3/+2
Currently all AA analyses marked as preserved are stateless, not taking into account their dependent analyses. So there's no need to mark them as preserved, they won't be invalidated unless their analyses are. SCEVAAResults was the one exception to this, it was treated like a typical analysis result. Make it like the others and don't invalidate unless SCEV is invalidated. Reviewed By: asbirlea Differential Revision: https://reviews.llvm.org/D102032
generated by cgit v1.1 at 2025-10-01 17:33:27 +0000