We introduce a codegen optimization pass which splits functions into hot and cold
parts. This pass leverages the basic block sections feature recently
introduced in LLVM from the Propeller project. The pass targets
functions with profile coverage, identifies cold blocks and moves them
to a separate section. The linker groups all cold blocks across
functions together, decreasing fragmentation and improving icache and
itlb utilization.

We evaluated the Machine Function Splitter pass on clang bootstrap and
SPECInt 2017.

For clang bootstrap we observe a mean 2.33% runtime improvement with a
~32% reduction in itlb and stlb misses. Additionally, L1 icache misses
reduced by 9.5% while L2 instruction misses reduced by 20%.

For SPECInt we report the change in IntRate the C/C++
benchmarks. All benchmarks apart from mcf and x264 improve, on average
by 0.6% with the max for deepsjeng at 1.6%.

Benchmark		% Change
500.perlbench_r		 0.78
502.gcc_r		 0.82
505.mcf_r		-0.30
520.omnetpp_r		 0.18
523.xalancbmk_r		 0.37
525.x264_r		-0.46
531.deepsjeng_r		 1.61
541.leela_r		 0.83
557.xz_r		 0.15

Differential Revision: https://reviews.llvm.org/D85368

Enable MVE gather/scatters by default, which requires some
minor adaptations in some tests.

Differential revision: https://reviews.llvm.org/D86776

When an error is associated with a symbol, it was marked with a flag
from Symbol::Flag. The problem with that is that you need a mutable
symbol to do that. Instead, store the set of error symbols in the
SemanticsContext. This allows for some const_casts to be eliminated.

Also, improve the internal error that occurs if SetError is called
but no fatal error has been reported.

Differential Revision: https://reviews.llvm.org/D86740

This implements the part of P0619R4 related to the default allocator.
This is incredibly important, since otherwise there is an ABI break
between C++17 and C++20 w.r.t. the default allocator's size_type on
platforms where std::size_t is not the same as std::make_unsigned<std::ptrdiff_t>.

These arm_mve_vldr_gather_offset_predicated and
arm_mve_vstr_scatter_offset_predicated have some extra parameters
meaning the predicate is at a later operand. If a loop contains _only_
those masked instructions, we would miss transforming the active lane
mask.

Differential Revision: https://reviews.llvm.org/D86791

This patch implements the builtins for Vector Load with Zero and Signed Extend Builtins (lxvr_x for b, h, w, d), and adds the appropriate test cases for these builtins. The builtins utilize the vector load instructions itnroduced with ISA 3.1.

Differential Revision: 	https://reviews.llvm.org/D82502#inline-797941

There is a subtle problem with new statepoint lowering scheme
when base and pointers are the same (see PR46917 for more context):

%1 = STATEPOINT ... %0, %0(tied-def 0)...

if, for some reason, register allocator desides to put two instances
of %0 into two different objects (registers or spill slots), we may
end up with

$reg3 = STATEPOINT ... $reg2, $reg1(tied-def 0)...

and nothing will prevent later passes to sink uses of $reg2 below
statepoint, which is incorrect.

As a short term solution, always put base pointers on stack during
lowering.
A longer term solution may be to rework MIR statepoint format to
avoid GC pointer duplication in statepoint argument list.

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D86712

The test command in `private_mapping.c` was set to expect failure by mistake. It is fixed in this patch.

Reviewed By: ABataev

Differential Revision: https://reviews.llvm.org/D86758

With gcc 6.3.0, I hit the following compilation error:
  ../lib/CodeGen/GlobalISel/Combiner.cpp: In member function
      ‘bool llvm::Combiner::combineMachineInstrs(llvm::MachineFunction&,
       llvm::GISelCSEInfo*)’:
  ../lib/CodeGen/GlobalISel/Combiner.cpp:156:54: error: suggest parentheses
       around ‘&&’ within ‘||’ [-Werror=parentheses]
     assert(!CSEInfo || !errorToBool(CSEInfo->verify()) &&
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
                            "CSEInfo is not consistent. Likely missing calls to "
                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                            "observer on mutations");

Fix the code as suggested by the compiler.

This is the follow up patch for https://reviews.llvm.org/D86183 as we miss to delete the node if NegX == NegY, which has use after we create the node.
```
    if (NegX && (CostX <= CostY)) {
      Cost = std::min(CostX, CostZ);
      RemoveDeadNode(NegY);
      return DAG.getNode(Opcode, DL, VT, NegX, Y, NegZ, Flags);  #<-- NegY is used here if NegY == NegX.
    }
```

Reviewed By: spatel

Differential Revision: https://reviews.llvm.org/D86689

This relands D85743 with a fix for test
CodeGen/attr-arm-sve-vector-bits-call.c that disables the new pass
manager with '-fno-experimental-new-pass-manager'. Test was failing due
to IR differences with the new pass manager which broke the Fuchsia
builder [1]. Reverted in 2e7041fd.

[1] http://lab.llvm.org:8011/builders/fuchsia-x86_64-linux/builds/10375

Original summary:

This patch implements codegen for the 'arm_sve_vector_bits' type
attribute, defined by the Arm C Language Extensions (ACLE) for SVE [1].
The purpose of this attribute is to define vector-length-specific (VLS)
versions of existing vector-length-agnostic (VLA) types.

VLSTs are represented as VectorType in the AST and fixed-length vectors
in the IR everywhere except in function args/return. Implemented in this
patch is codegen support for the following:

  * Implicit casting between VLA <-> VLS types.
  * Coercion of VLS types in function args/return.
  * Mangling of VLS types.

Casting is handled by the CK_BitCast operation, which has been extended
to support the two new vector kinds for fixed-length SVE predicate and
data vectors, where the cast is implemented through memory rather than a
bitcast which is unsupported. Implementing this as a normal bitcast
would require relaxing checks in LLVM to allow bitcasting between
scalable and fixed types. Another option was adding target-specific
intrinsics, although codegen support would need to be added for these
intrinsics. Given this, casting through memory seemed like the best
approach as it's supported today and existing optimisations may remove
unnecessary loads/stores, although there is room for improvement here.

Coercion of VLSTs in function args/return from fixed to scalable is
implemented through the AArch64 ABI in TargetInfo.

The VLA and VLS types are defined by the ACLE to map to the same
machine-level SVE vectors. VLS types are mangled in the same way as:

  __SVE_VLS<typename, unsigned>

where the first argument is the underlying variable-length type and the
second argument is the SVE vector length in bits. For example:

  #if __ARM_FEATURE_SVE_BITS==512
  // Mangled as 9__SVE_VLSIu11__SVInt32_tLj512EE
  typedef svint32_t vec __attribute__((arm_sve_vector_bits(512)));
  // Mangled as 9__SVE_VLSIu10__SVBool_tLj512EE
  typedef svbool_t pred __attribute__((arm_sve_vector_bits(512)));
  #endif

The latest ACLE specification (00bet5) does not contain details of this
mangling scheme, it will be specified in the next revision.  The
mangling scheme is otherwise defined in the appendices to the Procedure
Call Standard for the Arm Architecture, see [2] for more information.

[1] https://developer.arm.com/documentation/100987/latest
[2] https://github.com/ARM-software/abi-aa/blob/master/aapcs64/aapcs64.rst#appendix-c-mangling

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D85743

In this patch, a pc-rel based long branch thunk is added for the local
call protocol that caller and callee does not use TOC.

Reviewed By: sfertile, nemanjai

Differential Revision: https://reviews.llvm.org/D86706

MLIR build failed after ElementCount refactoring - updated code to
call isScalable() and getKnownMinValue().

This patch changes ElementCount so that the Min and Scalable
members are now private and can only be accessed via the get
functions getKnownMinValue() and isScalable(). In addition I've
added some other member functions for more commonly used operations.
Hopefully this makes the class more useful and will reduce the
need for calling getKnownMinValue().

Differential Revision: https://reviews.llvm.org/D86065

The abbrev codes in a new abbrev table should start from 1 (by default),
rather than inherit the value from the code in the previous table.

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D86545

Original D81646 had check for tied regs in foldPatchpoint().
Due to unfortunate miscommunication with review comments and
adressing some comments post commit, it turned into assertion.

We had an offline talk and agreed that with current implementation
this path is possible, so I'm changing it back to check.

Note that this is workaround until ussues described in PR46917 are
resolved.

Remove the code that tried to look for reduction patterns, since the
vectorizer and isel can now produce predicated arithmetic instructios
within the loop body. This has required some reorganisation and fixes
around live-out and predication checks, as well as looking for cases
where an input/output is initialised to zero.

Differential Revision: https://reviews.llvm.org/D86613

The refactored functions were
* `isReponsibleForCreatingDeclaration`
* `getQualifiedNameStart`

Differential Revision: https://reviews.llvm.org/D86719

Previously in addTypeForNeon, we would set the operations for bfloat vectors
like other generic types. But as bfloat is a storage-only type a number of
operations shouldn't be set. This patch fixes that.

Reviewed By: dmgreen

Differential Revision: https://reviews.llvm.org/D85101

When guessing whether a closing paren is then end of a cast expression also
skip over pointer qualifiers while looking for TT_PointerOrReference.
This prevents some address-of and dereference operators from being parsed
as a binary operator.

Before:
x = (foo *const) * v;
x = (foo *const volatile restrict __attribute__((foo)) _Nonnull _Null_unspecified _Nonnull) & v;

After:
x = (foo *const)*v;
x = (foo *const volatile restrict __attribute__((foo)) _Nonnull _Null_unspecified _Nonnull)&v;

Reviewed By: MyDeveloperDay

Differential Revision: https://reviews.llvm.org/D86716

Before:
void f() { MACRO(A * _Nonnull a); }
void f() { MACRO(A * _Nullable a); }
void f() { MACRO(A * _Null_unspecified a); }

After:
void f() { MACRO(A *_Nonnull a); }
void f() { MACRO(A *_Nullable a); }
void f() { MACRO(A *_Null_unspecified a); }

Reviewed By: JakeMerdichAMD

Differential Revision: https://reviews.llvm.org/D86713

Before: void f() { MACRO(A * __attribute((foo)) a); }
After:  void f() { MACRO(A *__attribute((foo)) a); }

Also check that the __attribute__ alias is handled.

Reviewed By: MyDeveloperDay

Differential Revision: https://reviews.llvm.org/D86711

Before: void f() { MACRO(A * restrict a); }
After:  void f() { MACRO(A *restrict a); }

Also check that the __restrict and __restrict__ aliases are handled.

Reviewed By: JakeMerdichAMD

Differential Revision: https://reviews.llvm.org/D86710

Before: void f() { MACRO(A * volatile a); }
After:  void f() { MACRO(A *volatile a); }

Also check that the __volatile and __volatile__ aliases are handled.

Reviewed By: JakeMerdichAMD

Differential Revision: https://reviews.llvm.org/D86708

This changes getDomMemoryDef to check if a Current is a valid
candidate for elimination before checking for reads. Before the change,
we were spending a lot of compile-time in checking for read accesses for
Current that might not even be removable.

This patch flips the logic, so we skip Current if they cannot be
removed before checking all their uses. This is much more efficient in
practice.

It also adds a more aggressive limit for checking partially overlapping
stores. The main problem with overlapping stores is that we do not know
if they will lead to elimination until seeing all of them. This patch
limits adds a new limit for overlapping store candidates, which keeps
the number of modified overlapping stores roughly the same.

This is another substantial compile-time improvement (while also
increasing the number of stores eliminated). Geomean -O3 -0.67%,
ReleaseThinLTO -0.97%.

http://llvm-compile-time-tracker.com/compare.php?from=0a929b6978a068af8ddb02d0d4714a2843dd8ba9&to=2e630629b43f64b60b282e90f0d96082fde2dacc&stat=instructions

Reviewed By: asbirlea

Differential Revision: https://reviews.llvm.org/D86487

This patch is mostly about removing the "Category" enum, which was
very useful when the Type enum contained a large number of types, but
now the two are completely identical.

It also removes some other artifacts like unused typedefs and macros.

using early exits. NFC.

`{@code xxxxx}` triggers a Doxygen bug. The bug may be matching the
close brace with the open brace of the namespace
declaration (`namespace clang {` or `namespace ento {`).

Differential Revision: https://reviews.llvm.org/D85105

Tests on Solaris/sparcv9 currently show about 250 failures when building
with gcc, most of them like the following:

  FAIL: LLVM-Unit :: Support/./SupportTests/TaskQueueTest.UnOrderedFutures (4269 of 67884)
  ******************** TEST 'LLVM-Unit :: Support/./SupportTests/TaskQueueTest.UnOrderedFutures' FAILED ********************
  Note: Google Test filter = TaskQueueTest.UnOrderedFutures
  [==========] Running 1 test from 1 test case.
  [----------] Global test environment set-up.
  [----------] 1 test from TaskQueueTest
  [ RUN      ] TaskQueueTest.UnOrderedFutures
  0  SupportTests        0x0000000100753b20 llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 32
  1  SupportTests        0x0000000100752974 llvm::sys::RunSignalHandlers() + 68
  2  SupportTests        0x0000000100752b18 SignalHandler(int) + 372
  3  libc.so.1           0xffffffff7eedc800 __sighndlr + 12
  4  libc.so.1           0xffffffff7eecf23c call_user_handler + 852
  5  libc.so.1           0xffffffff7eecf594 sigacthandler + 84
  6  SupportTests        0x00000001006f8cb8 std::thread::_State_impl<std::thread::_Invoker<std::tuple<llvm::ThreadPool::ThreadPool(llvm::ThreadPoolStrategy)::'lambda'()> > >::_M_run() + 512
  7  libstdc++.so.6.0.28 0xfffffffc628117cc execute_native_thread_routine + 16
  8  libc.so.1           0xffffffff7eedc6a0 _lwp_start + 0

Since it's effectively impossible to debug such a `SEGV` in a `Release`
build, I tried a `Debug` build instead, only to find that the failures had
gone away.

Further investigation revealed that most of the issue centers around
`llvm/lib/Support/ThreadPool.cpp`.  That file is built with `-O3 -fPIC` in
a `Release` build.  The failure vanishes if

- compiling without `-fPIC`
- compiling with `-O -fPIC`
- linking with GNU `ld` instead of Solaris `ld`

It has meanwhile been determined that `gcc` doesn't correctly heed some TLS
code sequences.  To make things worse, Solaris `ld` doesn't properly
validate its assumptions against the input, generating wrong code.

`gld` like `gcc` is more liberal here and correctly deals with the code it
gets fed from `gcc`.

There's PR target/96607: GCC feeds SPARC/Solaris linker with unrecognized
TLS sequences <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96607> now.

An attempt to build with `-DLLVM_ENABLE_PIC=Off` initially failed since
neither `libRemarks.so` (D85626 <https://reviews.llvm.org/D85626>) nor
`LLVMPolly.so` (D85627 <https://reviews.llvm.org/D85627>) heed that option.
Even with that fixed, a few codegen failures remain.

Next I tried to build just `ThreadPool.cpp` with `-O -fPIC`.  While that
fixed the vast majority of the failures, 16 `LLVM :: CodeGen/X86` failures
remained.

Given that that solution was both incomplete and fragile, I went for
building the whole tree with `-O -fPIC` for `Release` and `RelWithDebInfo`
builds.

As detailed in Bug 47304, 2-stage builds also show large numbers of
failures when building with `-O3` or `-O2`, which are likewise worked
around by building with `-O` until they are sufficiently analyzed and
fixed.

This way, all failures relative to a `Debug` build go away.

Tested on `sparcv9-sun-solaris2.11`.

Differential Revision: https://reviews.llvm.org/D85630

This patch adds support for memcmp in MemoryLocation::getForArgument.
memcmp reads from the first 2 arguments up to the number of bytes of the
third argument.

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D86725