Summary:

After https://reviews.llvm.org/D143436 response files stopped working
with CDB interpolation. It has happened because interpolation removes
all unknown flags and extra input files. Response file is treated as an
extra input because it is not a flag. Moreover inference needs full
command line for driver mode and file type detection so all response
files have to be expanded for correct inference.

This patch partially reverts D143436 and add additional response file
expansion in OverlayCDB for CDBs pushed via LSP.

Test Plan: check-clangd

Tasks: https://github.com/llvm/llvm-project/issues/69690

The `test-lower-to-nvvm` pipeline serves as the common and proper
pipeline for nvvm+host compilation, and it's used across our CUDA
integration tests.

This PR updates the `test-lower-to-nvvm` pipeline to `gpu-lower-to-nvvm`
and moves it within `InitAllPasses.h`. The aim is to call it from
Python, also having a standardize compilation process for nvvm.

There were two issues with the previous computation:
* it never looked at dimensions past the second one
* the definition was recursive, making each dimension have an extra
`elementSize` power

It is currently disabled by default. It will need experiments on a real
HW to tune and decide on the profitability.

---------

Co-authored-by: Stanislav Mekhanoshin <Stanislav.Mekhanoshin@amd.com>

In fact, this pass need `llc` to test. `TargetMachine` seems redundant,
because before adding this pass `CodeGenPassBuilder` already checks it:

https://github.com/llvm/llvm-project/blob/ed4194bb8dbca5222628c2cddbc032fff57193b5/llvm/include/llvm/CodeGen/CodeGenPassBuilder.h#L590-L592

It can be more efficiently accessed from TSFlags and won't require extra
storage.

NFC because it wasn't exported to the cpp PseudoTable and isn't used in
tree yet.

The RISC-V vector crypto extensions have been ratified. This patch
updates the Clang and LLVM support for these extensions to be
non-experimental, while leaving the C intrinsics as experimental since
the C intrinsics are not yet standardized.

Co-authored-by: Brandon Wu <brandon.wu@sifive.com>

According to spirv spec, OpConstantNull's result type can't be image
type. So we can't generate zeroinitializer for spirv.Image.

`FoldInsertPadIntoFill` used to generate an invalid
`tensor.insert_slice` op:
```
error: expected type to be 'tensor<?x?x?xf32>' or a rank-reduced version. (size mismatch)
```

This commit fixes tests such as
`mlir/test/Dialect/Linalg/canonicalize.mlir` when verifying the IR after
each pattern application (#74270).

Linalg op fusion (`Linalg/Transforms/Fusion.cpp`) used to generate
invalid fused producer ops:
```
error: 'linalg.conv_2d_nhwc_hwcf' op expected type of operand #2 ('tensor<1x8x16x4xf32>') to match type of corresponding result ('tensor<?x?x?x?xf32>')
note: see current operation:
%24 = "linalg.conv_2d_nhwc_hwcf"(%21, %22, %23) <{dilations = dense<1> : tensor<2xi64>, operandSegmentSizes = array<i32: 2, 1>, strides = dense<2> : tensor<2xi64>}> ({
^bb0(%arg9: f32, %arg10: f32, %arg11: f32):
  %28 = "arith.mulf"(%arg9, %arg10) <{fastmath = #arith.fastmath<none>}> : (f32, f32) -> f32
  %29 = "arith.addf"(%arg11, %28) <{fastmath = #arith.fastmath<none>}> : (f32, f32) -> f32
  "linalg.yield"(%29) : (f32) -> ()
}) {linalg.memoized_indexing_maps = [affine_map<(d0, d1, d2, d3, d4, d5, d6) -> (d0, d1 * 2 + d4, d2 * 2 + d5, d6)>, affine_map<(d0, d1, d2, d3, d4, d5, d6) -> (d4, d5, d6, d3)>, affine_map<(d0, d1, d2, d3, d4, d5, d6) -> (d0, d1, d2, d3)>]} : (tensor<1x?x?x3xf32>, tensor<3x3x3x4xf32>, tensor<1x8x16x4xf32>) -> tensor<?x?x?x?xf32>
```

This is a problem because the input IR to greedy pattern rewriter during
`-test-linalg-greedy-fusion` is invalid. This commit fixes tests such as
`mlir/test/Dialect/Linalg/tile-and-fuse-tensors.mlir` when verifying the
IR after each pattern application (#74270).

Move the ownership of the symbolizer into symbolizeAndFilterStackFrames
so that it is freed on exit, when we are done with it, to reduce peak
memory in the reader. This reduces about 9G from the peak for one large
profile.

Revert "Reland "[PGO][GlobalValue][LTO]In GlobalValues::getGlobalIdentifier, use semicolon as delimiter for local-linkage varibles. "" (#75888)

Reverts llvm/llvm-project#75860
- Mangled name mismatch on Windows
(https://lab.llvm.org/buildbot/#/builders/127/builds/59907/steps/8/logs/stdio)

Resolves buildbot failure reported on windows bot

As we've now finished parsing the constructs, we're moving onto
implementing 'clause' parsing. While some are complicated and require
their own patch, the handful added here are simple to parse (that is,
    they are a single identifier).

This patch adds the infrastructure to parse these and a clause-list in
its entirety. This adds some complication to how we are diagnosing
parsing errors elsewhere, so a few changes were made to better recover
from errors.

PR #75576 and #75735 update some implies in
llvm/lib/Support/RISCVISAInfo.cpp, but both of them miss the subtarget
feature part.
This patch still preserve predicate HasStdExtZfhOrZfhmin and
HasStdExtZhinxOrZhinxmin, since they could make error message more
readable. ( Users might not know that zfh implies zfhmin.)

Reland "[PGO][GlobalValue][LTO]In GlobalValues::getGlobalIdentifier, use semicolon as delimiter for local-linkage varibles. " (#75860)

Fixed build-bot failures caught by post-submit tests
1) Add the list of command line tools needed by new compiler-rt test into dependency.
2) Use `starts_with` to replace deprecated `startswith`.

**Original commit message**
Commit fe051934 (phab D156569), IRPGO names uses format
`[<filepath>;]<linkage-name>` while prior format is
`[<filepath>:<mangled-name>`. The format change would break the use case
demonstrated in (updated)
`llvm/test/Transforms/PGOProfile/thinlto_indirect_call_promotion.ll` and
`compiler-rt/test/profile/instrprof-thinlto-indirect-call-promotion.cpp`

This patch changes `GlobalValues::getGlobalIdentifer` to use the
semicolon.

To elaborate on the scenario how things break without this PR
1. IRPGO raw profiles stores (compressed) IRPGO names of functions in
one section, and per-function profile data in another section. The
[NameRef](https://github.com/llvm/llvm-project/blob/fc715e4cd942612a091097339841733757b53824/compiler-rt/include/profile/InstrProfData.inc#L72)
field in per-function profile data is the MD5 hash of IRPGO names.
2. When raw profiles are converted to indexed format profiles, the
profiled address is
[mapped](https://github.com/llvm/llvm-project/blob/fc715e4cd942612a091097339841733757b53824/llvm/lib/ProfileData/InstrProf.cpp#L876-L885)
to the MD5 hash of the callee.
3. In `pgo-instr-use` thin-lto prelink pipeline, MD5 hash of IRPGO names
will be
[annotated](https://github.com/llvm/llvm-project/blob/fc715e4cd942612a091097339841733757b53824/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp#L1707)
as value profiles, and used to import indirect-call-prom candidates. If
the annotated MD5 hash is computed from the new format while import uses
the prior format, the callee cannot be imported.

*
`compiler-rt/test/profile/instrprof-thinlto-indirect-call-promotion.cpp`
is added to have an end-to-end test.
* `llvm/test/Transforms/PGOProfile/thinlto_indirect_call_promotion.ll`
is updated to have better test coverage from another aspect (as runtime
tests are more sensitive to the environment and may be skipped by some
contributors)

https://reviews.llvm.org/D122407 added a static link against
libclang_rt.asan_static whenever ASAN is linked, but ignored the
-fno-sanitize-link-runtime flag. Every other conditional in
collectSanitizerRuntimes is already checking for SanArgs.linkRuntimes(),
move it to the top of the function so that newly added conditionals
don't need to remember to check it.

Move the implementation of raw_socket_stream from raw_ostream.h/cpp to
raw_socket_stream.h/cpp as requested in #73603.

If a sanitizer suppression file can not be parsed, add the supported
suppression types to the error message.

See https://github.com/llvm/llvm-project/issues/72060

.

---------

Co-authored-by: Vitaly Buka <vitalybuka@gmail.com>

> Add support for reading binary Mach-o dynamic libraries. It uses
libObject APIs for extracting information relevant to TAPI and tbd
files. This includes but is not limited to load commands encode data
like install names, current/compat versions, and symbols.

This originally broke because DylibReader uses Object and Object depends
on TextAPI. Breaking this up in a nested library prevents this cycle.

This makes these variable names consisent with the capitalization
used in RISCVFeatures.td.

    The legalFor check did not work as expected. Use legalIf instead. Also
    do some simplification by removing IdxZeroTy and IdxOneTy.

There are many issues that popped up with the counted_by feature. The
patch #73730 has grown too large and approval is blocking Linux testing.

Includes reverts of:
commit 769bc11f ("[Clang] Implement the 'counted_by' attribute
(#68750)")
commit bc09ec69 ("[CodeGen] Revamp counted_by calculations
(#70606)")
commit 1a09cfb2 ("[Clang] counted_by attr can apply only to C99
flexible array members (#72347)")
commit a76adfb9 ("[NFC][Clang] Refactor code to calculate flexible
array member size (#72790)")
commit d8447c78 ("[Clang] Correct handling of negative and
out-of-bounds indices (#71877)")
Partial commit b31cd07d ("[Clang] Regenerate test checks (NFC)")

Closes #73168
Closes #75173

Many systems (glibc<2.30, Bionic before 2019, musl, macOS, etc) do not have the function.

The number of vector elements considered 'small' enough to extract is
parameterized.                                                   
                                                                 
This is to avoid going into specialized reduction lowering when a
single/couple of arith ops can do. Targets without dedicated reduction  
intrinsics can use that as an emulation path too.                  
                                                                   
Depends on https://github.com/llvm/llvm-project/pull/75846.

Propagate fast math flags.
Distinguish `minf`/`maxf` and `minimumf`/`maximumf`.

Required for future patterns in
https://github.com/llvm/llvm-project/pull/75727.

As reported in [1], it looks like the Windows headers are picky about
the order in which they are included, and the clang-format change broke
the build by reordering the headers.

[1]: https://github.com/llvm/llvm-project/pull/74334#issuecomment-1861719927

requirement, NFC.

This change does not affect functionality, just fixes the assertions in
some standard c++ library implementations.

This will result in larger atomic operations getting expanded to
`__atomic_*` libcalls via AtomicExpandPass, which matches what Clang
already does in the frontend.

While AMDGPU currently disables the use of all libcalls, I've changed it
to instead disable all of them _except_ the atomic ones. Those are
already be emitted by the Clang frontend, and enabling them in the
backend allows the same behavior there.

A displacement is an 8-, 16-, or 32-bit value.
LLVM integrated assembler silently encodes an out-of-range displacement.

GNU assembler checks the displacement and may report a warning or error
(error is for 64-bit addressing, done as part of
https://sourceware.org/PR10636).
```
movq 0x80000000(%rip), %rax
Error: 0x80000000 out of range of signed 32bit displacement

movq -0x080000001(%rax), %rax
Error: 0xffffffff7fffffff out of range of signed 32bit displacement

movl 0x100000001(%eax), %eax
Warning: 0x100000001 shortened to 0x1
```

For 32-bit addressing, GNU assembler gives no diagnostic when the
displacement is within `[-2**32,2**32)`. 16-bit addressing is similar.
```
movl 0xffffffff(%eax), %eax  # no diagnostic
movl -0xffffffff(%eax), %eax # no diagnostic
```

Supporting a larger range is probably because wraparound using a large
constant is more reasonable. E.g. Linux kernel arch/x86/kernel/head_32.S
has `leal -__PAGE_OFFSET(%ecx),%esp` where `__PAGE_OFFSET` is
0xc0000000.

This patch implements a similar behavior.

Fix #57086: when ASAN_SHADOW_OFFSET_CONST >= 0x80000000 (FreeBSD,
NetBSD, etc), `movsbl ASAN_SHADOW_OFFSET_CONST(%r10),%r10d` has an
invalid displacement (not representable as a signed 32-bit integer),
which will be diagnosed by GNU assembler.

```
% cat a.s
movsbl 0x80000000(%r10),%r10d
% as a.s
a.s: Assembler messages:
a.s:1: Error: 0x80000000 out of range of signed 32bit displacement
% clang -c a.s
```

The integrated assembler after #75747 will diagnose the invalid
displacement as well.
```
% clang -c a.s
a.s:1:19: error: displacement 2147483648 is not within [-2147483648, 2147483647]
movsbl 0x80000000(%r10),%r10d
                  ^
```

If ASAN_SHADOW_OFFSET_CONST cannot be encoded as a displacement, switch
to `movabsq+movsbl`.

The function `pthread_mutex_clocklock` is not supported by TSAN yet,
which is mentioned by[
llvm/llvm-project/issues/62623](https://github.com/llvm/llvm-project/issues/62623#issue-1701600538).
This patch is to handle this function.