1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
|
===================
DTLTO
===================
.. contents::
:local:
:depth: 2
.. toctree::
:maxdepth: 1
Distributed ThinLTO (DTLTO)
===========================
Distributed ThinLTO (DTLTO) enables the distribution of backend ThinLTO
compilations via external distribution systems, such as Incredibuild, during the
link step.
DTLTO extends the existing ThinLTO distribution support which uses separate
*thin-link*, *backend compilation*, and *link* steps. This method is documented
here:
https://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html
Using the *separate thin-link* approach requires a build system capable of
handling the dynamic dependencies specified in the individual summary index
files, such as Bazel. DTLTO removes this requirement, allowing it to be used
with any build process that supports in-process ThinLTO.
The following commands show the steps used for the *separate thin-link*
approach for a basic example:
.. code-block:: console
1. clang -flto=thin -O2 t1.c t2.c -c
2. clang -flto=thin -O2 t1.o t2.o -fuse-ld=lld -Wl,--thinlto-index-only
3. clang -O2 -o t1.native.o t1.o -c -fthinlto-index=t1.o.thinlto.bc
4. clang -O2 -o t2.native.o t2.o -c -fthinlto-index=t2.o.thinlto.bc
5. clang t1.native.o t2.native.o -o a.out -fuse-ld=lld
With DTLTO, steps 2-5 are performed internally as part of the link step. The
equivalent DTLTO commands for the above are:
.. code-block:: console
clang -flto=thin -O2 t1.c t2.c -c
clang -flto=thin -O2 t1.o t2.o -fuse-ld=lld -fthinlto-distributor=<distributor_process>
For DTLTO, LLD prepares the following for each ThinLTO backend compilation job:
- An individual index file and a list of input and output files (corresponds to
step 2 above).
- A Clang command line to perform the ThinLTO backend compilations.
This information is supplied, via a JSON file, to ``distributor_process``, which
executes the backend compilations using a distribution system (corresponds to
steps 3 and 4 above). Upon completion, LLD integrates the compiled native object
files into the link process and completes the link (corresponds to step 5
above).
This design keeps the details of distribution systems out of the LLVM source
code.
An example distributor that performs all work on the local system is included in
the LLVM source tree. To run an example with that distributor, a command line
such as the following can be used:
.. code-block:: console
clang -flto=thin -fuse-ld=lld -O2 t1.o t2.o -fthinlto-distributor=$(which python3) \
-Xthinlto-distributor=$LLVMSRC/llvm/utils/dtlto/local.py
Distributors
------------
Distributors are programs responsible for:
1. Consuming the JSON backend compilations job description file.
2. Translating job descriptions into requests for the distribution system.
3. Blocking execution until all backend compilations are complete.
Distributors must return a non-zero exit code on failure. They can be
implemented as platform native executables or in a scripting language, such as
Python.
Clang and LLD provide options to specify a distributor program for managing
backend compilations. Distributor options and backend compilation options can
also be specified. Such options are transparently forwarded.
The backend compilations are currently performed by invoking Clang. For further
details, refer to:
* Clang documentation: https://clang.llvm.org/docs/ThinLTO.html
* LLD documentation: https://lld.llvm.org/DTLTO.html
When invoked with a distributor, LLD generates a JSON file describing the
backend compilation jobs and executes the distributor, passing it this file.
JSON Schema
-----------
The JSON format is explained by reference to the following example, which
describes the backend compilation of the modules ``t1.o`` and ``t2.o``:
.. code-block:: json
{
"common": {
"linker_output": "dtlto.elf",
"args": ["/usr/bin/clang", "-O2", "-c", "-fprofile-sample-use=my.prof"],
"inputs": ["my.prof"]
},
"jobs": [
{
"args": ["t1.o", "-fthinlto-index=t1.o.thinlto.bc", "-o", "t1.native.o", "-fproc-stat-report=t1.stats.txt"],
"inputs": ["t1.o", "t1.o.thinlto.bc"],
"outputs": ["t1.native.o", "t1.stats.txt"]
},
{
"args": ["t2.o", "-fthinlto-index=t2.o.thinlto.bc", "-o", "t2.native.o", "-fproc-stat-report=t2.stats.txt"],
"inputs": ["t2.o", "t2.o.thinlto.bc"],
"outputs": ["t2.native.o", "t2.stats.txt"]
}
]
}
Each entry in the ``jobs`` array represents a single backend compilation job.
Each job object records its own command-line arguments and input/output files.
Shared arguments and inputs are defined once in the ``common`` object.
Reserved Entries:
- The first entry in the ``common.args`` array specifies the compiler
executable to invoke.
- The first entry in each job's ``inputs`` array is the bitcode file for the
module being compiled.
- The second entry in each job's ``inputs`` array is the corresponding
individual summary index file.
- The first entry in each job's ``outputs`` array is the primary output object
file.
For the ``outputs`` array, only the first entry is reserved for the primary
output file; there is no guaranteed order for the remaining entries. The primary
output file is specified in a reserved entry because some distribution systems
rely on this path - for example, to provide a meaningful user label for
compilation jobs. Initially, the DTLTO implementation will not produce more than
one output file. However, in the future, if LTO options are added that imply
additional output files, those files will also be included in this array.
Command-line arguments and input/output files are stored separately to allow
the remote compiler to be changed without updating the distributors, as the
distributors do not need to understand the details of the compiler command
line.
To generate the backend compilation commands, the common and job-specific
arguments are concatenated.
When consuming the example JSON above, a distributor is expected to issue the
following backend compilation commands with maximum parallelism:
.. code-block:: console
/usr/bin/clang -O2 -c -fprofile-sample-use=my.prof t1.o -fthinlto-index=t1.o.thinlto.bc -o t1.native.o \
-fproc-stat-report=t1.stats.txt
/usr/bin/clang -O2 -c -fprofile-sample-use=my.prof t2.o -fthinlto-index=t2.o.thinlto.bc -o t2.native.o \
-fproc-stat-report=t2.stats.txt
TODOs
-----
The following features are planned for DTLTO but not yet implemented:
- Support for the ThinLTO in-process cache.
- Support for platforms other than ELF and COFF.
- Support for archives with bitcode members.
- Support for more LTO configurations; only a very limited set of LTO
configurations is supported currently, e.g., support for basic block sections
is not currently available.
Constraints
-----------
- Matching versions of Clang and LLD should be used.
- The distributor used must support the JSON schema generated by the version of
LLD in use.
|
