configury: --enable-link-serialization support

When performing LTO bootstraps, especially when using tmpfs for /tmp,
one can run a machine to halt when using higher levels of parallelism
and a large number of FEs, because there are too many concurrent LTO
link commands running at the same time and each one of them puts most of the
middle-end/backend objects into /tmp.

We have --enable-link-mutex configure option, but --enable-link-mutex has
a big problem that it decreases number of available jobs by the number of
link commands waiting for the lock, so e.g. when doing make -j32 build with
11 different big programs linked with $(LLINKER) we end up with just 22
effective jobs, and with e.g. make -j8 with those 11 different big programs
we actually most likely serialize everything during linking onto a single job.

The following patch implements a new configure option,
--enable-link-serialization, which implements different serialization and
as it doesn't use the mutex, just modifying the old option to be implemented
differently would be strange.  We can deprecate and later remove the old
option.  The new option doesn't use any shell mutexes, but uses make
dependencies.

The option is implemented inside of gcc/ configure and Makefiles,
which means that even inside of gcc/ make all (as well as e.g. make lto-dump)
will serialize and build all previous large binaries when configured this
way.
One can always make -j32 cc1 DO_LINK_SERIALIZATION=
to avoid that.
Furthermore, I've implemented the idea I wrote about, so that
--enable-link-serialization
is the same as
--enable-link-serialization=1
and means the large link commands are serialized, one can (the default)
--disable-link-serialization
which will cause all links to be parallelizable, but one can also
--enable-link-serialization=3
etc. which says that at most 3 of the large link commands can run
concurrently.
And finally I've implemented (only if the serialization is enabled) simple
progress bars for the linking.
With --enable-link-serialization and e.g. the 5 large links I have in my
current tree (cc1, cc1plus, f951, lto1 and lto-dump), before the linking it
prints
Linking |==--      | 20%
and after it
Linking |====      | 40%
(each == characters stand for already finished links, each --
characters stand for the link being started).
With --enable-link-serialization=3 it will change the way the start is
printed, one will get:
Linking |--        | 0%
at the start of cc1 link,
Linking |>>--      | 0%
at the start of the second large link and
Linking |>>>>--    | 0%
at the start of the third large link, where the >> characters stand for
already pending links.  The printing at the end of link command is
the same as with the full serialization, i.e. for the above 3:
Linking |==        | 20%
Linking |====      | 40%
Linking |======    | 60%
but one could actually get them in any order depending on which of those 3
finishes first - to get it 100% accurate I'd need to add some directory with
files representing finished links or similar, doesn't seem worth it.

2020-11-18  Jakub Jelinek  <jakub@redhat.com>

gcc/
	* configure.ac: Add $lang.prev rules, INDEX.$lang and SERIAL_LIST and
	SERIAL_COUNT variables to Make-hooks.
	(--enable-link-serialization): New configure option.
	* Makefile.in (DO_LINK_SERIALIZATION, LINK_PROGRESS): New variables.
	* doc/install.texi (--enable-link-serialization): Document.
	* configure: Regenerated.
gcc/c/
	* Make-lang.in (c.serial): New goal.
	(.PHONY): Add c.serial c.prev.
	(cc1$(exeext)): Call LINK_PROGRESS.
gcc/cp/
	* Make-lang.in (c++.serial): New goal.
	(.PHONY): Add c++.serial c++.prev.
	(cc1plus$(exeext)): Depend on c++.prev.  Call LINK_PROGRESS.
gcc/fortran/
	* Make-lang.in (fortran.serial): New goal.
	(.PHONY): Add fortran.serial fortran.prev.
	(f951$(exeext)): Depend on fortran.prev.  Call LINK_PROGRESS.
gcc/lto/
	* Make-lang.in (lto, lto1.serial, lto2.serial): New goals.
	(.PHONY): Add lto lto1.serial lto1.prev lto2.serial lto2.prev.
	(lto.all.cross, lto.start.encap): Remove dependencies.
	($(LTO_EXE)): Depend on lto1.prev.  Call LINK_PROGRESS.
	($(LTO_DUMP_EXE)): Depend on lto2.prev.  Call LINK_PROGRESS.
gcc/objc/
	* Make-lang.in (objc.serial): New goal.
	(.PHONY): Add objc.serial objc.prev.
	(cc1obj$(exeext)): Depend on objc.prev.  Call LINK_PROGRESS.
gcc/objcp/
	* Make-lang.in (obj-c++.serial): New goal.
	(.PHONY): Add obj-c++.serial obj-c++.prev.
	(cc1objplus$(exeext)): Depend on obj-c++.prev.  Call LINK_PROGRESS.
gcc/ada/
	* gcc-interface/Make-lang.in (ada.serial): New goal.
	(.PHONY): Add ada.serial ada.prev.
	(gnat1$(exeext)): Depend on ada.prev.  Call LINK_PROGRESS.
gcc/brig/
	* Make-lang.in (brig.serial): New goal.
	(.PHONY): Add brig.serial brig.prev.
	(brig1$(exeext)): Depend on brig.prev.  Call LINK_PROGRESS.
gcc/go/
	* Make-lang.in (go.serial): New goal.
	(.PHONY): Add go.serial go.prev.
	(go1$(exeext)): Depend on go.prev.  Call LINK_PROGRESS.
gcc/jit/
	* Make-lang.in (jit.serial): New goal.
	(.PHONY): Add jit.serial jit.prev.
	($(LIBGCCJIT_FILENAME)): Depend on jit.prev.  Call LINK_PROGRESS.
gcc/d/
	* Make-lang.in (d.serial): New goal.
	(.PHONY): Add d.serial d.prev.
	(d21$(exeext)): Depend on d.prev.  Call LINK_PROGRESS.

1 files changed, 6 insertions, 2 deletions

diff --git a/gcc/objcp/Make-lang.in b/gcc/objcp/Make-lang.in
index 5bfef24..2f6954e 100644
--- a/gcc/objcp/Make-lang.in
+++ b/gcc/objcp/Make-lang.in
@@ -39,9 +39,10 @@
 #
 # Define the names for selecting Objective-C++ in LANGUAGES.
 obj-c++: cc1objplus$(exeext)
+obj-c++.serial: obj-c++
 
 # Tell GNU make to ignore these if they exist.
-.PHONY: obj-c++
+.PHONY: obj-c++ obj-c++.serial obj-c++.prev
 
 # Use maximal warnings for this front end.  Also, make ObjC and C++
 # headers accessible.
@@ -66,9 +67,12 @@ cc1objplus-checksum.c : build/genchecksum$(build_exeext) checksum-options \
 	$(srcdir)/../move-if-change cc1objplus-checksum.c.tmp \
 	cc1objplus-checksum.c
 
-cc1objplus$(exeext): $(OBJCXX_OBJS) cc1objplus-checksum.o $(BACKEND) $(LIBDEPS)
+cc1objplus$(exeext): $(OBJCXX_OBJS) cc1objplus-checksum.o $(BACKEND) \
+		     $(LIBDEPS) obj-c++.prev
+	@$(call LINK_PROGRESS,$(INDEX.obj-c++),start)
 	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
 		$(OBJCXX_OBJS) cc1objplus-checksum.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
+	@$(call LINK_PROGRESS,$(INDEX.obj-c++),end)
 
 # Objective C++ language specific files.