Generate virtual locations for tokens

This second instalment uses the infrastructure of the previous patch
to allocate a macro map for each macro expansion and assign a virtual
location to each token resulting from the expansion.

To date when cpp_get_token comes across a token that happens to be a
macro, the macro expander kicks in, expands the macro, pushes the
resulting tokens onto a "token context" and returns a dummy padding
token. The next call to cpp_get_token goes look into the token context
for the next token [which is going to result from the previous macro
expansion] and returns it.  If the token is a macro, the macro expander
kicks in and you know the story.

This patch piggy-backs on that macro expansion process, so to speak.
First it modifies the macro expander to make it create a macro map for
each macro expansion. It then allocates a virtual location for each
resulting token.  Virtual locations of tokens resulting from macro
expansions are then stored on a special kind of context called an
"expanded tokens context".  In other words, in an expanded tokens
context, there are tokens resulting from macro expansion and their
associated virtual locations.  cpp_get_token_with_location is modified
to return the virtual location of tokens resulting from macro
expansion.  Note that once all tokens from an expanded token context have
been consumed and the context and is freed, the memory used to store the
virtual locations of the tokens held in that context is freed as well.
This helps reducing the overall peak memory consumption.

The client code that was getting macro expansion point location from
cpp_get_token_with_location now gets virtual location from it. Those
virtual locations can in turn be resolved into the different
interesting physical locations thanks to the linemap API exposed by
the previous patch.

Expensive progress. Possibly. So this whole virtual location
allocation business is switched off by default. So by default no
extended token is created. No extended token context is created
either. One has to use -ftrack-macro-expansion to switch this on. This
complicates the code but I believe it can be useful as some of our
friends found out at http://llvm.org/bugs/show_bug.cgi?id=5610

The patch tries to reduce the memory consumption by freeing some token
context memory that was being reused before. I didn't notice any
compilation slow down due to this immediate freeing on my GNU/Linux
system.

As no client code tries to resolve virtual locations to anything but
what was being done before, no new test case has been added.

Co-Authored-By: Dodji Seketeli <dodji@redhat.com>

From-SVN: r180082

7 files changed, 58 insertions, 4 deletions

diff --git a/gcc/ChangeLog b/gcc/ChangeLog
index a6f0e00..4c3b76d 100644
--- a/gcc/ChangeLog
+++ b/gcc/ChangeLog
@@ -1,3 +1,10 @@
+2011-10-15  Tom Tromey  <tromey@redhat.com>
+	    Dodji Seketeli  <dodji@redhat.com>
+
+	* doc/cppopts.texi (-ftrack-macro-expansion): Document new option.
+	* doc/invoke.texi (-ftrack-macro-expansion): Add this to the list of
+	preprocessor related options.
+
 2011-10-15  Tom Tromey  <tromey@redhat>
 	    Dodji Seketeli  <dodji@redhat.com>
 
diff --git a/gcc/c-family/ChangeLog b/gcc/c-family/ChangeLog
index b6ed818..789e448 100644
--- a/gcc/c-family/ChangeLog
+++ b/gcc/c-family/ChangeLog
@@ -1,6 +1,15 @@
 2011-10-15  Tom Tromey  <tromey@redhat.com>
 	    Dodji Seketeli  <dodji@redhat.com>
 
+	* c.opt (ftrack-macro-expansion): New option. Handle it with and
+	without argument.
+	* c-opts.c (c_common_handle_option)<case
+	OPT_ftrack_macro_expansion_, case OPT_ftrack_macro_expansion>: New
+	cases. Handle -ftrack-macro-expansion with and without argument.
+
+2011-10-15  Tom Tromey  <tromey@redhat.com>
+	    Dodji Seketeli  <dodji@redhat.com>
+
 	* c-ppoutput.c (scan_translation_unit, maybe_print_line)
 	(print_line, cb_define, do_line_change): Adjust to avoid touching
 	the internals of struct line_map.  Use the public API instead.
diff --git a/gcc/c-family/c-opts.c b/gcc/c-family/c-opts.c
index 49ff80d..3184539 100644
--- a/gcc/c-family/c-opts.c
+++ b/gcc/c-family/c-opts.c
@@ -628,6 +628,18 @@ c_common_handle_option (size_t scode, const char *arg, int value,
       cpp_opts->preprocessed = value;
       break;
 
+    case OPT_ftrack_macro_expansion:
+      if (value)
+	value = 2;
+      /* Fall Through.  */
+
+    case OPT_ftrack_macro_expansion_:
+      if (arg && *arg != '\0')
+	cpp_opts->track_macro_expansion = value;
+      else
+	cpp_opts->track_macro_expansion = 2;
+      break;
+
     case OPT_frepo:
       flag_use_repository = value;
       if (value)
diff --git a/gcc/c-family/c.opt b/gcc/c-family/c.opt
index a1b25ba..d977115 100644
--- a/gcc/c-family/c.opt
+++ b/gcc/c-family/c.opt
@@ -945,6 +945,14 @@ fpreprocessed
 C ObjC C++ ObjC++
 Treat the input file as already preprocessed
 
+ftrack-macro-expansion
+C ObjC C++ ObjC++ JoinedOrMissing RejectNegative UInteger
+; converted into ftrack-macro-expansion=
+
+ftrack-macro-expansion=
+C ObjC C++ ObjC++ JoinedOrMissing RejectNegative UInteger
+-ftrack-macro-expansion=<0|1|2>  Track locations of tokens coming from macro expansion and display them in error messages
+
 fpretty-templates
 C++ ObjC++ Var(flag_pretty_templates) Init(1)
 -fno-pretty-templates Do not pretty-print template specializations as the template signature followed by the arguments
diff --git a/gcc/doc/cppopts.texi b/gcc/doc/cppopts.texi
index 5212478..b225236 100644
--- a/gcc/doc/cppopts.texi
+++ b/gcc/doc/cppopts.texi
@@ -583,6 +583,24 @@ correct column numbers in warnings or errors, even if tabs appear on the
 line.  If the value is less than 1 or greater than 100, the option is
 ignored.  The default is 8.
 
+@item -ftrack-macro-expansion@r{[}=@var{level}@r{]}
+@opindex ftrack-macro-expansion
+Track locations of tokens across macro expansions. This allows the
+compiler to emit diagnostic about the current macro expansion stack
+when a compilation error occurs in a macro expansion. Using this
+option makes the preprocessor and the compiler consume more
+memory. The @var{level} parameter can be used to choose the level of
+precision of token location tracking thus decreasing the memory
+consumption if necessary. Value @samp{0} of @var{level} de-activates
+this option just as if no @option{-ftrack-macro-expansion} was present
+on the command line. Value @samp{1} tracks tokens locations in a
+degraded mode for the sake of minimal memory overhead. In this mode
+all tokens resulting from the expansion of an argument of a
+function-like macro have the same location. Value @samp{2} tracks
+tokens locations completely. This value is the most memory hungry.
+When this option is given no argument, the default parameter value is
+@samp{2}.
+
 @item -fexec-charset=@var{charset}
 @opindex fexec-charset
 @cindex character set, execution
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 7d1ae1b..b2c26ed 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -429,9 +429,9 @@ Objective-C and Objective-C++ Dialects}.
 -iwithprefixbefore @var{dir}  -isystem @var{dir} @gol
 -imultilib @var{dir} -isysroot @var{dir} @gol
 -M  -MM  -MF  -MG  -MP  -MQ  -MT  -nostdinc  @gol
--P  -fworking-directory  -remap @gol
--trigraphs  -undef  -U@var{macro}  -Wp,@var{option} @gol
--Xpreprocessor @var{option}}
+-P -ftrack-macro-expansion -fworking-directory @gol
+-remap -trigraphs  -undef  -U@var{macro}  @gol
+-Wp,@var{option} -Xpreprocessor @var{option}}
 
 @item Assembler Option
 @xref{Assembler Options,,Passing Options to the Assembler}.
diff --git a/gcc/input.c b/gcc/input.c
index 83344d7..89af274 100644
--- a/gcc/input.c
+++ b/gcc/input.c
@@ -1,5 +1,5 @@
 /* Data and functions related to line maps and input files.
-   Copyright (C) 2004, 2007, 2008, 2009, 2010
+   Copyright (C) 2004, 2007, 2008, 2009, 2010, 2011
    Free Software Foundation, Inc.
 
 This file is part of GCC.