[Syntax] Remove delayed folding from tree building.

Summary:
This patch removes delayed folding and replaces it with forward peeking.

Delayed folding was previously used as a solution to the problem that
declaration doesn't have a representation in the AST. For example following
code:

```
int a,b;
```

is expressed in the AST as:

```
TranslationUnitDecl
|-...
|-VarDecl `int a`
`-VarDecl `int b`
```

And in the syntax tree we need:

```
*: TranslationUnit
`-SimpleDeclaration
  |-int
  |-SimpleDeclarator
  | `-a
  |-,
  |-SimpleDeclarator
  | `-b
  |-;
```

So in words, we need to create SimpleDeclaration to be a parent of
SimpleDeclarator nodes. Previously we used delayed folding to make sure SimpleDeclarations will be
eventually created. And in case multiple declarators requested declaration
creation, declaration range was extended to cover all declarators.

This design started to be hard to reason about, so we decided to replace it with
forward peeking. The last declarator node in the chain is responsible for creating
SimpleDeclaration for the whole chain. Range of the declaration corresponds to
the source range of the declarator node. Declarator decides whether its the last
one by peeking to the next AST node (see `isResponsibleForCreatingDeclaration`).

This patch does following:
* Removed delayed folding logic
* Tweaks Token.dumpForTests
* Moves getQualifiedNameStart inside BuildTreeVisitor
* Extracts BuildTreeVisitor.ProcessDeclaratorAndDeclaration
* Renames Builder.getDeclRange to Builder.getDeclarationRange and uses the
  method in all places.
* Adds a bunch of tests

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76922

1 files changed, 68 insertions, 3 deletions

diff --git a/clang/unittests/Tooling/Syntax/TreeTest.cpp b/clang/unittests/Tooling/Syntax/TreeTest.cpp
index ddd9092..8230457 100644
--- a/clang/unittests/Tooling/Syntax/TreeTest.cpp
+++ b/clang/unittests/Tooling/Syntax/TreeTest.cpp
@@ -224,6 +224,59 @@ void foo() {}
 )txt");
 }
 
+TEST_F(SyntaxTreeTest, SimpleVariable) {
+  expectTreeDumpEqual(
+      R"cpp(
+int a;
+int b = 42;
+    )cpp",
+      R"txt(
+*: TranslationUnit
+|-SimpleDeclaration
+| |-int
+| |-SimpleDeclarator
+| | `-a
+| `-;
+`-SimpleDeclaration
+  |-int
+  |-SimpleDeclarator
+  | |-b
+  | |-=
+  | `-UnknownExpression
+  |   `-42
+  `-;
+)txt");
+}
+
+TEST_F(SyntaxTreeTest, SimpleFunction) {
+  expectTreeDumpEqual(
+      R"cpp(
+void foo(int a, int b) {}
+    )cpp",
+      R"txt(
+*: TranslationUnit
+`-SimpleDeclaration
+  |-void
+  |-SimpleDeclarator
+  | |-foo
+  | `-ParametersAndQualifiers
+  |   |-(
+  |   |-SimpleDeclaration
+  |   | |-int
+  |   | `-SimpleDeclarator
+  |   |   `-a
+  |   |-,
+  |   |-SimpleDeclaration
+  |   | |-int
+  |   | `-SimpleDeclarator
+  |   |   `-b
+  |   `-)
+  `-CompoundStatement
+    |-{
+    `-}
+)txt");
+}
+
 TEST_F(SyntaxTreeTest, If) {
   expectTreeDumpEqual(
       R"cpp(
@@ -541,20 +594,32 @@ void test() {
 TEST_F(SyntaxTreeTest, MultipleDeclaratorsGrouping) {
   expectTreeDumpEqual(
       R"cpp(
-      int *a, b;
+      int *a, b; int *c, d;
   )cpp",
       R"txt(
 *: TranslationUnit
+|-SimpleDeclaration
+| |-int
+| |-SimpleDeclarator
+| | |-*
+| | `-a
+| |-,
+| |-SimpleDeclarator
+| | `-b
+| `-;
 `-SimpleDeclaration
   |-int
   |-SimpleDeclarator
   | |-*
-  | `-a
+  | `-c
   |-,
   |-SimpleDeclarator
-  | `-b
+  | `-d
   `-;
   )txt");
+}
+
+TEST_F(SyntaxTreeTest, MultipleDeclaratorsGroupingTypedef) {
   expectTreeDumpEqual(
       R"cpp(
     typedef int *a, b;