diff options
author | Yitzhak Mandelbaum <yitzhakm@google.com> | 2021-11-15 18:55:50 +0000 |
---|---|---|
committer | Yitzhak Mandelbaum <yitzhakm@google.com> | 2021-11-17 13:40:46 +0000 |
commit | 2b4948448f03104b4b957860dd8c019d0b9df2f0 (patch) | |
tree | 636fb0b25e8759e8439cfebd3c794ffd5c84239d /clang/docs/ClangTransformerTutorial.rst | |
parent | f2d402e58b90eb2ee747ed96a2f7c5e7667023bd (diff) | |
download | llvm-2b4948448f03104b4b957860dd8c019d0b9df2f0.zip llvm-2b4948448f03104b4b957860dd8c019d0b9df2f0.tar.gz llvm-2b4948448f03104b4b957860dd8c019d0b9df2f0.tar.bz2 |
Add a clang-transformer tutorial
Differential Revision: https://reviews.llvm.org/D114011
Diffstat (limited to 'clang/docs/ClangTransformerTutorial.rst')
-rw-r--r-- | clang/docs/ClangTransformerTutorial.rst | 400 |
1 files changed, 400 insertions, 0 deletions
diff --git a/clang/docs/ClangTransformerTutorial.rst b/clang/docs/ClangTransformerTutorial.rst new file mode 100644 index 0000000..33931ad --- /dev/null +++ b/clang/docs/ClangTransformerTutorial.rst @@ -0,0 +1,400 @@ +========================== +Clang Transformer Tutorial +========================== + +A tutorial on how to write a source-to-source translation tool using Clang Transformer. + +.. contents:: + :local: + +What is Clang Transformer? +-------------------------- + +Clang Transformer is a framework for writing C++ diagnostics and program +transformations. It is built on the clang toolchain and the LibTooling library, +but aims to hide much of the complexity of clang's native, low-level libraries. + +The core abstraction of Transformer is the *rewrite rule*, which specifies how +to change a given program pattern into a new form. Here are some examples of +tasks you can achieve with Transformer: + +* warn against using the name ``MkX`` for a declared function, +* change ``MkX`` to ``MakeX``, where ``MkX`` is the name of a declared function, +* change ``s.size()`` to ``Size(s)``, where ``s`` is a ``string``, +* collapse ``e.child().m()`` to ``e.m()``, for any expression ``e`` and method named + ``m``. + +All of the examples have a common form: they identify a pattern that is the +target of the transformation, they specify an *edit* to the code identified by +the pattern, and their pattern and edit refer to common variables, like ``s``, +``e``, and ``m``, that range over code fragments. Our first and second examples also +specify constraints on the pattern that aren't apparent from the syntax alone, +like "``s`` is a ``string``." Even the first example ("warn ...") shares this form, +even though it doesn't change any of the code -- it's "edit" is simply a no-op. + +Transformer helps users succinctly specify rules of this sort and easily execute +them locally over a collection of files, apply them to selected portions of +a codebase, or even bundle them as a clang-tidy check for ongoing application. + +Who is Clang Transformer for? +----------------------------- + +Clang Transformer is for developers who want to write clang-tidy checks or write +tools to modify a large number of C++ files in (roughly) the same way. What +qualifies as "large" really depends on the nature of the change and your +patience for repetitive editing. In our experience, automated solutions become +worthwhile somewhere between 100 and 500 files. + +Getting Started +--------------- + +Patterns in Transformer are expressed with :doc:`clang's AST matchers <LibASTMatchers>`. +Matchers are a language of combinators for describing portions of a clang +Abstract Syntax Tree (AST). Since clang's AST includes complete type information +(within the limits of single `Translation Unit (TU)`_, +these patterns can even encode rich constraints on the type properties of AST +nodes. + +.. _`Translation Unit (TU)`: https://en.wikipedia.org/wiki/Translation_unit_\(programming\) + +We assume a familiarity with the clang AST and the corresponding AST matchers +for the purpose of this tutorial. Users who are unfamiliar with either are +encouraged to start with the recommended references in `Related Reading`_. + +Example: style-checking names +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Assume you have a style-guide rule which forbids functions from being named +"MkX" and you want to write a check that catches any violations of this rule. We +can express this a Transformer rewrite rule: + +.. code-block:: c++ + + makeRule(functionDecl(hasName("MkX").bind("fun"), + noopEdit(node("fun")), + cat("The name ``MkX`` is not allowed for functions; please rename")); + +``makeRule`` is our go-to function for generating rewrite rules. It takes three +arguments: the pattern, the edit, and (optionally) an explanatory note. In our +example, the pattern (``functionDecl(...)``) identifies the declaration of the +function ``MkX``. Since we're just diagnosing the problem, but not suggesting a +fix, our edit is an no-op. But, it contains an *anchor* for the diagnostic +message: ``node("fun")`` says to associate the message with the source range of +the AST node bound to "fun"; in this case, the ill-named function declaration. +Finally, we use ``cat`` to build a message that explains the change. Regarding the +name ``cat`` -- we'll discuss it in more detail below, but suffice it to say that +it can also take multiple arguments and concatenate their results. + +Note that the result of ``makeRule`` is a value of type +``clang::transformer::RewriteRule``, but most users don't need to care about the +details of this type. + +Example: renaming a function +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now, let's extend this example to a *transformation*; specifically, the second +example above: + +.. code-block:: c++ + + makeRule(declRefExpr(to(functionDecl(hasName("MkX")))), + changeTo(cat("MakeX")), + cat("MkX has been renamed MakeX")); + +In this example, the pattern (``declRefExpr(...)``) identifies any *reference* to +the function ``MkX``, rather than the declaration itself, as in our previous +example. Our edit (``changeTo(...)``) says to *change* the code matched by the +pattern *to* the text "MakeX". Finally, we use ``cat`` again to build a message +that explains the change. + +Here are some example changes that this rule would make: + ++--------------------------+----------------------------+ +| Original | Result | ++==========================+============================+ +| ``X x = MkX(3);`` | ``X x = MakeX(3);`` | ++--------------------------+----------------------------+ +| ``CallFactory(MkX, 3);`` | ``CallFactory(MakeX, 3);`` | ++--------------------------+----------------------------+ +| ``auto f = MkX;`` | ``auto f = MakeX;`` | ++--------------------------+----------------------------+ + +Example: method to function +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Next, let's write a rule to replace a method call with a (free) function call, +applied to the original method call's target object. Specifically, "change +``s.size()`` to ``Size(s)``, where ``s`` is a ``string``." We start with a simpler +change that ignores the type of ``s``. That is, it will modify *any* method call +where the method is named "size": + +.. code-block:: c++ + + llvm::StringRef s = "str"; + makeRule( + cxxMemberCallExpr( + on(expr().bind(s)), + callee(cxxMethodDecl(hasName("size")))), + changeTo(cat("Size(", node(s), ")")), + cat("Method ``size`` is deprecated in favor of free function ``Size``")); + +We express the pattern with the given AST matcher, which binds the method call's +target to ``s`` [#f1]_. For the edit, we again use ``changeTo``, but this +time we construct the term from multiple parts, which we compose with ``cat``. The +second part of our term is ``node(s)``, which selects the source code +corresponding to the AST node ``s`` that was bound when a match was found in the +AST for our rule's pattern. ``node(s)`` constructs a ``RangeSelector``, which, when +used in ``cat``, indicates that the selected source should be inserted in the +output at that point. + +Now, we probably don't want to rewrite *all* invocations of "size" methods, just +those on ``std::string``\ s. We can achieve this change simply by refining our +matcher. The rest of the rule remains unchanged: + +.. code-block:: c++ + + llvm::StringRef s = "str"; + makeRule( + cxxMemberCallExpr( + on(expr(hasType(namedDecl(hasName("std::string")))) + .bind(s)), + callee(cxxMethodDecl(hasName("size")))), + changeTo(cat("Size(", node(s), ")")), + cat("Method ``size`` is deprecated in favor of free function ``Size``")); + +Example: rewriting method calls +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In this example, we delete an "intermediary" method call in a string of +invocations. This scenario can arise, for example, if you want to collapse a +substructure into its parent. + +.. code-block:: c++ + + llvm::StringRef e = "expr", m = "member"; + auto child_call = cxxMemberCallExpr(on(expr().bind(e)), + callee(cxxMethodDecl(hasName("child")))); + makeRule(cxxMemberCallExpr(on(child_call), callee(memberExpr().bind(m)), + changeTo(cat(e, ".", member(m), "()"))), + cat("``child`` accessor is being removed; call ", + member(m), " directly on parent")); + +This rule isn't quite what we want: it will rewrite ``my_object.child().foo()`` to +``my_object.foo()``, but it will also rewrite ``my_ptr->child().foo()`` to +``my_ptr.foo()``, which is not what we intend. We could fix this by restricting +the pattern with ``not(isArrow())`` in the definition of ``child_call``. Yet, we +*want* to rewrite calls through pointers. + +To capture this idiom, we provide the ``access`` combinator to intelligently +construct a field/method access. In our example, the member access is expressed +as: + +.. code-block:: c++ + + access(e, cat(member(m))) + +The first argument specifies the object being accessed and the second, a +description of the field/method name. In this case, we specify that the method +name should be copied from the source -- specifically, the source range of ``m``'s +member. To construct the method call, we would use this expression in ``cat``: + +.. code-block:: c++ + + cat(access(e, cat(member(m))), "()") + +Reference: ranges, stencils, edits, rules +----------------------------------------- + +The above examples demonstrate just the basics of rewrite rules. Every element +we touched on has more available constructors: range selectors, stencils, edits +and rules. In this section, we'll briefly review each in turn, with references +to the source headers for up-to-date information. First, though, we clarify what +rewrite rules are actually rewriting. + +Rewriting ASTs to... Text? +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The astute reader may have noticed that we've been somewhat vague in our +explanation of what the rewrite rules are actually rewriting. We've referred to +"code", but code can be represented both as raw source text and as an abstract +syntax tree. So, which one is it? + +Ideally, we'd be rewriting the input AST to a new AST, but clang's AST is not +terribly amenable to this kind of transformation. So, we compromise: we express +our patterns and the names that they bind in terms of the AST, but our changes +in terms of source code text. We've designed Transformer's language to bridge +the gap between the two representations, in an attempt to minimize the user's +need to reason about source code locations and other, low-level syntactic +details. + +Range Selectors +^^^^^^^^^^^^^^^ + +Transformer provides a small API for describing source ranges: the +``RangeSelector`` combinators. These ranges are most commonly used to specify the +source code affected by an edit and to extract source code in constructing new +text. + +Roughly, there are two kinds of range combinators: ones that select a source +range based on the AST, and others that combine existing ranges into new ranges. +For example, ``node`` selects the range of source spanned by a particular AST +node, as we've seen, while ``after`` selects the (empty) range located immediately +after its argument range. So, ``after(node("id"))`` is the empty range immediately +following the AST node bound to ``id``. + +For the full collection of ``RangeSelector``\ s, see the header, +`clang/Tooling/Transformer/RangeSelector.h <https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Tooling/Transformer/RangeSelector.h>`_ + +Stencils +^^^^^^^^ + +Transformer offers a large and growing collection of combinators for +constructing output. Above, we demonstrated ``cat``, the core function for +constructing stencils. It takes a series of arguments, of three possible kinds: + +#. Raw text, to be copied directly to the output. +#. Selector: specified with a ``RangeSelector``, indicates a range of source text + to copy to the output. +#. Builder: an operation that constructs a code snippet from its arguments. For + example, the ``access`` function we saw above. + +Data of these different types are all represented (generically) by a ``Stencil``. +``cat`` takes text and ``RangeSelector``\ s directly as arguments, rather than +requiring that they be constructed with a builder; other builders are +constructed explicitly. + +In general, ``Stencil``\ s produce text from a match result. So, they are not +limited to generating source code, but can also be used to generate diagnostic +messages that reference (named) elements of the matched code, like we saw in the +example of rewriting method calls. + +Further details of the ``Stencil`` type are documented in the header file +`clang/Tooling/Transformer/Stencil.h <https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Tooling/Transformer/Stencil.h>`_. + +Edits +^^^^^ + +Transformer supports additional forms of edits. First, in a ``changeTo``, we can +specify the particular portion of code to be replaced, using the same +``RangeSelector`` we saw earlier. For example, we could change the function name +in a function declaration with: + +.. code-block:: c++ + + makeRule(functionDecl(hasName("bad")).bind(f), + changeTo(name(f), cat("good")), + cat("bad is now good")); + +We also provide simpler editing primitives for insertion and deletion: +``insertBefore``, ``insertAfter`` and ``remove``. These can all be found in the header +file +`clang/Tooling/Transformer/RewriteRule.h <https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Tooling/Transformer/RewriteRule.h>`_. + +We are not limited one edit per match found. Some situations require making +multiple edits for each match. For example, suppose we wanted to swap two +arguments of a function call. + +For this, we provide an overload of ``makeRule`` that takes a list of edits, +rather than just a single one. Our example might look like: + +.. code-block:: c++ + + makeRule(callExpr(...), + {changeTo(node(arg0), cat(node(arg2))), + changeTo(node(arg2), cat(node(arg0)))}, + cat("swap the first and third arguments of the call")); + +``EditGenerator``\ s (Advanced) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The particular edits we've seen so far are all instances of the ``ASTEdit`` class, +or a list of such. But, not all edits can be expressed as ``ASTEdit``\ s. So, we +also support a very general signature for edit generators: + +.. code-block:: c++ + + using EditGenerator = MatchConsumer<llvm::SmallVector<Edit, 1>>; + +That is, an ``EditGenerator`` is function that maps a ``MatchResult`` to a set +of edits, or fails. This signature supports a very general form of computation +over match results. Transformer provides a number of functions for working with +``EditGenerator``\ s, most notably +`flatten <https://github.com/llvm/llvm-project/blob/1fabe6e51917bcd7a1242294069c682fe6dffa45/clang/include/clang/Tooling/Transformer/RewriteRule.h#L165-L167>`_ +``EditGenerator``\ s, like list flattening. For the full list, see the header file +`clang/Tooling/Transformer/RewriteRule.h <https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Tooling/Transformer/RewriteRule.h>`_. + +Rules +^^^^^ + +We can also compose multiple *rules*, rather than just edits within a rule, +using ``applyFirst``: it composes a list of rules as an ordered choice, where +Transformer applies the first rule whose pattern matches, ignoring others in the +list that follow. If the matchers are independent then order doesn't matter. In +that case, ``applyFirst`` is simply joining the set of rules into one. + +The benefit of ``applyFirst`` is that, for some problems, it allows the user to +more concisely formulate later rules in the list, since their patterns need not +explicitly exclude the earlier patterns of the list. For example, consider a set +of rules that rewrite compound statements, where one rule handles the case of an +empty compound statement and the other handles non-empty compound statements. +With ``applyFirst``, these rules can be expressed compactly as: + +.. code-block:: c++ + + applyFirst({ + makeRule(compoundStmt(statementCountIs(0)).bind("empty"), ...), + makeRule(compoundStmt().bind("non-empty"),...) + }) + +The second rule does not need to explicitly specify that the compound statement +is non-empty -- it follows from the rules position in ``applyFirst``. For more +complicated examples, this can lead to substantially more readable code. + +Sometimes, a modification to the code might require the inclusion of a +particular header file. To this end, users can modify rules to specify include +directives with ``addInclude``. + +For additional documentation on these functions, see the header file +`clang/Tooling/Transformer/RewriteRule.h <https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Tooling/Transformer/RewriteRule.h>`_. + +Using a RewriteRule as a clang-tidy check +----------------------------------------- + +Transformer supports executing a rewrite rule as a +`clang-tidy <https://clang.llvm.org/extra/clang-tidy/>`_ check, with the class +``clang::tidy::utils::TransformerClangTidyCheck``. It is designed to require +minimal code in the definition. For example, given a rule +``MyCheckAsRewriteRule``, one can define a tidy check as follows: + +.. code-block:: c++ + + class MyCheck : public TransformerClangTidyCheck { + public: + MyCheck(StringRef Name, ClangTidyContext *Context) + : TransformerClangTidyCheck(MyCheckAsRewriteRule, Name, Context) {} + }; + +``TransformerClangTidyCheck`` implements the virtual ``registerMatchers`` and +``check`` methods based on your rule specification, so you don't need to implement +them yourself. If the rule needs to be configured based on the language options +and/or the clang-tidy configuration, it can be expressed as a function taking +these as parameters and (optionally) returning a ``RewriteRule``. This would be +useful, for example, for our method-renaming rule, which is parameterized by the +original name and the target. For details, see +`clang-tools-extra/clang-tidy/utils/TransformerClangTidyCheck.h <https://github.com/llvm/llvm-project/blob/main/clang-tools-extra/clang-tidy/utils/TransformerClangTidyCheck.h>`_ + +Related Reading +--------------- + +A good place to start understanding the clang AST and its matchers is with the +introductions on clang's site: + +* :doc:`Introduction to the Clang AST <IntroductionToTheClangAST>` +* :doc:`Matching the Clang AST <LibASTMatchers>` +* `AST Matcher Reference <https://clang.llvm.org/docs/LibASTMatchersReference.html>`_ + +.. rubric:: Footnotes + +.. [#f1] Technically, it binds it to the string "str", to which our + variable ``s`` is bound. But, the choice of that id string is + irrelevant, so elide the difference. |