Handle doc comment strings in lexer and parser

Remove (unused) comment related tokens and replace them with
INNER_DOC_COMMENT and OUTER_DOC_COMMENT tokens, which keep the comment
text as a string. These can be constructed with the new
make_inner_doc_comment and make_outer_doc_comment methods.

Make sure to not confuse doc strings with normal comments in the lexer
when detecting shebang lines. Both single line //! and /*! */ blocks
are turned into INNER_DOC_COMMENT tokens. And both single line /// and
/** */ blocks are turned into OUTER_DOC_COMMENT tokens.

Also fixes some issues with cr/lf line endings and keeping the line
map correct when seeing \n in a comment.

In the parser handle INNER_DOC_COMMENT and OUTER_DOC_COMMENTS where
inner (#[]) and outer (#![]) attributes are handled. Add a method
parse_doc_comment which turns the tokens into an "doc" Attribute with
the string as literal expression.

Add get_locus method to Attribute class for better error reporting.

Tests are added for correctly placed and formatted doc strings, with
or without cr/lf line endings. Incorrect formatted (isolated CRs) doc
strings and badly placed inner doc strings. No tests add handling of
the actual doc attributes yet. These could be tested once we add
support for the #![warn(missing_docs)] attribute.

0 files changed, 0 insertions, 0 deletions