aboutsummaryrefslogtreecommitdiff
path: root/clang/lib/Lex/Lexer.cpp
AgeCommit message (Collapse)AuthorFilesLines
2011-07-26Migrate 'Instantiation' data and API bits of SLocEntry to 'Expansion'Chandler Carruth1-3/+2
etc. With this I think essentially all of the SourceManager APIs are converted. Comments and random other bits of cleanup should be all thats left. llvm-svn: 136057
2011-07-26Convert InstantiationInfo and much of the related code to ExpansionInfoChandler Carruth1-2/+2
and various other 'expansion' based terms. I've tried to reformat where appropriate and catch as many references in comments but I'm going to do several more passes. Also I've tried to expand parameter names to be more clear where appropriate. llvm-svn: 136056
2011-07-26Rename create(MacroArg)InstantiationLoc to create(MacroArg)ExpansionLoc.Chandler Carruth1-4/+4
llvm-svn: 136054
2011-07-25Rename SourceManager::getImmediateInstantiationRange toChandler Carruth1-1/+1
getImmediateExpansionRange. llvm-svn: 135960
2011-07-25Rename SourceManager::getInstantiationRange to getExpansionRange.Chandler Carruth1-1/+1
llvm-svn: 135915
2011-07-25Mechanically rename SourceManager::getInstantiationLoc andChandler Carruth1-1/+1
FullSourceLoc::getInstantiationLoc to ...::getExpansionLoc. This is part of the API and documentation update from 'instantiation' as the term for macros to 'expansion'. llvm-svn: 135914
2011-07-23remove unneeded llvm:: namespace qualifiers on some core types now that ↵Chris Lattner1-15/+15
LLVM.h imports them into the clang namespace. llvm-svn: 135852
2011-07-20SpellingJoerg Sonnenberger1-1/+1
llvm-svn: 135545
2011-07-19Revamp the SourceManager to separate the representation of parsedDouglas Gregor1-3/+3
source locations from source locations loaded from an AST/PCH file. Previously, loading an AST/PCH file involved carefully pre-allocating space at the beginning of the source manager for the source locations and FileIDs that correspond to the prefix, and then appending the source locations/FileIDs used for parsing the remaining translation unit. This design forced us into loading PCH files early, as a prefix, whic has become a rather significant limitation. This patch splits the SourceManager space into two parts: for source location "addresses", the lower values (growing upward) are used to describe parsed code, while upper values (growing downward) are used for source locations loaded from AST/PCH files. Similarly, positive FileIDs are used to describe parsed code while negative FileIDs are used to file/macro locations loaded from AST/PCH files. As a result, we can load PCH/AST files even during parsing, making various improvemnts in the future possible, e.g., teaching #include <foo.h> to look for and load <foo.h.gch> if it happens to be already available. This patch was originally written by Sebastian Redl, then brought forward to the modern age by Jonathan Turner, and finally polished/finished by me to be committed. llvm-svn: 135484
2011-07-14Convert terminology in the Lexer from 'instantiate' and variants toChandler Carruth1-26/+27
'expand'. Also update the public API it provides to the new term, and propagate that update to the various clients. No functionality changed. llvm-svn: 135138
2011-07-07Move SourceManager::isAt[Start/End]OfMacroInstantiation functions to the ↵Argyrios Kyrtzidis1-1/+52
Lexer, since they depend on it now. llvm-svn: 134644
2011-07-07Make the Preprocessor more memory efficient and improve macro instantiation ↵Argyrios Kyrtzidis1-1/+1
diagnostics. When a macro instantiation occurs, reserve a SLocEntry chunk with length the full length of the macro definition source. Set the spelling location of this chunk to point to the start of the macro definition and any tokens that are lexed directly from the macro definition will get a location from this chunk with the appropriate offset. For any tokens that come from argument expansion, '##' paste operator, etc. have their instantiation location point at the appropriate place in the instantiated macro definition (the argument identifier and the '##' token respectively). This improves macro instantiation diagnostics: Before: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:5:11: note: instantiated from: int y = M(/); ^ After: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:3:20: note: instantiated from: \#define M(op) (foo op 3); ~~~ ^ ~ t.c:5:11: note: instantiated from: int y = M(/); ^ The memory savings for a candidate boost library that abuses the preprocessor are: - 32% less SLocEntries (37M -> 25M) - 30% reduction in PCH file size (900M -> 635M) - 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M) llvm-svn: 134587
2011-06-24Allow Lexer::getLocForEndOfToken to return the location just passed the ↵Argyrios Kyrtzidis1-2/+10
macro instantiation if the location given points at the last token of the macro instantiation. Fixes rdar://9045701. llvm-svn: 133804
2011-05-10Don't strlen() every file before parsing it.Eli Friedman1-1/+2
llvm-svn: 131132
2011-04-15fix a bunch of comment typos found by codespell. Patch byChris Lattner1-1/+1
Luis Felipe Strano Moraes! llvm-svn: 129559
2011-04-14Implement C++0x [lex.pptoken]p3's handling of <::.Richard Smith1-0/+15
llvm-svn: 129525
2011-04-09Eat the UTF-8 BOM at the beginning of a file since it's ignored anyhow.Eric Christopher1-1/+14
Nom Nom Nom. Patch by Anton Korobeynikov! llvm-svn: 129174
2011-04-06Fix getLocForEndOfToken to not double-count spurious internal characters John McCall1-1/+1
within a token, like trigraphs and escaped newlines. Patch by Marcin Kowalczyk! llvm-svn: 128978
2011-03-18Lexer: Add extremely limited support for -traditional-cpp, ignoring BCPLDaniel Dunbar1-3/+5
comments. llvm-svn: 127910
2011-03-08Fix my earlier commit to work with escaped newlines and leave breadcrumbsJohn McCall1-0/+48
in case we want to make a world where we can check intermediate instantiations for this kind of breadcrumb. llvm-svn: 127221
2011-02-28Rename tok::eom to tok::eod.Peter Collingbourne1-7/+7
The previous name was inaccurate as this token in fact appears at the end of every preprocessing directive, not just macro definitions. No functionality change, except for a diagnostic tweak. llvm-svn: 126631
2011-02-15Warn for missing terminating " or ' instead of error for gcc compatibility. ↵Argyrios Kyrtzidis1-2/+2
Fixed rdar://8914293. llvm-svn: 125616
2011-02-09Lexer: add CUDA kernel call tokensPeter Collingbourne1-0/+8
llvm-svn: 125218
2011-01-31Harden Lexer::GetBeginningOfToken() against bogus source locations andDouglas Gregor1-0/+6
the disappearance/alteration of files. llvm-svn: 124616
2010-12-22Introduced raw_identifier token kind.Abramo Bagnara1-25/+24
llvm-svn: 122394
2010-11-17move getSpelling from Preprocessor to Lexer, which it is more conceptually ↵Chris Lattner1-0/+101
related to. llvm-svn: 119479
2010-11-17move AdvanceToTokenCharacter and getLocForEndOfToken fromChris Lattner1-0/+77
Preprocessor to Lexer where they make more sense. llvm-svn: 119474
2010-10-23Update remaining attribute macros to new style.Chandler Carruth1-4/+2
llvm-svn: 117204
2010-09-30In MeasureTokenLength, the FileLoc supplied to the lexer must point to the ↵Sebastian Redl1-1/+2
start of the buffer, or we risk overflow. llvm-svn: 115117
2010-08-31improve isHexaLiteral to work with escaped newlines and trigraphs,Chris Lattner1-7/+8
patch by Francois Pichet! llvm-svn: 112602
2010-08-30silence a warningChris Lattner1-1/+1
llvm-svn: 112549
2010-08-30Revert my user-defined literal commits - r1124{58,60,67} pendingAlexis Hunt1-29/+2
some issues being sorted out. llvm-svn: 112493
2010-08-30add a fixme.Chris Lattner1-1/+5
llvm-svn: 112491
2010-08-30use 'features' instead of 'PP->getLangOptions'.Chris Lattner1-8/+8
llvm-svn: 112490
2010-08-30In Microsoft compatibility mode, don't parse the exponent as part ofDouglas Gregor1-1/+10
the pp-number in a hexadecimal floating point literal, from Francois Pichet! Fixes PR7968. llvm-svn: 112481
2010-08-29Implement C++0x user-defined string literals.Alexis Hunt1-2/+29
The extra data stored on user-defined literal Tokens is stored in extra allocated memory, which is managed by the PreprocessorLexer because there isn't a better place to put it that makes sure it gets deallocated, but only after it's used up. My testing has shown no significant slowdown as a result, but independent testing would be appreciated. llvm-svn: 112458
2010-08-25Introduce a preprocessor code-completion hook for contexts where weDouglas Gregor1-6/+22
expect "natural" language and should not provide any completions, e.g., comments, string literals, #error. llvm-svn: 112054
2010-08-24Introduce basic code-completion support for preprocessor directives,Douglas Gregor1-18/+18
e.g., after a "#" we'll suggest #if, #ifdef, etc. llvm-svn: 111943
2010-08-12Don't emit end-of-file diagnostics like "unterminated conditional" orDouglas Gregor1-4/+7
"unterminated string" when we're performing code completion. llvm-svn: 110933
2010-08-11Random temporary string cleanup.Benjamin Kramer1-1/+1
llvm-svn: 110807
2010-08-09Use precompiled preambles for in-process code completion.Douglas Gregor1-2/+16
llvm-svn: 110596
2010-07-26Introduce basic support for loading a precompiled preamble whileDouglas Gregor1-2/+13
reparsing an ASTUnit. When saving a preamble, create a buffer larger than the actual file we're working with but fill everything from the end of the preamble to the end of the file with spaces (so the lexer will quickly skip them). When we load the file, create a buffer of the same size, filling it with the file and then spaces. Then, instruct the lexer to start lexing after the preamble, therefore continuing the parse from the spot where the preamble left off. It's now possible to perform a simple preamble build + parse (+ reparse) with ASTUnit. However, one has to disable a bunch of checking in the PCH reader to do so. That part isn't committed; it will likely be handled with some other kind of flag (e.g., -fno-validate-pch). As part of this, fix some issues with null termination of the memory buffers created for the preamble; we were trying to explicitly NULL-terminate them, even though they were also getting implicitly NULL terminated, leading to excess warnings about NULL characters in source files. llvm-svn: 109445
2010-07-22Improve performance during cursor traversal when a region of interestDouglas Gregor1-0/+53
is present. Rather than using clang_getCursorExtent(), which requires us to lex the token at the ending position to determine its length. Then, we'd be comparing [a, b) source ranges that cover the characters in the range rather than the normal behavior for Clang's source ranges, which covers the tokens in the range. However, relexing causes us to read the source file (which may come from a precompiled header), which is rather unfortunate and affects performance. In the new scheme, we only use Clang-style source ranges that cover the tokens in the range. At the entry points where this matters (clang_annotateTokens, clang_getCursor), we make sure to move source locations to the start of the token. Addresses most of <rdar://problem/8049381>. llvm-svn: 109134
2010-07-20Introduce a new lexer function to compute the "preamble" of a file,Douglas Gregor1-0/+125
which is the part of the file that contains all of the initial comments, includes, and preprocessor directives that occur before any of the actual code. Added a new -print-preamble cc1 action that is only used for testing. llvm-svn: 108913
2010-07-07fix PR4499, patch by Kyle Dean!Chris Lattner1-24/+16
llvm-svn: 107836
2010-05-30simpler fix for rdar://8044135 - escaped newlines have alreadyChris Lattner1-10/+7
been processed, so they don't have to be tip-toed around. llvm-svn: 105182
2010-05-30Improve our handling of NULL after an escaping '\' in a stringDouglas Gregor1-2/+7
literal. Fixes <rdar://problem/8044135>. llvm-svn: 105181
2010-05-25Improve code completion in failure cases in two ways:Douglas Gregor1-0/+3
1) Suppress diagnostics as soon as we form the code-completion token, so we don't get any error/warning spew from the early end-of-file. 2) If we consume a code-completion token when we weren't expecting one, go into a code-completion recovery path that produces the best results it can based on the context that the parser is in. llvm-svn: 104585
2010-05-17robustify the conflict marker stuff. Don't add 7 twice, which wouldChris Lattner1-1/+2
make it miss (invalid) things like: <<<<<<< >>>>>>> and crash if <<<<<<< was at the end of the line. When we find a >>>>>>> that is not at the end of the line, make sure to reset Pos so we don't crash on something like: <<<<<<< >>>>>>> This isn't worth making testcases for, since each would require a new file. rdar://7987078 - signal 11 compiling "<<<<<<<<<<" llvm-svn: 103968
2010-05-16when code completing inside a C-style block comment, don't emit errors aboutChris Lattner1-2/+3
a missing */ since we truncated the file. This fixes rdar://7948776 llvm-svn: 103913
generated by cgit v1.1 at 2025-11-03 04:51:40 +0000