diff options
| author | Raiki Tamura <tamaron1203@gmail.com> | 2023-06-18 23:25:31 +0900 |
|---|---|---|
| committer | CohenArthur <arthur.cohen@embecosm.com> | 2023-06-28 15:43:33 +0000 |
| commit | 3d2a0c0445c2a1417dcc290b7a14e551c14fa421 (patch) | |
| tree | a1bfcb94275a2daf39700b72abfb55b8f68ed7b2 /gcc/rust/rust-lang.cc | |
| parent | 2983beb0d9101a8bf31890951f4a4094bd603f08 (diff) | |
| download | gcc-3d2a0c0445c2a1417dcc290b7a14e551c14fa421.zip gcc-3d2a0c0445c2a1417dcc290b7a14e551c14fa421.tar.gz gcc-3d2a0c0445c2a1417dcc290b7a14e551c14fa421.tar.bz2 | |
gccrs: Refactor lexer to handle UTF-8
gcc/rust/ChangeLog:
* lex/rust-lex.cc (is_float_digit):Change types of param to `uint32_t`
(is_x_digit):Likewise
(is_octal_digit):Likewise
(is_bin_digit):Likewise
(check_valid_float_dot_end):Likewise
(is_whitespace):Likewise
(is_non_decimal_int_literal_separator):Likewise
(is_identifier_start):Likewise
(is_identifier_continue):Likewise
(Lexer::skip_broken_string_input):Likewise
(Lexer::build_token):Remove handling BOM
(Lexer::parse_in_type_suffix):Modify use of `current_char`
(Lexer::parse_in_decimal):Likewise
(Lexer::parse_escape):Likewise
(Lexer::parse_utf8_escape):Likewise
(Lexer::parse_partial_string_continue):Likewise
(Lexer::parse_partial_hex_escape):Likewise
(Lexer::parse_partial_unicode_escape):Likewise
(Lexer::parse_byte_char):Likewise
(Lexer::parse_byte_string):Likewise
(Lexer::parse_raw_byte_string):Likewise
(Lexer::parse_raw_identifier):Likewise
(Lexer::parse_non_decimal_int_literal):Likewise
(Lexer::parse_decimal_int_or_float):Likewise
(Lexer::peek_input):Change return type to `Codepoint`
(Lexer::get_input_codepoint_length):Change to return 1
(Lexer::peek_codepoint_input):Change to be wrapper of `peek_input`
(Lexer::skip_codepoint_input):Change to be wrapper of `skip_input`
(Lexer::test_get_input_codepoint_n_length):Deleted
(Lexer::split_current_token):Deleted
(Lexer::test_peek_codepoint_input):Deleted
(Lexer::start_line):Move backwards
(assert_source_content):New helper function for selftest
(test_buffer_input_source):New helper function for selftest
(test_file_input_source):Likewise
(rust_input_source_test):New test
* lex/rust-lex.h (rust_input_source_test):New test
* rust-lang.cc (run_rust_tests):Add selftest
Signed-off-by: Raiki Tamura <tamaron1203@gmail.com>
Diffstat (limited to 'gcc/rust/rust-lang.cc')
| -rw-r--r-- | gcc/rust/rust-lang.cc | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/gcc/rust/rust-lang.cc b/gcc/rust/rust-lang.cc index 1f04701..9d05bc8 100644 --- a/gcc/rust/rust-lang.cc +++ b/gcc/rust/rust-lang.cc @@ -38,6 +38,7 @@ #include "rust-privacy-ctx.h" #include "rust-ast-resolve-item.h" #include "rust-optional.h" +#include "rust-lex.h" #include <mpfr.h> // note: header files must be in this order or else forward declarations don't @@ -452,6 +453,7 @@ void run_rust_tests () { // Call tests for the rust frontend here + rust_input_source_test (); rust_cfg_parser_test (); rust_privacy_ctx_test (); rust_crate_name_validation_test (); |