Allow non-ASCII characters in Rust identifiers

Rust 1.53 (quite a while ago now) ungated the support for non-ASCII
identifiers.  This didn't work in gdb.  This is PR rust/20166.

This patch fixes the problem by allowing non-ASCII characters to be
considered as identifier components.  It seemed simplest to just pass
them through -- doing any extra checking didn't seem worthwhile.

The new test also verifies that such characters are allowed in strings
and character literals as well.  The latter also required a bit of
work in the lexer.

Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=20166

1 files changed, 26 insertions, 0 deletions

diff --git a/gdb/testsuite/gdb.rust/unicode.rs b/gdb/testsuite/gdb.rust/unicode.rs
new file mode 100644
index 0000000..c6ca90e
--- /dev/null
+++ b/gdb/testsuite/gdb.rust/unicode.rs
@@ -0,0 +1,26 @@
+// Copyright (C) 2022 Free Software Foundation, Inc.
+
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+#![allow(dead_code)]
+#![allow(unused_variables)]
+#![allow(unused_assignments)]
+#![allow(uncommon_codepoints)]
+#![allow(non_snake_case)]
+
+fn main() {
+    let 𝕯 = 98;
+    let cç = 97;
+    println!("{}, {}", 𝕯, cç);        // set breakpoint here
+}