MSYS2+MinGW testing: Unix <-> Windows path conversion

On an MSYS2 system, I have:

 # which tclsh
 /mingw64/bin/tclsh

 # which tclsh86
 /mingw64/bin/tclsh86

 # which tclsh8.6
 /usr/bin/tclsh8.6

 # which expect
 /usr/bin/expect

The ones under /usr/bin are MSYS2 programs (linked with msys-2.0.dll).
I.e., they are really Cygwin (unix) ports of the programs.

The ones under /mingw64 are native Windows programs (NOT linked with
msys-2.0.dll).  You can check that with CYGWIN/MSYS2 ldd.

The MSYS2/Cygwin port of TCL (and thus expect) does not treat a file
name that starts with a drive letter as an absolute file name, while
the native/MinGW port does.  Vis:

 # cat file-join.exp
 puts [file join c:/ d:/]

 # /mingw64/bin/tclsh.exe file-join.exp
 d:/

 # /mingw64/bin/tclsh86.exe file-join.exp
 d:/

 # /usr/bin/expect.exe file-join.exp
 c:/d:

 # /usr/bin/tclsh8.6.exe file-join.exp
 c:/d:

When running the testsuite under MSYS2 to test mingw32 (Windows
native) GDB, we use MSYS2 expect (there is no MinGW port of expect
AFAIK).  Any TCL file manipulation routine will thus not consider
drive letters special, and just treats them as relative file names.

This results in several cases of the testsuite passing to GDB broken
file names, like:

 "C:/foo/C:/foo/bar"

or:

 "/c/foo/C:/foo/bar"

E.g., there is a "file join" in standard_output_file that results in
this:

 (gdb) file C:/gdb/build/outputs/gdb.base/info_sources_2/C:/gdb/build/outputs/gdb.base/info_sources_2/info_sources_2
 C:/gdb/build/outputs/gdb.base/info_sources_2/C:/gdb/build/outputs/gdb.base/info_sources_2/info_sources_2: No such file or directory.
 (gdb) ERROR: (info_sources_2) No such file or directory
 delete breakpoints

The bad "file join" comes from clean_restart $binfile, where $binfile
is an absolute host file name (thus has a drive letter), clean_restart
doing:

	set binfile [standard_output_file ${executable}]
	return [gdb_load ${binfile}]

and standard_output_file doing:

    # If running on MinGW, replace /c/foo with c:/foo
    if { [ishost *-*-mingw*] } {
        set dir [exec sh -c "cd ${dir} && pwd -W"]
    }
    return [file join $dir $basename]

Here, BASENAME was already an absolute file name that starts with a
drive letter, but "file join" treated it as a relative file name.

Another spot where we mishandle Unix vs drive letter file names, is in
the "dir" command that we issue when starting every testcase under
GDB.  We currently always pass the file name as seen from the build
machine (i.e., from MSYS2), which is a Unix file name that native
Windows GDB does not understand, resulting in:

 (gdb) dir /c/gdb/src/gdb/testsuite/gdb.rocm
 warning: /c/gdb/src/gdb/testsuite/gdb.rocm: No such file or directory
 Source directories searched: /c/gdb/src/gdb/testsuite/gdb.rocm;$cdir;$cwd

This patch introduces a systematic approach to handle all this, by
introducing the concepts of build file names (what DejaGnu sees) vs
host file names (what GDB sees).

This patches implements that in the following way:

1) - Keep standard_output_file's host-side semantics

standard_output_file currently converts the file name to a Windows
file name, using the "cd $dir; pwd -W" trick.  standard_output_file is
used pervasively, so I think it should keep the semantics that it
returns a host file name.

Note there is already a preexisting host_standard_output_file
procedure.  The difference to standard_output_file is that
host_standard_output_file handles remote hosts, while
standard_output_file assumes the build and host machines share a
filesystem.  The MSYS2 Unix path vs MinGW GDB drive letter case fall
in the "shared filesystem" bucket.  An NFS mount on the host at the
same mount point as on the build machine falls in that bucket too.

2) - Introduce build_standard_output_file

In some places, we are calling standard_output_file to find the
build-side file name, most often just to find the standard output
directory file name, and then immediately use that file name with TCL
file manipulation procedures, to do some file manipulation on the
build machine.  clean_standard_output_dir is an example of such a
case.  That code path is responsible for this bogus 'rm -rf' in
current MSYS2 testing:

 Running /c/gdb/src/gdb/testsuite/gdb.base/break.exp ...
 Executing on build: rm -rf /c/msys2/home/alves/gdb/build-testsuite/C:/msys2/home/alves/gdb/build-tests...

For these cases, add a variant of standard_output_file called
build_standard_output_file.  The main difference to
standard_output_file is that it doesn't do the "cd $dir; pwd -W"
trick.  I.e., it returns a path on the build machine.

3) Introduce host_file_sanitize

In some cases, we read an absolute file name out of GDB's output, and
then want to compare it against some other file name.  The file name
may originally come from the DWARF, and sometimes may have forward
slashes, and other times, it may have backward slashes.  Or the drive
letter may be uppercase, or it may be lowercase.  To make comparisons
easier, add a new host_file_sanitize procedure, that normalizes
slashes, and uppercases the drive letter.  It does no other
normalization.  Particularly, it does not turn a relative file name
into an absolute file name.

It's arguable whether GDB itself should do this sanitization.  I
suspect it should.  I personally dislike seeing backward slashes in
e.g., "info shared" output, or worse, mixed backward and forward
slashes.  Still, I propose starting with a testsuite adjustment that
moves us forward, and handle that separately.  I won't be surprised if
we need the new routine for some cases even if we adjust GDB.

4) build_file_normalize / host_file_normalize

In several places in the testsuite, we call "file normalize" on some
file name.  If we pass it a drive-letter file name, that TCL procedure
treats the passed in file name as a relative file name, so produces
something like /c/foo/C:/foo/bar.txt.

If the context calls for a build file name, then the "file normalize"
call should produce /c/foo/bar.txt.  If OTOH we need a host file name,
then it should produce "C:/foo/bar.txt".  Handle this by adding two
procedures that wrap "file normalize":

  - build_file_normalize
  - host_file_normalize

Initialy I implemented them in a very simple way, calling into
cygpath:

 proc build_file_normalize {filename} {
    if { [ishost *-*-mingw*] } {
      return [exec cygpath -ua $filename]
    } else {
      return [file normalize $filename]
    }
 }

 proc host_file_normalize {filename} {
    if { [ishost *-*-mingw*] } {
      return [exec cygpath -ma $filename]
    } else {
      return [file normalize $filename]
    }
 }

"cygpath" is a utility that comes OOTB with both Cygwin and MSYS2,
that does Windows <-> Cygwin file name conversion.

This works well, but because running the testsuite on Windows is so
slow, I thought of trying to avoid or minimize the cost of calling an
external utility ("cygpath").

On my system, calling into cygpath takes between 200ms to 350ms, and
these smallish costs (OK, not so small!) can creep up and compound an
already bad situation.  Note that the current call to "cd $dir; pwd
-W" has about the same cost as a "cygpath" call (though a little bit
cheaper).

So with this patch, we actually don't call cygpath at all, and no
longer use the "cd $dir; pwd -W" trick.  Instead we run the "mount"
command once, and cache the mapping (via gdb_caching_proc) between
Windows file names and Unix mount points, and then use that mapping in
host_file_normalize and build_file_normalize, to do the Windows <=>
Unix file name conversions ourselves.

One other small advantage here is that this approach works the same
for 'cygwin x mingw' testing [1], and 'msys x mingw' testing, while
"pwd -W" only works on MSYS2.

So I think the end result is that we should end up faster (or less
slow) than the current state.

(No, I don't have actual timings for the effect over a whole testsuite
run.)

5) Introduce host_file_join

For the "file join" call done from within standard_output_file (and
probably in future other places), since that procedure works with host
file names, add a new host_file_join procedure that is a wrapper
around "file join" that is aware of Windows drive letters.

======

With the infrastructure described above in place, the "dir" case is
fixed by simply calling host_file_normalize on the directory name,
before passing it to GDB.  That turns:

 (gdb) dir /c/gdb/src/gdb/testsuite/gdb.base
 warning: /c/gdb/src/gdb/testsuite/gdb.base: No such file or directory
 Source directories searched: /c/gdb/src/gdb/testsuite/gdb.base;$cdir;$cwd

Into:

 (gdb) dir C:/gdb/src/gdb/testsuite/gdb.base
 Source directories searched: C:/gdb/src/gdb/testsuite/gdb.base;$cdir;$cwd

Running the testsuite on GNU/Linux reveals that that change requires
tweaks to gdb.guile/scm-parameter.exp and gdb.python/py-parameter.exp,
to run the expected directory by host_file_normalize too, so that it
matches the directory we initially pass GDB at startup time.  Without
that fix, there could be a mismatch if the GDB sources path has a
symlink component, which now gets resolved by the host_file_normalize
call.

The theory is that most standard_output_file uses will not need to be
adjusted.

I grepped for "file normalize" and "file join", to find cases that
might need adjustment, and fixed those that required fixing.  The
fixes are included in this patch, to make it easier to reason about
the overall change.  E.g., in gdb.base/fullname.exp, without the fix,
we get:

 Running /c/gdb/src/gdb/testsuite/gdb.base/fullname.exp ...
 ERROR: tcl error sourcing /c/gdb/src/gdb/testsuite/gdb.base/fullname.exp.
 ERROR: tcl error code NONE
 ERROR: C:/msys2/home/alves/gdb/build-testsuite/outputs/gdb.base/fullname/tmp-fullname.c not a subdir of /c/msys2/home/alves/gdb/build-testsuite

In gdb.base/source-dir.exp, we have several issues.  E.g., we see the
"/c/foo/c:/foo" problem there too:

 dir /c/msys2/home/alves/gdb/build-testsuite/C:/msys2/home/alves/gdb/build-testsuite/outputs/gdb.base/source-dir/C:/msys2/home/alves/gdb/build-testsuite/outputs
 warning: /c/msys2/home/alves/gdb/build-testsuite/C:/msys2/home/alves/gdb/build-testsuite/outputs/gdb.base/source-dir/C:/msys2/home/alves/gdb/build-testsuite/outputs: No such file or directory
 Source directories searched: /c/msys2/home/alves/gdb/build-testsuite/C:/msys2/home/alves/gdb/build-testsuite/outputs/gdb.base/source-dir/C:/msys2/home/alves/gdb/build-testsuite/outputs;$cdir;$cwd
 (gdb) PASS: gdb.base/source-dir.exp: setup source path search directory

...

Executing on host: x86_64-w64-mingw32-gcc  \
  -fno-stack-protector \
  /c/msys2/home/alves/gdb/build-testsuite/C:/msys2/home/alves/gdb/build-testsuite/outputs/gdb.base/macro-source-path/cwd/macro-source-path.c ...
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

... and we need to handle Unix file names that we pass to the compiler
(on the build side), vs file names that GDB prints out (the host
side).

Similarly in the other testcases.

I haven't yet tried to do a full testsuite run on MSYS2, and I'm quite
confident there will be more places that will need similar adjustment,
but I'd like to land the infrastructure early, so that the rest of the
testsuite can be adjusted incrementally, and others can help.

Change-Id: I664dbb86d0efa4fa8db405577bea2b4b4a96a613

0 files changed, 0 insertions, 0 deletions