Reapply [DebugInfo] Re-engineer a test to be stricter, add XFails

Was e05c1038, reverted in d7d0b17d, see D95617 for details. I've
added "arm64" to the XFail list (as well as aarch64), will follow up on
the mailing list about whether there's anything else to be done.