Add `skip_on_jobname` to test.json

Plan to replace the hard-coded list of 'may be skipped' framework tests in
skippable() with annotations in test.json which record 'will be skipped
in these specific CI jobs'.

If the value of the MESON_CI_JOBNAME env var (an arbitrary string
expected to be unique for each CI configuration) contains any of the
strings in the `skip_on_jobname` key in test.json, the test is expected
to output MESON_SKIP_TEST.

Unexpected skips or runs are treated as an error.

Future work: Maybe we should add additional count categories 'unexpected
skip' and 'unexpected not skipped', rather than counting those as 'skipped'
and 'failed', respectively.

1 files changed, 10 insertions, 0 deletions

diff --git a/docs/markdown/Contributing.md b/docs/markdown/Contributing.md
index aec73ec..77981d7 100644
--- a/docs/markdown/Contributing.md
+++ b/docs/markdown/Contributing.md
@@ -380,6 +380,16 @@ matched:
 | `literal` | Literal match (default) |
 | `re`      | regex match             |
 
+#### skip_on_jobname
+
+The `skip_on_jobname` key contains a list of strings. If the `MESON_CI_JOBNAME`
+environment variable is set, and any of them are a sub-string of it, the test is
+expected to be skipped (that is, it is expected that the test will output
+`MESON_SKIP_TEST`, because the CI environment is not one in which it can run,
+for whatever reason).
+
+The test is failed if it skips or runs unexpectedly.
+
 ### Skipping integration tests
 
 Meson uses several continuous integration testing systems that have