Github: related to #131 "minimal trace mode"

Github: Closes #131 "minimal trace mode"

The bail out condition will soon need to ask about the state depth as well.

Github: related to #131 "minimal trace mode"

In upcoming changes we'll need to read running_count within error() to determine
whether we're single threaded or not. To do this while avoiding torn reads, we
need to use atomic reads and writes with running_count.

Github: related to #131 "minimal trace mode"

Similarly to state_depth, we're going to need to refer to this earlier.

Github: related to #131 "minimal trace mode"

The purpose of this seemingly irrelevant change is to ensure we opt out of the
rendezvous protocol as soon as error() is called when MAX_ERRORS == 1. In
upcoming changes to ensure we get the shortest resulting trace, we're going to
loop indefinitely in error(). This is only possible without livelocking if we're
no longer part of the thread rendezvous protocol.

Github: related to #131 "minimal trace mode"

We were always passing the set_update function for this parameter, so just hard
code a call to this. The larger motivation for this is that we want to move the
rendezvous opt out to an earlier location in header.c. This backs out part of
8cf9d785.

Github: related to #131 "minimal trace mode"

In some upcoming changes we need to refer to it in preprocessor guards.

Github: related to #131 "minimal trace mode"

We're about to need to call it earlier in header.c.

Github: related to #131 "minimal trace mode"

Github: related to #132 "handle wider scalar types"

There was a race condition involving the rendezvous opt out and set expansion. I
can think of two ways in which the same problem occurs. Firstly:

  1. Thread A calls error() and begins the exit flow
  2. Other threads begin a set expansion and migration
  3. Thread A opts out of the rendezvous protocol
  4. Other threads finish the migration and pass through the rendezvous point

Secondly:

  1. Thread A calls error() and begins the exit flow
  2. Other threads begin a set expansion and migration
  3. Other threads finish the migration and arrive at the rendezvous point
  4. Thread A opts out of the rendezvous protocol

Observe that in both these scenarios the reference count of the old seen set
never drops to zero because the rendezvous opt out does not decrement it. This
means that no thread thinks it is the leader and the global_seen pointer doesn't
get updated. As a result, thread A now has a stale local_seen pointer that
points to a set full of tombstones.

We fix this with three relevant changes in this commit:

  1. Shift the next seen set into global_seen in the rendezvous interstice
     instead of prior. This ensures that this action is always performed by a
     thread when we pass through the rendezvous point.
  2. Re-acquire a local_seen pointer in exit_with() after thread joining. This
     ensures that the final thread has a local_seen that points at the currently
     active global_seen set.
  3. Drop the reference count to the global_seen set when opting out of the
     rendezvous protocol. This is not strictly necessary, but it allows us to
     preserve the property (and assertion) in refcounted_ptr_shift that the
     reference count is zero at this point.

The problem addressed here was never observed in the wild but rather something I
realised could happen by thinking about the concurrency interactions. The
changes described above should have no measurable impact on runtime or memory
usage.

Github: closes #124 "can the final thread see a stale local_seen?"

This change has no functional effect, just neater.

If this is required, you can use $PATH manipulation to make a different version
of afl-fuzz preferred/enabled. This is not necessary for the Travis CI job.

This means we pick up some garbage input like the Python and C files in the test
directory, but Rumur should be able to handle these without crashing as well so
we may as well include them in the testing corpus.

Curiously this was breaking compilation on FreeBSD by hiding some type
definitions. Casting around on the internet to debug the problem, most
references led me back to [0]. This seems odd, but apparently the right thing to
do is #define nothing on FreeBSD.

  [0]: https://lists.freebsd.org/pipermail/freebsd-standards/2004-March/000474.html

Github: related to #134 "Tests fail: UnicodeDecodeError: 'ascii' codec can't
  decode byte 0xe2 in position 120: ordinal not in range(128)"

This sort of all just worked when using a UTF-8 locale, but when your locale is
not setup this way exceptions occur. The problem is that we should have been
converting the binary data we received from a subprocess to a string immediately
if we intended to then go on and treat it like a string.

Github: related to #134 "Tests fail: UnicodeDecodeError: 'ascii' codec can't
  decode byte 0xe2 in position 120: ordinal not in range(128)"

If a non-UTF-8 locale is set, reading UTF-8 characters from the test files
fails.

Github: related to #134 "Tests fail: UnicodeDecodeError: 'ascii' codec can't
  decode byte 0xe2 in position 120: ordinal not in range(128)"

This is part of work to reduce the logic in afl-wrapper to a minimum. Ideally
almost everything it needs to do can be expressed in .travis.yml making the
fuzzing build more transparent to Travis.

This simplifies the code by (a) removing the intermediate allocation (b)
streamlining the expression of what we mean.

Github: related to #133 "invalid XML when expression contains sensitive
  characters"

Previously we were only escaping the format string which was not only
insufficient, but probably also wrong. We now construct the entire error message
and then escape it.

Github: closes #133 "invalid XML when expression contains sensitive characters"

Github: related to #133 "invalid XML when expression contains sensitive
  characters"

The long term plan here is to use the CI setup to build an AFL-instrumented
Rumur, in which case we'll be running with CC=afl-gcc, so we want a way of
avoiding picking this compiler up in the harness.

Since this script is only ever intended to run in CI, we may as well just let it
fail messily so we can more easily diagnose problems from the exception trace.

Github: closes #122 "AFL fuzzing in CI"

Github: related to #122 "AFL fuzzing in CI"

Github: related to #122 "AFL fuzzing in CI"