Commit 5637524b authored by Matthew Fernandez's avatar Matthew Fernandez
Browse files

switch to a higher performance set implementation

Taking a leaf out of some academic literature, this commit implements a simpler
but more high performance state set implementation. It's essentially an open
addressed hash table with linear probing, but with the ability to expand the
capacity as it fills up. It's not very well documented in the source yet because
it's only half finished. We also need to parallelise the expansion and make it
thread-safe, but I think this is best postponed until we regain multithreaded
checking.

This also modifies a command-line flag and introduces a new one:

  * --set-capacity: This now specifies the initial size of the set that is
    allocated. That is, if you pass 'x', a set size will be allocated such that
    when full it will occupy 'x' bytes. This is perhaps counterintuitive and
    maybe we should consider renaming this flag.
  * --set-expand-threshold: The percentage occupancy at which we consider set
    insertion inefficient and choose to expand the set.

TODO:

  * Option to set "never" for --set-expand-threshold. This would just terminate
    when we fill the set.
  * More precise initial capacity calculation. See the FIXME around this.
  * Parallelisation of the expansion logic.

With these changes long-running2 is checkable using --set-capacity 8589934592
--set-expand-threshold 100 in 79 seconds on a single core. It is actually slowed
by hitting swap on the machine I'm on (synecdoche). It looks like we could do it
closer to 45-50 seconds without touching swap.
parent f67cf8c8
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment