switch to a higher performance set implementation
Taking a leaf out of some academic literature, this commit implements a simpler
but more high performance state set implementation. It's essentially an open
addressed hash table with linear probing, but with the ability to expand the
capacity as it fills up. It's not very well documented in the source yet because
it's only half finished. We also need to parallelise the expansion and make it
thread-safe, but I think this is best postponed until we regain multithreaded
checking.
This also modifies a command-line flag and introduces a new one:
* --set-capacity: This now specifies the initial size of the set that is
allocated. That is, if you pass 'x', a set size will be allocated such that
when full it will occupy 'x' bytes. This is perhaps counterintuitive and
maybe we should consider renaming this flag.
* --set-expand-threshold: The percentage occupancy at which we consider set
insertion inefficient and choose to expand the set.
TODO:
* Option to set "never" for --set-expand-threshold. This would just terminate
when we fill the set.
* More precise initial capacity calculation. See the FIXME around this.
* Parallelisation of the expansion logic.
With these changes long-running2 is checkable using --set-capacity 8589934592
--set-expand-threshold 100 in 79 seconds on a single core. It is actually slowed
by hitting swap on the machine I'm on (synecdoche). It looks like we could do it
closer to 45-50 seconds without touching swap.
parent
f67cf8c8
Please register or sign in to comment