Check the allocation was successful before freeing existing hashtable buckets

and increasing hashtable order

Fixes a crash observed when there is OOM in hashtable_do_rehash