pk: avoid out-of-memory errors

Estimate available memory and return -1 from mmap if not enough
is available, rather than assert-failing.