Cleanup vm module memory leakage

There are some paths here that led to dangling contexts. By being
smarter with handle management we can get rid of all the cleanup code
and fix those issues.