Eliminate RefMap

It's an unconvincing means to pretend there isn't mutable state when
there really is.  It's more confusing and less performant than just
calling a spade a spade.