InferAddressSpaces: Fix assert on inferred source for inttoptr/ptrtoint

If we had some source value we could infer an address space from that
went through a ptrtoint/inttoptr pair, this would fail since bitcast
can't change the address space.

Fixes issue 53665.