Fix UB on signed overflow in mulh routine

We want to evaluate whether the product of a and b is zero mod 2^64,
but the product might overflow, resulting in UB.  If we instead perform
the computation in unsigned arithmetic, the overflow behavior is defined.

Resolves #1538