lib: remove and restructure calls to isNaN()

Switch condition order to check for null before calling isNaN().
Also remove two unnecessary calls to isNaN() that are already
covered by calls to isFinite(). This commit targets v0.10, as
opposed to #7891, which targets master (suggested by
@bnoordhuis). Closes #7840.

Signed-off-by: Fedor Indutny <fedor@indutny.com>