[libc] Fix type errors on Windows

Fix the errors caused by having some numbers too large for a 32 bit
number in the tests for windows. Also fix the base causing some type
confusion.

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D108653