Un-break "--enable-32bit"

We had the wrong set of compiler options for 32-bit builds, "-m32"
hasn't been supported in a while.

This should fix https://github.com/riscv/riscv-gnu-toolchain/issues/294