fs: fix flag and mode validation

The `flag` and `mode` options were not being validated correctly.

Signed-off-by: James M Snell <jasnell@gmail.com>
Fixes: https://github.com/nodejs/node/issues/37430

PR-URL: https://github.com/nodejs/node/pull/37480


Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>