fs: improve error message for invalid flag

Flags on fs.open and others can be passed as strings or int.
Previously, if passing anything other than string or int,
the error message would only say that flags must be an int.

PR-URL: https://github.com/nodejs/node/pull/5590


Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Myles Borins <myles.borins@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>