test: expand test case for unknown file open flags

PR-URL: https://github.com/nodejs/node/pull/5590


Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Myles Borins <myles.borins@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>