crypto: throw proper errors if out enc is UTF-16

Throw `Error`s instead of hard crashing when the `.digest()` output
encoding is UTF-16.

Fixes: https://github.com/nodejs/node/issues/9817
PR-URL: https://github.com/nodejs/node/pull/12752


Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>