crypto: fix error handling

This fixes multiple cases where the wrong error was returned in
case of e.g. a overflow / wrong type.

PR-URL: https://github.com/nodejs/node/pull/19445


Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>