test: fix crypto test case to use correct encoding

The callback would have errored out anyway due to the incorrect
encoding, and that error is not the point of this test case.

PR-URL: https://github.com/nodejs/node/pull/17956


Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>