crypto: fix UB in computing max message size

Before this commit it computed `(1<<(8*(15-iv_len)))-1` for `iv_len>=11`
and that reduces to `(1<<32)-1` for `iv_len==11`.  Left-shifting past
the sign bit and overflowing a signed integral type are both undefined
behaviors.

This commit switches to fixed values and restricts the `iv_len==11`
case to `INT_MAX`, as was already the case for all `iv_len<=10`.

PR-URL: https://github.com/nodejs/node/pull/21462


Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>