Fix hostname validation in the command-line tool to honour negative return values.

Specifically, an ASN.1 NumericString in the certificate CN will fail UTF-8 conversion
and result in a negative return value, which the "x509 -checkhost" command-line option
incorrectly interpreted as success.

Also update X509_check_host docs to reflect reality.

Thanks to Sean Burford (Google) for reporting this issue.

Reviewed-by: Richard Levitte <levitte@openssl.org>

1 files changed, 6 insertions, 1 deletions

diff --git a/crypto/x509v3/v3_utl.c b/crypto/x509v3/v3_utl.c
index 1ad3999..cd7a980 100644
--- a/crypto/x509v3/v3_utl.c
+++ b/crypto/x509v3/v3_utl.c
@@ -889,8 +889,13 @@ static int do_check_string(ASN1_STRING *a, int cmp_type, equal_fn equal,
         int astrlen;
         unsigned char *astr;
         astrlen = ASN1_STRING_to_UTF8(&astr, a);
-        if (astrlen < 0)
+        if (astrlen < 0) {
+            /*
+             * -1 could be an internal malloc failure or a decoding error from
+             * malformed input; we can't distinguish.
+             */
             return -1;
+        }
         rv = equal(astr, astrlen, (unsigned char *)b, blen, flags);
         if (rv > 0 && peername)
             *peername = BUF_strndup((char *)astr, astrlen);