X509_digest_sig(): Improve default hash for EdDSA and allow to return the chosen default

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15762)

1 files changed, 19 insertions, 10 deletions

diff --git a/doc/man3/X509_digest.pod b/doc/man3/X509_digest.pod
index 5d61679..4928e98 100644
--- a/doc/man3/X509_digest.pod
+++ b/doc/man3/X509_digest.pod
@@ -17,7 +17,8 @@ PKCS7_ISSUER_AND_SERIAL_digest
 
  int X509_digest(const X509 *data, const EVP_MD *type, unsigned char *md,
                  unsigned int *len);
- ASN1_OCTET_STRING *X509_digest_sig(const X509 *cert);
+ ASN1_OCTET_STRING *X509_digest_sig(const X509 *cert,
+                                    EVP_MD **md_used, int *md_is_fallback);
 
  int X509_CRL_digest(const X509_CRL *data, const EVP_MD *type, unsigned char *md,
                      unsigned int *len);
@@ -39,21 +40,29 @@ PKCS7_ISSUER_AND_SERIAL_digest
 
 =head1 DESCRIPTION
 
-X509_digest_sig() calculates a digest of the given certificate
-using the same hash algorithm as in its signature with a fallback to B<SHA-256>
-for algorithms where the digest is an integral part of the signature algorithm
-such as with B<EdDSA> signatures.
+X509_digest_sig() calculates a digest of the given certificate I<cert>
+using the same hash algorithm as in its signature, if the digest
+is an integral part of the certificate signature algorithm identifier.
+Otherwise, a fallback hash algorithm is determined as follows:
+SHA512 if the signature alorithm is ED25519,
+SHAKE256 if it is ED448, otherwise SHA256.
+The output parmeters are assigned as follows.
+Unless I<md_used> is NULL, the hash algorithm used is provided
+in I<*md_used> and must be freed by the caller (if it is not NULL).
+Unless I<md_is_fallback> is NULL,
+the I<*md_is_fallback> is set to 1 if the hash algorithm used is a fallback,
+otherwise to 0.
 
 X509_pubkey_digest() returns a digest of the DER representation of the public
-key in the specified X509 B<data> object.
+key in the specified X509 I<data> object.
 
 All other functions described here return a digest of the DER representation
-of their entire B<data> objects.
+of their entire I<data> objects.
 
-The B<type> parameter specifies the digest to
-be used, such as EVP_sha1(). The B<md> is a pointer to the buffer where the
+The I<type> parameter specifies the digest to
+be used, such as EVP_sha1(). The I<md> is a pointer to the buffer where the
 digest will be copied and is assumed to be large enough; the constant
-B<EVP_MAX_MD_SIZE> is suggested. The B<len> parameter, if not NULL, points
+B<EVP_MAX_MD_SIZE> is suggested. The I<len> parameter, if not NULL, points
 to a place where the digest size will be stored.
 
 =head1 RETURN VALUES