hrss: use less stack space.

The stack consumption of the HRSS functions is causing issues in
stack-constrained environments. Therefore allocate many variables on the
heap. This means that several HRSS_ functions now allocate, and thus can
fail, where they couldn't before. Callers that ignore the return value
and don't have crash-on-failure mallocs will still be safe, although
things will fail to decrypt later on.

Somehow, this actually makes key generation _faster_ on my machine. (I
don't know. Better alignment? Fewer L1 collisions?) The other operations
are slightly slower, as expected.

Before:

Did 17390 HRSS generate operations in 3054088us (5694.0 ops/sec)
Did 225000 HRSS encap operations in 3000512us (74987.2 ops/sec)
Did 87000 HRSS decap operations in 3014525us (28860.3 ops/sec)

After:

Did 21300 HRSS generate operations in 3026637us (7037.5 ops/sec)
Did 221000 HRSS encap operations in 3008911us (73448.5 ops/sec)
Did 84000 HRSS decap operations in 3007622us (27929.0 ops/sec)

Change-Id: I2312df8909af7d8d250c7c483c65038123f21ad9
Reviewed-on: https://boringssl-review.googlesource.com/c/boringssl/+/48345
Commit-Queue: Adam Langley <agl@google.com>
Reviewed-by: David Benjamin <davidben@google.com>

1 files changed, 14 insertions, 12 deletions

diff --git a/include/openssl/hrss.h b/include/openssl/hrss.h
index 5390696..016fe67 100644
--- a/include/openssl/hrss.h
+++ b/include/openssl/hrss.h
@@ -59,29 +59,31 @@ struct HRSS_public_key {
   (HRSS_POLY3_BYTES * 2 + HRSS_PUBLIC_KEY_BYTES + 2 + 32)
 
 // HRSS_generate_key is a deterministic function that outputs a public and
-// private key based on the given entropy.
-OPENSSL_EXPORT void HRSS_generate_key(
+// private key based on the given entropy. It returns one on success or zero
+// on malloc failure.
+OPENSSL_EXPORT int HRSS_generate_key(
     struct HRSS_public_key *out_pub, struct HRSS_private_key *out_priv,
     const uint8_t input[HRSS_GENERATE_KEY_BYTES]);
 
 // HRSS_encap is a deterministic function the generates and encrypts a random
 // session key from the given entropy, writing those values to |out_shared_key|
-// and |out_ciphertext|, respectively.
-OPENSSL_EXPORT void HRSS_encap(uint8_t out_ciphertext[HRSS_CIPHERTEXT_BYTES],
-                               uint8_t out_shared_key[HRSS_KEY_BYTES],
-                               const struct HRSS_public_key *in_pub,
-                               const uint8_t in[HRSS_ENCAP_BYTES]);
+// and |out_ciphertext|, respectively. It returns one on success or zero on
+// malloc failure.
+OPENSSL_EXPORT int HRSS_encap(uint8_t out_ciphertext[HRSS_CIPHERTEXT_BYTES],
+                              uint8_t out_shared_key[HRSS_KEY_BYTES],
+                              const struct HRSS_public_key *in_pub,
+                              const uint8_t in[HRSS_ENCAP_BYTES]);
 
 // HRSS_decap decrypts a session key from |ciphertext_len| bytes of
 // |ciphertext|. If the ciphertext is valid, the decrypted key is written to
 // |out_shared_key|. Otherwise the HMAC of |ciphertext| under a secret key (kept
 // in |in_priv|) is written. If the ciphertext is the wrong length then it will
 // leak which was done via side-channels. Otherwise it should perform either
-// action in constant-time.
-OPENSSL_EXPORT void HRSS_decap(uint8_t out_shared_key[HRSS_KEY_BYTES],
-                               const struct HRSS_private_key *in_priv,
-                               const uint8_t *ciphertext,
-                               size_t ciphertext_len);
+// action in constant-time. It returns one on success (whether the ciphertext
+// was valid or not) and zero on malloc failure.
+OPENSSL_EXPORT int HRSS_decap(uint8_t out_shared_key[HRSS_KEY_BYTES],
+                              const struct HRSS_private_key *in_priv,
+                              const uint8_t *ciphertext, size_t ciphertext_len);
 
 // HRSS_marshal_public_key serialises |in_pub| to |out|.
 OPENSSL_EXPORT void HRSS_marshal_public_key(