Modern browsers have built-in crypto, WebCrypto . It is fast and secure. OpenSSL is pervasive on the server side. This article is about how to encrypt with a public key in the browser and decrypt it on the server with OpenSSL. The problem I encountered is a thinly referenced component: message padding with a secure hash.



Security

A few essential browser security facts:

  • window.crypto.subtle is read-only so it cannot be overwritten with polyfill
  • crypto keys can be made unexportable so that malicious code can't steal them.
  • secure random number generation, an essential for secure crypto

Support in older browsers vary. I'm not going into those details and how if you decide to use WebCrypto things won't work for everyone everywhere. But this post from 2016 covers it from then. My view is that if people do not keep their browsers up to date then the security implications are much broader than WebCrypto support.

Public Key Encryption

I am interested in using the public key of an RSA pair to encrypt a small message that only the private key holder can decrypt. In this case the encryption is happening in the browser. Depending on the version of OpenSSL, I found that the padding digest algorithm support varied. The default for OpenSSL RSA encrypt and decrypt is SHA-1. Let me be more specific, when using RSA-OAEP, the RSA-OAEP-MD default is SHA-1. This MD business is part of the encryption process and doesn't pertain to the keys.

Importing a public key

To use a public key (or any key for that matter) WebCrypto has to import it. For my case:

window.crypto.subtle.importKey(
      "spki",
      binaryDer,
      {
        name: "RSA-OAEP",
        hash: "SHA-256"
      },
      true,
      ["encrypt"]
    );

The importKey specifics here are:

  • spki - this is specifically the mode of encrypting with a public key
  • binaryDer - is the DER encoded public key
  • algorithm - We are using RSA-OAEP and specifying the digest function "SHA-256" as the RSA-OAEP-MD function.
  • true - This sets if the key is exportable. Set to false if you want to inhibit export.
  • [ "encrypt" ] - a list of operations that this key can be used for

As you may see, some of these parameters are not about importing the key but how it can be used. The part that causes problems is the hash option. This option applies to the encryption process and not import. However, if you have an old version of OpenSSL or some software that tries to track the capabilities of OpenSSL and guesses wrong, being able to change the MD parameter may not be possible.

The simplest answer is to upgrade OpenSSL and whatever software you're using. Sometimes that's not possible and the way to fix it is to use hash: "SHA-1". Using SHA-1 is strongly discouraged because it is considered vulnerable. However, SHA-1 is the default for OpenSSL, so pick your poison.

One thing that surprised me in this investigation is that I could not find a utility that would indicate what RSA-OAEP-MD option was used. I'm not sure how consensus is supposed to be established except for ahead of time.

Here is an example page that covers the browser side.

OpenSSL

The question of padding with hashes came up in 2014 and it is a combination of openssl commands being "legacy" as well as deficiencies in the documentation. Let's look at some examples. All the examples use the same key pair and all encryption is done with the above browser example.

Encrypting some text using SHA-256 padding hash and trying to use the rasutl command on a system with OpenSSL 1.0.2k-fips results in:

$ base64 -d | openssl rsautl -inkey key -decrypt
QJYpNJ5O1RkupwXVbaRr5Qrn8QghF4vl7MXpUyPI9IKqPbc8+VdLurUCdWlzztA1lmCCuppPXabGF1TETcJH3nq4PsxO43hq2Kb2EVZBad+bLjWQCOr8fXlZPUftJqbQAi/NmW0TTneI49EK3nBMIzq+XfLGvH9cnJgS9zqTtlamrXaio2Hvc+SfS8S2dyowRI82+WsSsX3nQbgVOdGY9ctq8sDnBvfUm0nd3ml0CrGaXqJf4hCHOSvH4CgSiBKzmK9wYQ3ZgpqBEJzQDFl+aU+hfOqryUB4dzYL/ceYGwWqkwg0vHncXasCGj+u7JH51V0n+SMfennBcyZI3uEmWg==
RSA operation error
139719484266384:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:rsa_pk1.c:301:
139719484266384:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:643:

On a different system (using LibreSSL 2.6.5):

$ base64 -D | openssl rsautl -inkey key -decrypt
QJYpNJ5O1RkupwXVbaRr5Qrn8QghF4vl7MXpUyPI9IKqPbc8+VdLurUCdWlzztA1lmCCuppPXabGF1TETcJH3nq4PsxO43hq2Kb2EVZBad+bLjWQCOr8fXlZPUftJqbQAi/NmW0TTneI49EK3nBMIzq+XfLGvH9cnJgS9zqTtlamrXaio2Hvc+SfS8S2dyowRI82+WsSsX3nQbgVOdGY9ctq8sDnBvfUm0nd3ml0CrGaXqJf4hCHOSvH4CgSiBKzmK9wYQ3ZgpqBEJzQDFl+aU+hfOqryUB4dzYL/ceYGwWqkwg0vHncXasCGj+u7JH51V0n+SMfennBcyZI3uEmWg==
RSA operation error
4572378732:error:04FFF06B:rsa routines:CRYPTO_internal:block type is not 02:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/crypto/rsa/rsa_pk1.c:185:
4572378732:error:04FFF072:rsa routines:CRYPTO_internal:padding check failed:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/crypto/rsa/rsa_eay.c:580:

The error message, as is the general trend, will only make sense after you know what the problem is. What they mean is that the required padding has the wrong content. RSA encryption pad is essential, to quote:

For example RSA Encryption padding is randomized, ensuring that the same message encrypted multiple times looks different each time. It also avoids other weaknesses, such as encrypting the same message using different RSA keys leaking the message, or an attacker creating messages derived from some other ciphertexts.

From https://crypto.stackexchange.com/questions/3608/why-is-padding-used-for-rsa-encryption-given-that-it-is-not-a-block-cipher

You can find other more technical answers as well.

Multiple problems are afoot:

  • OpenSSL uses SHA-1 as the default padding algorithm
  • SHA-1 is considered vulnerable, though how that affects the RSA padding requirement is unclear to me.
  • rsautl has no options to change the padding algorithm
  • rsautl is considered deprecated and no docs say to use pkeyutl instead
  • The replacement pkeyutl is of a more generic nature and options are passed through to the cipher through this more generic interface
  • Nowhere in all the openssl documentation does it mention a way to set the padding algorithm.

The above github question from 2014 covers all of these issues, but you have to be lucky enough to find it! In the end, this is how it's done:

$ base64 -d | openssl pkeyutl -inkey key -decrypt -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256
QJYpNJ5O1RkupwXVbaRr5Qrn8QghF4vl7MXpUyPI9IKqPbc8+VdLurUCdWlzztA1lmCCuppPXabGF1TETcJH3nq4PsxO43hq2Kb2EVZBad+bLjWQCOr8fXlZPUftJqbQAi/NmW0TTneI49EK3nBMIzq+XfLGvH9cnJgS9zqTtlamrXaio2Hvc+SfS8S2dyowRI82+WsSsX3nQbgVOdGY9ctq8sDnBvfUm0nd3ml0CrGaXqJf4hCHOSvH4CgSiBKzmK9wYQ3ZgpqBEJzQDFl+aU+hfOqryUB4dzYL/ceYGwWqkwg0vHncXasCGj+u7JH51V0n+SMfennBcyZI3uEmWg==
The eagle flies at twilight