RSA_PUBLIC_ENCRYPT(3SSL)

📦
version 4.0.3 · 14 versions
, in ubuntu stonking

💖 RSA_PUBLIC_ENCRYPT(3SSL) 💖

okay bestie, so this is a page from the OpenSSL manual, section 3SSL, and it's literally called RSA_PUBLIC_ENCRYPT. Grab a strawberry 🍓 because we're talking RSA public key cryptography and there's some drama (deprecation drama, the messiest kind).

✨ NAME ✨

RSA_public_encrypt, RSA_private_decrypt — RSA public key cryptography, the classic one-two punch 🫶

🌸 SYNOPSIS 🌸

#include <openssl/rsa.h>

The following functions have been deprecated since OpenSSL 3.0, and can be hidden entirely by defining OPENSSL_API_COMPAT with a suitable version value, see openssl_user_macros(7):

int RSA_public_encrypt(int flen, const unsigned char *from,
                       unsigned char *to, RSA *rsa, int padding);

int RSA_private_decrypt(int flen, const unsigned char *from,
                        unsigned char *to, RSA *rsa, int padding);

Like, they still work, they're just in their retirement era 💅

🦋 DESCRIPTION 🦋

Not me having to say this first: both of the functions described on this page are deprecated. Girl, they're deprecated. Applications should instead use EVP_PKEY_encrypt_init_ex(3), EVP_PKEY_encrypt(3), EVP_PKEY_decrypt_init_ex(3) and EVP_PKEY_decrypt(3). That's the glow-up path, okay?

RSA_public_encrypt() encrypts the flen bytes at from (usually a session key) using the public key rsa and stores the ciphertext in to. to must point to RSA_size(rsa) bytes of memory. No cap.

padding denotes one of the following modes:

  • RSA_PKCS1_PADDING — PKCS #1 v1.5 padding. This currently is the most widely used mode. However, it is highly recommended to use RSA_PKCS1_OAEP_PADDING in new applications. SEE WARNING BELOW. 👀 (yes there is a warning, we get to it, stay with me)
  • RSA_PKCS1_OAEP_PADDING — EME-OAEP as defined in PKCS #1 v2.0 with SHA-1, MGF1 and an empty encoding parameter. This mode is recommended for all new applications. This is the main character. 👑
  • RSA_NO_PADDING — Raw RSA encryption. This mode should only be used to implement cryptographically sound padding modes in the application code. Encrypting user data directly with RSA is insecure. Literally insecure. Do not.

Now the numbers, and bestie the numbers matter:

  • When encrypting, flen must not be more than RSA_size(rsa) - 11 for the PKCS #1 v1.5 based padding modes, not more than RSA_size(rsa) - 42 for RSA_PKCS1_OAEP_PADDING and exactly RSA_size(rsa) for RSA_NO_PADDING.
  • When a padding mode other than RSA_NO_PADDING is in use, then RSA_public_encrypt() will include some random bytes into the ciphertext and therefore the ciphertext will be different each time, even if the plaintext and the public key are exactly identical. (iconic behaviour, fr fr — randomness is the vibe)
  • The returned ciphertext in to will always be zero padded to exactly RSA_size(rsa) bytes.
  • to and from may overlap.

RSA_private_decrypt() decrypts the flen bytes at from using the private key rsa and stores the plaintext in to. flen should be equal to RSA_size(rsa) but may be smaller, when leading zero bytes are in the ciphertext. Those are not important and may be removed, but RSA_public_encrypt() does not do that. (rude, but okay.)

to must point to a memory section large enough to hold the maximal possible decrypted data (which is equal to RSA_size(rsa) for RSA_NO_PADDING, RSA_size(rsa) - 11 for the PKCS #1 v1.5 based padding modes and RSA_size(rsa) - 42 for RSA_PKCS1_OAEP_PADDING). Size your buffer right or cry. 😭

padding is the padding mode that was used to encrypt the data.

to and from may overlap.

🎀 RETURN VALUES 🎀

RSA_public_encrypt() returns the size of the encrypted data (i.e., RSA_size(rsa)). RSA_private_decrypt() returns the size of the recovered plaintext. A return value of 0 is not an error and means only that the plaintext was empty. Zero is slaying, not failing. ✨

On error, -1 is returned; the error codes can be obtained by ERR_get_error(3).

💀 WARNINGS 💀

Decryption failures in the RSA_PKCS1_PADDING mode leak information which can potentially be used to mount a Bleichenbacher padding oracle attack. This is an inherent weakness in the PKCS #1 v1.5 padding design. Prefer RSA_PKCS1_OAEP_PADDING. This is the WARNING the description was yelling about, bestie. It's giving security nightmare.

In OpenSSL before version 3.2.0, both the return value and the length of returned value could be used to mount the Bleichenbacher attack. Since version 3.2.0, the default provider in OpenSSL does not return an error when padding checks fail. Instead it generates a random message based on used private key and provided ciphertext so that application code doesn't have to implement a side-channel secure error handling. 🙈

Applications that want to be secure against side-channel attacks with providers that don't implement implicit rejection, still need to handle the returned values using side-channel free code. Side-channel free handling of the error stack can be performed using either a pair of unconditional ERR_set_mark(3) and ERR_pop_to_mark(3) calls or by using the ERR_clear_error(3) call.

lowkey this whole section is the reason you read manpages instead of vibing your way through crypto. please handle your errors side-channel free, I'm begging 🫶

🌈 CONFORMING TO 🌈

SSL, PKCS #1 v2.0

👑 SEE ALSO 👑

ERR_get_error(3), RAND_bytes(3), RSA_size(3), EVP_PKEY_decrypt(3), EVP_PKEY_encrypt(3)

⭐ HISTORY ⭐

Both of these functions were deprecated in OpenSSL 3.0. They've been in their "we don't do that anymore" era since then. 💅

Copyright 2000-2024 The OpenSSL Project Authors. All Rights Reserved.

Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliance with the License. You can obtain a copy in the file LICENSE in the source distribution or at <https://www.openssl.org/source/license.html>.

okay that's the whole page, you ate it, no crumbs 🍓✨ this was OpenSSL version 4.0.3, dated 2026-09-29, manual section 3SSL — mwah, take care of your padding modes bestie 💖🎀🦋

📦 where this page comes from