RSA_PUBLIC_ENCRYPT(3ossl)

💖 RSA_PUBLIC_ENCRYPT 💖

okay bestie, so this is the RSA_PUBLIC_ENCRYPT(3ossl) page from the OpenSSL fam — it's giving "old money crypto API that's lowkey retired" and honestly? we need to talk about it. 🌸

✨ NAME ✨

RSA_public_encrypt, RSA_private_decrypt — RSA public key cryptography, the OG way 🫶

✨ SYNOPSIS ✨

#include <openssl/rsa.h>

The following functions have been deprecated since OpenSSL 3.0, and can be hidden entirely by defining OPENSSL_API_COMPAT with a suitable version value, see openssl_user_macros(7):

int RSA_public_encrypt(int flen, const unsigned char *from,
                       unsigned char *to, RSA *rsa, int padding);

int RSA_private_decrypt(int flen, const unsigned char *from,
                        unsigned char *to, RSA *rsa, int padding);

✨ DESCRIPTION ✨

Okay so real talk first: both of the functions on this page are deprecated. Not me telling you they're past their prime — that's literally what the docs say. 😭 If you're writing new code, the glow-up replacements are EVP_PKEY_encrypt_init_ex(3), EVP_PKEY_encrypt(3), EVP_PKEY_decrypt_init_ex(3) and EVP_PKEY_decrypt(3). That's the main character energy now, periodt. 👑

---

🌟 RSA_public_encrypt() — the encrypting bestie

So RSA_public_encrypt() takes those flen bytes sitting at from (usually a session key, iykyk) and encrypts them using the public key rsa, dropping the ciphertext into to. And to? It must point to RSA_size(rsa) bytes of memory. No more, no less, don't skimp. 💅

🎀 padding modes (pick one, choose wisely)

padding is one of these:

• RSA_PKCS1_PADDING — PKCS #1 v1.5 padding. Currently the most widely used mode, it's everywhere, it's iconic in a "we all did this" way. BUT it is highly recommended to use RSA_PKCS1_OAEP_PADDING in new applications. SEE WARNING BELOW. 🚨
• RSA_PKCS1_OAEP_PADDING — EME-OAEP as defined in PKCS #1 v2.0 with SHA-1, MGF1 and an empty encoding parameter. This one is recommended for all new applications. slay, she's secure, she's the moment. ✨
• RSA_NO_PADDING — Raw RSA encryption. Only use this to implement cryptographically sound padding modes in the application code itself. Encrypting user data directly with RSA is insecure. fr fr, do not. 💀

🦋 the size rules (write these down)

When encrypting, flen must not be more than:

• RSA_size(rsa) - 11 for the PKCS #1 v1.5 based padding modes
• RSA_size(rsa) - 42 for RSA_PKCS1_OAEP_PADDING
• exactly RSA_size(rsa) for RSA_NO_PADDING

And here's the cute part: when a padding mode other than RSA_NO_PADDING is in use, then RSA_public_encrypt() will include some random bytes into the ciphertext, and therefore the ciphertext will be different each time, even if the plaintext and the public key are exactly identical. Same input, different output, every single time — it's giving unpredictable in the best way. 🌈

The returned ciphertext in to will always be zero padded to exactly RSA_size(rsa) bytes. And yes, to and from may overlap. No cap.

---

🌟 RSA_private_decrypt() — the decrypting bestie

RSA_private_decrypt() decrypts the flen bytes at from using the private key rsa and stores the plaintext in to.

flen should be equal to RSA_size(rsa) but may be smaller, when leading zero bytes are in the ciphertext. Those are not important and may be removed, but RSA_public_encrypt() does not do that. 🙈

to must point to a memory section large enough to hold the maximal possible decrypted data, which is equal to:

• RSA_size(rsa) for RSA_NO_PADDING
• RSA_size(rsa) - 11 for the PKCS #1 v1.5 based padding modes
• RSA_size(rsa) - 42 for RSA_PKCS1_OAEP_PADDING

padding is the padding mode that was used to encrypt the data — so remember what you encrypted with, bestie. 🧚 And yep, to and from may overlap here too.

✨ RETURN VALUES ✨

• RSA_public_encrypt() returns the size of the encrypted data (i.e., RSA_size(rsa)).
• RSA_private_decrypt() returns the size of the recovered plaintext.
• A return value of 0 is not an error and means only that the plaintext was empty. Like, she's fine, she's just empty. 💜
• On error, -1 is returned; the error codes can be obtained by ERR_get_error(3).

✨ WARNINGS ✨

Decryption failures in the RSA_PKCS1_PADDING mode leak information which can potentially be used to mount a Bleichenbacher padding oracle attack. This is an inherent weakness in the PKCS #1 v1.5 padding design. Prefer RSA_PKCS1_OAEP_PADDING.

Okay so this next bit is important, don't scroll past it. 😭

In OpenSSL before version 3.2.0, both the return value and the length of returned value could be used to mount the Bleichenbacher attack. Since version 3.2.0, the default provider in OpenSSL does not return an error when padding checks fail. Instead it generates a random message based on used private key and provided ciphertext so that application code doesn't have to implement a side-channel secure error handling. 🫶

Applications that want to be secure against side-channel attacks with providers that don't implement implicit rejection, still need to handle the returned values using side-channel free code. Side-channel free handling of the error stack can be performed using either a pair of unconditional ERR_set_mark(3) and ERR_pop_to_mark(3) calls or by using the ERR_clear_error(3) call.

✨ CONFORMING TO ✨

SSL, PKCS #1 v2.0 ⭐

✨ SEE ALSO ✨

ERR_get_error(3), RAND_bytes(3), RSA_size(3), EVP_PKEY_decrypt(3), EVP_PKEY_encrypt(3)

✨ HISTORY ✨

Both of these functions were deprecated in OpenSSL 3.0. yeah. it happened. we move. 💅

✨ COPYRIGHT ✨

Copyright 2000-2024 The OpenSSL Project Authors. All Rights Reserved.

Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliance with the License. You can obtain a copy in the file LICENSE in the source distribution or at <https://www.openssl.org/source/license.html>.

---

and that's the whole page bestie!! this one came from OpenSSL 3.5.9, dated 2026-09-29, and it lives under the name RSA_PUBLIC_ENCRYPT(3ossl) 🎀💖✨ mwah, go forth and use OAEP 🦋