RSA_PUBLIC_ENCRYPT(3)

💖 RSA_PUBLIC_ENCRYPT(3) 💖

okay bestie, so this is the RSA_PUBLIC_ENCRYPT(3) page from the Library Functions Manual and it's literally all about RSA public key cryptography — we're talking RSA_public_encrypt, RSA_private_decrypt, EVP_PKEY_encrypt_old and EVP_PKEY_decrypt_old. grab a snack, we're doing crypto 🎀

---

✨ NAME ✨

RSA_public_encrypt, RSA_private_decrypt, EVP_PKEY_encrypt_old, EVP_PKEY_decrypt_old — RSA public key cryptography, the whole squad 🫶

---

🌸 SYNOPSIS 🌸

first, the RSA girlies — pull them in with:

#include <openssl/rsa.h>

int
RSA_public_encrypt(int flen, const unsigned char *from,
    unsigned char *to, RSA *rsa, int padding);

int
RSA_private_decrypt(int flen, const unsigned char *from,
    unsigned char *to, RSA *rsa, int padding);

and then the EVP versions, different header, don't @ me:

#include <openssl/evp.h>

int
EVP_PKEY_encrypt_old(unsigned char *to, const unsigned char *from,
    int flen, EVP_PKEY *pkey);

int
EVP_PKEY_decrypt_old(unsigned char *to, const unsigned char *from,
    int flen, EVP_PKEY *pkey);

---

💅 DESCRIPTION 💅

the encrypting side 🦋

RSA_public_encrypt() takes those flen bytes sitting at from (usually a session key, iykyk) and encrypts them using the public key rsa, then drops the ciphertext in to. and to? it must point to RSA_size(rsa) bytes of memory. no cap, that's the vibe check. ✨

padding is one of these modes, and honestly each one has main character energy of its own:

• RSA_PKCS1_PADDING — PKCS #1 v1.5 padding. currently the most widely used mode, the popular girl of padding 💖
• RSA_PKCS1_OAEP_PADDING — EME-OAEP as defined in PKCS #1 v2.0, with SHA-1, MGF1 and an empty encoding parameter. this one is recommended for all new applications 👑
• RSA_NO_PADDING — raw RSA encryption 💀 this mode should only be used to implement cryptographically sound padding modes inside your own application code. encrypting user data directly with RSA is insecure, fr fr, do not do it.

now the size rules, and yes they matter, memorize them:

flen must be less than RSA_size(rsa) - 11 for the PKCS #1 v1.5 based padding modes, less than RSA_size(rsa) - 41 for RSA_PKCS1_OAEP_PADDING, and exactly RSA_size(rsa) for RSA_NO_PADDING. that's it, that's the math 🧮

the decrypting side 🌈

RSA_private_decrypt() decrypts the flen bytes at from using the private key rsa and stores the plaintext in to. and to here needs to point to a memory section big enough to hold the decrypted data — which, fun fact, is smaller than RSA_size(rsa). padding is whatever padding mode was used to encrypt the data in the first place, so keep track bestie 😭

the deprecated wrappers 🙈

• EVP_PKEY_encrypt_old() — a deprecated wrapper around RSA_public_encrypt() that uses the RSA public key stored in pkey and RSA_PKCS1_PADDING.
• EVP_PKEY_decrypt_old() — a deprecated wrapper around RSA_private_decrypt() that uses the RSA private key stored in pkey and RSA_PKCS1_PADDING.

---

⭐ RETURN VALUES ⭐

• RSA_public_encrypt() and EVP_PKEY_encrypt_old() return the size of the encrypted data — i.e. RSA_size(rsa). iconic.
• RSA_private_decrypt() and EVP_PKEY_decrypt_old() return the size of the recovered plaintext.

on error? -1 comes back 😭 and you can grab the error codes with ERR_get_error(3).

and there's a bonus rule on top of everything above: EVP_PKEY_encrypt_old() may also return 0 if the EVP_PKEY_id(3) of pkey is not EVP_PKEY_RSA. rude but fair 💀

---

🦋 SEE ALSO 🦋

EVP_PKEY_decrypt(3), EVP_PKEY_encrypt(3), RSA_meth_set_priv_dec(3), RSA_new(3), RSA_size(3) — go read these next, they're the supporting cast 🫶

---

🌟 STANDARDS 🌟

SSL, PKCS #1 v2.0 🎀

---

🧚 HISTORY 🧚

RSA_public_encrypt() and RSA_private_decrypt() showed up in SSLeay 0.4 or earlier and have been available since OpenBSD 2.4. veterans, honestly 👑

EVP_PKEY_encrypt() and EVP_PKEY_decrypt() first appeared in SSLeay 0.9.0 and have been available since OpenBSD 2.4. then they got renamed to EVP_PKEY_encrypt_old() and EVP_PKEY_decrypt_old() in OpenSSL 1.0.0 and OpenBSD 4.9. glow-up via rebranding ✨

RSA_NO_PADDING is available since SSLeay 0.9.0. and OAEP was added in OpenSSL 0.9.2b 🌸

---

💀 BUGS 💀

okay this part is serious, listen up:

decryption failures in the RSA_PKCS1_PADDING mode leak information which can potentially be used to mount a Bleichenbacher padding oracle attack. this is an inherent weakness in the PKCS #1 v1.5 padding design. prefer RSA_PKCS1_OAEP_PADDING. 🔥

like, actually prefer it. for real. 🫶

---

and that's the page bestie! 💜 this one came from Debian, dated September 10, 2023 — RSA_PUBLIC_ENCRYPT(3), signing off 🎀✨