RSA_PUBLIC_ENCRYPT(3)

📦 version 4.4.0, in alpine 3.24, arch rolling

✨ RSA_PUBLIC_ENCRYPT(3) ✨

Okay bestie, so this is from the libcrypto manual (you know, the crypto library that's literally everywhere). It's all about RSA public key cryptography, and I'm gonna walk you through it like we're studying together in the group chat. No cap, this stuff is actually iconic once you get it. 💅

🌸 NAME 🌸

RSA_public_encrypt, RSA_private_decrypt, EVP_PKEY_encrypt_old, EVP_PKEY_decrypt_old — RSA public key cryptography, periodt.

🎀 SYNOPSIS 🎀

First things first, you gotta include the library and the header, okay?

library “libcrypto”
#include <openssl/rsa.h>

Then here are the function signatures, literally copy-paste these vibes:

int
RSA_public_encrypt(int flen, const unsigned char *from,
    unsigned char *to, RSA *rsa, int padding);

int
RSA_private_decrypt(int flen, const unsigned char *from,
    unsigned char *to, RSA *rsa, int padding);

And if you're using the EVP layer (which is kinda the newer vibe), you need this header too:

#include <openssl/evp.h>

Then these deprecated cuties:

int
EVP_PKEY_encrypt_old(unsigned char *to, const unsigned char *from,
    int flen, EVP_PKEY *pkey);

int
EVP_PKEY_decrypt_old(unsigned char *to, const unsigned char *from,
    int flen, EVP_PKEY *pkey);

🦋 DESCRIPTION 🦋

So RSA_public_encrypt() does exactly what it says on the tin: it encrypts the flen bytes at from (usually a session key, iykyk) using the public key rsa, and stores the ciphertext in to. And listen—to must point to RSA_size(rsa) bytes of memory. Don't skimp on that, bestie. 😭

Now, padding is the vibe check here. It denotes one of these modes:

  • RSA_PKCS1_PADDING — PKCS #1 v1.5 padding. This is currently the most widely used mode, fr fr. It's giving "everyone's doing it."
  • RSA_PKCS1_OAEP_PADDING — EME-OAEP as defined in PKCS #1 v2.0 with SHA-1, MGF1 and an empty encoding parameter. This mode is recommended for all new applications. Periodt. If you're building something new, this is the one. 👑
  • RSA_NO_PADDING — Raw RSA encryption. Okay but hear me: this mode should only be used to implement cryptographically sound padding modes in the application code. Encrypting user data directly with RSA is insecure. Like, don't do it. I'm serious. 🚩

Also, flen has rules, and you gotta respect them:

  • For the PKCS #1 v1.5 based padding modes: flen must be less than RSA_size(rsa) - 11
  • For RSA_PKCS1_OAEP_PADDING: flen must be less than RSA_size(rsa) - 41
  • For RSA_NO_PADDING: flen must be exactly RSA_size(rsa)

No exceptions, bestie. The math is mathing.

Now RSA_private_decrypt() is the reverse vibe: it decrypts the flen bytes at from using the private key rsa and stores the plaintext in to. And to must point to a memory section large enough to hold the decrypted data (which is smaller than RSA_size(rsa)—so don't panic, it's not huge). The padding here is the padding mode that was used to encrypt the data. Match it up, okay? 💖

And then we have the deprecated wrappers. They're still here but like, they're old. EVP_PKEY_encrypt_old() is a deprecated wrapper around RSA_public_encrypt() that uses the RSA public key stored in pkey and RSA_PKCS1_PADDING. Same energy for EVP_PKEY_decrypt_old()—it's a deprecated wrapper around RSA_private_decrypt() that uses the RSA private key stored in pkey and RSA_PKCS1_PADDING. They ate, but they're retired. 🙈

🌈 RETURN VALUES 🌈

Okay so here's what you get back:

  • RSA_public_encrypt() and EVP_PKEY_encrypt_old() return the size of the encrypted data (i.e. RSA_size(rsa)). Slay.
  • RSA_private_decrypt() and EVP_PKEY_decrypt_old() return the size of the recovered plaintext. Iconic.
  • On error, -1 is returned. And you can get the error codes by calling ERR_get_error(3). Bestie, always check your errors. 🫶

And one special little detail: in addition to the return values documented above, EVP_PKEY_encrypt_old() may return 0 if the EVP_PKEY_id(3) of pkey is not EVP_PKEY_RSA. So if you get a zero, check your key type, okay? It's giving "not all keys are RSA" energy. 💀

👑 SEE ALSO 👑

Like, go read these too, they're all connected:

⭐ STANDARDS ⭐

SSL, PKCS #1 v2.0. That's the standard tea. ☕

🧚 HISTORY 🧚

RSA_public_encrypt() and RSA_private_decrypt() appeared in SSLeay 0.4 or earlier and have been available since OpenBSD 2.4. So they're basically boomers but in a cool way. 💜

EVP_PKEY_encrypt() and EVP_PKEY_decrypt() first appeared in SSLeay 0.9.0 and have been available since OpenBSD 2.4. They were renamed to EVP_PKEY_encrypt_old() and EVP_PKEY_decrypt_old() in OpenSSL 1.0.0 and OpenBSD 4.9. A glow-up and a rebrand, honestly. ✨

RSA_NO_PADDING is available since SSLeay 0.9.0. OAEP was added in OpenSSL 0.9.2b. So OAEP is the newer vibe, which checks out with the whole "recommended for all new applications" thing. 🌸

💀 BUGS 💀

Okay, this is the serious part, so listen up:

Decryption failures in the RSA_PKCS1_PADDING mode leak information which can potentially be used to mount a Bleichenbacher padding oracle attack. This is an inherent weakness in the PKCS #1 v1.5 padding design. Prefer RSA_PKCS1_OAEP_PADDING.

Like, that's not a vibe. That's a whole security vulnerability. So if you're doing new stuff, use OAEP. Protect your data, bestie. 🔥

That's the whole page, no cap. Written by the Debian crew, dated June 8, 2025, and sourced from RSA_PUBLIC_ENCRYPT(3). Go forth and encrypt responsibly! 💖✨🎀

📦 where this page comes from