RSA_PKEY_CTX_CTRL(3)

💅 RSA_PKEY_CTX_CTRL(3) 💅

okay so bestie, welcome to the libcrypto manual, section 3 — this is the one about RSA private key control operations, and honestly it's giving "backstage pass to the RSA keygen green room" energy. grab a strawberry drink 🍓 and let's get into it.

---

✨ NAME ✨

Here's the whole squad of functions this page covers, no cap:

• RSA_pkey_ctx_ctrl
• EVP_PKEY_CTX_set_rsa_padding
• EVP_PKEY_CTX_get_rsa_padding
• EVP_PKEY_CTX_set_rsa_keygen_bits
• EVP_PKEY_CTX_set_rsa_keygen_pubexp
• EVP_PKEY_CTX_set_rsa_mgf1_md
• EVP_PKEY_CTX_get_rsa_mgf1_md
• EVP_PKEY_CTX_set_rsa_oaep_md
• EVP_PKEY_CTX_get_rsa_oaep_md
• EVP_PKEY_CTX_set0_rsa_oaep_label
• EVP_PKEY_CTX_get0_rsa_oaep_label
• EVP_PKEY_CTX_set_rsa_pss_saltlen
• EVP_PKEY_CTX_get_rsa_pss_saltlen
• EVP_PKEY_CTX_set_rsa_pss_keygen_md
• EVP_PKEY_CTX_set_rsa_pss_keygen_mgf1_md
• EVP_PKEY_CTX_set_rsa_pss_keygen_saltlen

collectively: RSA private key control operations. iconic, honestly. 👑

---

🎀 SYNOPSIS 🎀

library “libcrypto”

#include <openssl/rsa.h>

int
RSA_pkey_ctx_ctrl(EVP_PKEY_CTX *ctx, int optype, int cmd, int p1,
    void *p2);

int
EVP_PKEY_CTX_set_rsa_padding(EVP_PKEY_CTX *ctx, int pad);

int
EVP_PKEY_CTX_get_rsa_padding(EVP_PKEY_CTX *ctx, int *ppad);

int
EVP_PKEY_CTX_set_rsa_keygen_bits(EVP_PKEY_CTX *ctx, int mbits);

int
EVP_PKEY_CTX_set_rsa_keygen_pubexp(EVP_PKEY_CTX *ctx, BIGNUM *pubexp);

int
EVP_PKEY_CTX_set_rsa_mgf1_md(EVP_PKEY_CTX *ctx, const EVP_MD *md);

int
EVP_PKEY_CTX_get_rsa_mgf1_md(EVP_PKEY_CTX *ctx, const EVP_MD **pmd);

int
EVP_PKEY_CTX_set_rsa_oaep_md(EVP_PKEY_CTX *ctx, const EVP_MD *md);

int
EVP_PKEY_CTX_get_rsa_oaep_md(EVP_PKEY_CTX *ctx, const EVP_MD **pmd);

int
EVP_PKEY_CTX_set0_rsa_oaep_label(EVP_PKEY_CTX *ctx,
    unsigned char *label, int len);

int
EVP_PKEY_CTX_get0_rsa_oaep_label(EVP_PKEY_CTX *ctx,
    unsigned char **plabel);

int
EVP_PKEY_CTX_set_rsa_pss_saltlen(EVP_PKEY_CTX *ctx, int len);

int
EVP_PKEY_CTX_get_rsa_pss_saltlen(EVP_PKEY_CTX *ctx, int *plen);

int
EVP_PKEY_CTX_set_rsa_pss_keygen_md(EVP_PKEY_CTX *pctx,
    const EVP_MD *md);

int
EVP_PKEY_CTX_set_rsa_pss_keygen_mgf1_md(EVP_PKEY_CTX *pctx,
    const EVP_MD *md);

int
EVP_PKEY_CTX_set_rsa_pss_keygen_saltlen(EVP_PKEY_CTX *pctx,
    int saltlen);

---

🌸 DESCRIPTION 🌸

So. RSA_pkey_ctx_ctrl() is basically a shallow little wrapper around EVP_PKEY_CTX_ctrl(3) — and it's picky, bestie. it only succeeds if ctx matches either EVP_PKEY_RSA or EVP_PKEY_RSA_PSS. no RSA energy, no entry. 💅

now the plot twist: all the remaining "functions" are implemented as macros. yes, the quotes around "functions" are in the original, the authors were being shady and I respect it.

---

🔥 the padding one

EVP_PKEY_CTX_set_rsa_padding() sets the RSA padding mode for ctx. pad can be:

• RSA_PKCS1_PADDING — PKCS#1 padding
• RSA_NO_PADDING — no padding
• RSA_PKCS1_OAEP_PADDING — OAEP padding (encrypt and decrypt only)
• RSA_X931_PADDING — X9.31 padding (signature operations only)
• RSA_PKCS1_PSS_PADDING — sign and verify only

heads up 🫶: only the last one, RSA_PKCS1_PSS_PADDING, can be used with keys of the type EVP_PKEY_RSA_PSS.

two of these padding modes get weird (affectionate) if EVP_PKEY_CTX_set_signature_md(3) is used:

• PKCS#1 padding: if this macro is called, the plaintext buffer is an actual digest value and gets encapsulated in a DigestInfo structure according to PKCS#1 when signing — and when verifying, that structure is expected (and stripped off). if this control is not used with RSA and PKCS#1 padding, then the supplied data is used directly and is not encapsulated.
• X9.31 padding for RSA: the algorithm identifier byte is added or checked and removed if this control is called. if it is not called, then the first byte of the plaintext buffer is expected to be the algorithm identifier byte.

and EVP_PKEY_CTX_get_rsa_padding() just retrieves the RSA padding mode for ctx. simple, slay. ✨

---

🦋 keygen bits & pubexp

EVP_PKEY_CTX_set_rsa_keygen_bits() sets the RSA key length for RSA or RSA-PSS key generation to mbits. the smallest supported value is 512 bits, and if you don't specify anything, 1024 bits is used. (512 is the floor, do not test it 🙈)

EVP_PKEY_CTX_set_rsa_keygen_pubexp() sets the public exponent value for RSA or RSA-PSS key generation to pubexp. currently it should be an odd integer. and listen — the pubexp pointer is used internally by this function, so it should not be modified or freed after the call. the library is holding onto it, it's theirs now. if this macro is never called, 65537 is used.

---

🧚 MGF1 digest

EVP_PKEY_CTX_set_rsa_mgf1_md() sets the MGF1 digest for RSA padding schemes to md. unless explicitly specified, the signing digest is used. the padding mode must have been set to RSA_PKCS1_OAEP_PADDING or RSA_PKCS1_PSS_PADDING.

😭 also: if the key is of the type EVP_PKEY_RSA_PSS and has usage restrictions, an error occurs if an attempt is made to set the digest to anything other than the restricted value. the key said no and it meant it.

EVP_PKEY_CTX_get_rsa_mgf1_md() retrieves the MGF1 digest for ctx. unless explicitly specified, the signing digest is used, and again the padding mode must have been set to RSA_PKCS1_OAEP_PADDING or RSA_PKCS1_PSS_PADDING.

---

🌈 Optimal asymmetric encryption padding 🌈

the following macros require that the padding mode was set to RSA_PKCS1_OAEP_PADDING. it's a whole prerequisite vibe check, bestie. 🎀

• EVP_PKEY_CTX_set_rsa_oaep_md() — sets the message digest type used in RSA OAEP to md.
• EVP_PKEY_CTX_get_rsa_oaep_md() — gets the message digest type used in RSA OAEP to *pmd.
• EVP_PKEY_CTX_set0_rsa_oaep_label() — sets the RSA OAEP label to label and its length to len. if label is NULL or len is 0, the label is cleared. the library takes ownership of the label, so the caller should not free the original memory pointed to by label. it's not yours anymore, you gave it away 💀
• EVP_PKEY_CTX_get0_rsa_oaep_label() — gets the RSA OAEP label to *plabel. the return value is the label length. the resulting pointer is owned by the library and should not be freed by the caller. hands off, respectfully. 🫶

---

⭐ Probabilistic signature scheme ⭐

these macros require that the padding mode was set to RSA_PKCS1_PSS_PADDING. same deal, different padding.

EVP_PKEY_CTX_set_rsa_pss_saltlen() sets the RSA PSS salt length to len. three special values are supported:

• RSA_PSS_SALTLEN_DIGEST — sets the salt length to the digest length.
• RSA_PSS_SALTLEN_MAX — sets the salt length to the maximum permissible value.
• RSA_PSS_SALTLEN_AUTO — when signing, sets the salt length to the maximum permissible value. when verifying, it causes the salt length to be automatically determined based on the PSS block structure.

if this macro is not called, a salt length value of RSA_PSS_SALTLEN_AUTO is used by default. ✨

💀 warning zone: if the key has usage restrictions and an attempt is made to set the salt length below the minimum value, an error occurs. also, if the key has usage restrictions, RSA_PSS_SALTLEN_AUTO is not supported for verification.

EVP_PKEY_CTX_get_rsa_pss_saltlen() retrieves the RSA PSS salt length for ctx.

---

🧚 PSS keygen restrictions

optional parameter restrictions can be specified when generating a PSS key. and here's the catch: if any restrictions are set using the macros described below, then ALL parameters are restricted. for example, setting a minimum salt length also restricts the digest and MGF1 algorithms. it's all or nothing, main character energy only 👑

if any restrictions are in place, then they are reflected in the corresponding parameters of the public key when (for example) a certificate request is signed.

• EVP_PKEY_CTX_set_rsa_pss_keygen_md() — restricts the digest algorithm the generated key can use to md.
• EVP_PKEY_CTX_set_rsa_pss_keygen_mgf1_md() — restricts the MGF1 algorithm the generated key can use to md.
• EVP_PKEY_CTX_set_rsa_pss_keygen_saltlen() — restricts the minimum salt length to saltlen.

---

👑 RETURN VALUES 👑

these functions return a positive value for success, or 0 or a negative value for failure. and in particular, a return value of -2 indicates the operation is not supported by the public key algorithm. so if you see -2, that's the algorithm going "not my department, bestie" 🙈

---

🌟 SEE ALSO 🌟

go read these next, iykyk:

EVP_DigestInit(3), EVP_PKEY_CTX_ctrl(3), EVP_PKEY_CTX_new(3), EVP_PKEY_decrypt(3), EVP_PKEY_derive(3), EVP_PKEY_encrypt(3), EVP_PKEY_get_default_digest_nid(3), EVP_PKEY_keygen(3), EVP_PKEY_sign(3), EVP_PKEY_verify(3), EVP_PKEY_verify_recover(3)

---

🌈 HISTORY 🌈

a little timeline moment for you, because the lore is real:

• EVP_PKEY_CTX_set_rsa_padding(), EVP_PKEY_CTX_set_rsa_keygen_bits(), EVP_PKEY_CTX_set_rsa_keygen_pubexp(), and EVP_PKEY_CTX_set_rsa_pss_saltlen() first appeared in OpenSSL 1.0.0 and have been available since OpenBSD 4.9.
• EVP_PKEY_CTX_get_rsa_padding(), EVP_PKEY_CTX_set_rsa_mgf1_md(), EVP_PKEY_CTX_get_rsa_mgf1_md(), and EVP_PKEY_CTX_get_rsa_pss_saltlen() first appeared in OpenSSL 1.0.1 and have been available since OpenBSD 5.3.
• EVP_PKEY_CTX_set_rsa_oaep_md(), EVP_PKEY_CTX_get_rsa_oaep_md(), EVP_PKEY_CTX_set0_rsa_oaep_label(), and EVP_PKEY_CTX_get0_rsa_oaep_label() first appeared in OpenSSL 1.0.2 and have been available since OpenBSD 6.7.
• RSA_pkey_ctx_ctrl() first appeared in OpenSSL 1.1.1 and has been available since OpenBSD 6.7.

---

and that's the page, bestie! 🫶 this Debian edition is dated June 8, 2025 — so if you're reading it later, just know the RSA_PKEY_CTX_CTRL(3) girlies have been holding it down since forever. mwah 💖✨🎀