RSA_SET_METHOD(3)

💅 RSA_SET_METHOD(3) — ok so this is the RSA method selection page, bestie

hey so this is from the libcrypto library (that's the manual name, iykyk) and it's all about picking which RSA implementation your keys use 🎀

---

✨ NAME ✨

RSA_set_default_method, RSA_get_default_method, RSA_set_method, RSA_get_method, RSA_PKCS1_SSLeay, RSA_flags, RSA_new_method — literally all of these are about selecting the RSA method. that's the whole vibe. 💅

---

🌸 SYNOPSIS 🌸

so you're pulling from libcrypto, and you include this:

#include <openssl/rsa.h>

then the functions go like this, no cap:

void
RSA_set_default_method(const RSA_METHOD *meth);

const RSA_METHOD *
RSA_get_default_method(void);

int
RSA_set_method(RSA *rsa, const RSA_METHOD *meth);

const RSA_METHOD *
RSA_get_method(const RSA *rsa);

const RSA_METHOD *
RSA_PKCS1_SSLeay(void);

int
RSA_flags(const RSA *rsa);

RSA *
RSA_new_method(ENGINE *engine);

---

🦋 DESCRIPTION 🦋

ok so picture this: an RSA_METHOD object is basically a little bundle of function pointers — it holds pointers to all the functions used for RSA operations. by default you get the internal implementation that RSA_PKCS1_SSLeay() hands you. but bestie, the whole point is that you can pick another method and get alternative implementations, like hardware accelerators. slay. 🔥

• RSA_set_default_method() — picks meth as the default method for every single RSA structure created after this call. future keys only, periodt.
• RSA_get_default_method() — returns a pointer to whatever the current default method is. that's it, that's the function.
• RSA_set_method() — picks meth to perform all operations using the key rsa. this replaces the previous RSA_METHOD that the RSA key was using, and it calls the finish function (the one you set up with RSA_meth_set_finish(3), if any). and if meth has an init function set up with RSA_meth_set_init(3), that function gets called right before RSA_set_method() returns. ✨
• RSA_get_method() — returns a pointer to the RSA_METHOD that rsa is currently using.

⚠️ okay so this is important: it's totally possible to have RSA keys that only work with certain RSA_METHOD implementations. so if you try to switch the RSA_METHOD on one of those keys, the results can be unexpected. not me warning you twice 💀

• RSA_flags() — and here's the tea: this function is misleadingly named 🙈 what it actually returns is the flags set for the current RSA_METHOD of rsa — not the flags of rsa itself. if you wanna test the flags used by rsa itself, use RSA_test_flags(3) instead. more on this mess in the “BUGS” section below, iykyk.
• RSA_new_method() — allocates and initializes an RSA structure. plot twist: the engine argument is ignored, and it just uses the default method controlled by RSA_set_default_method(). 💫

and one more thing — the initial flags get copied from the RSA_METHOD object that's being used, and they will not be affected by later changes to that object. they can however be modified by the optional init function, which may have been set up with RSA_meth_set_init(3) and which is called just before RSA_new_method() returns. 🫶

---

👑 RETURN VALUES 👑

• RSA_PKCS1_SSLeay(), RSA_get_default_method(), and RSA_get_method() — return pointers to the respective RSA_METHOD. simple.
• RSA_set_method() — returns 1 on success or 0 on failure. though fr fr, currently it cannot fail. 😌
• RSA_new_method() — returns NULL and sets an error code (grab it with ERR_get_error(3)) if the allocation fails. otherwise it returns a pointer to the newly allocated structure. 🎀

---

🌈 SEE ALSO 🌈

RSA_meth_new(3), RSA_new(3)

go read those next bestie, they're the supporting cast 💜

---

⭐ HISTORY ⭐

ok so the lore:

• RSA_set_default_method(), RSA_PKCS1_SSLeay(), and RSA_new_method() first appeared in SSLeay 0.8.0.
• RSA_flags() first appeared in SSLeay 0.9.0.
• all of these have been available since OpenBSD 2.4. iconic longevity honestly 👑

then later:

• RSA_get_default_method(), RSA_set_method(), and RSA_get_method(), plus the rsa_sign and rsa_verify components of RSA_METHOD, first appeared in OpenSSL 0.9.4 and have been available since OpenBSD 2.6.

---

💀 BUGS 💀

so the behaviour of RSA_flags() is a misfeature — it's been left exactly as-is for now, purely to avoid creating compatibility problems. 😭

here's the situation: RSA functionality (stuff like the encryption functions) is controlled by the flags value in the RSA key itself, not by the flags value in the RSA_METHOD attached to the RSA key — and that method flags value is what this function returns. so if the flags element of an RSA key gets changed, RSA functionality will honour those changes, but the return value of RSA_flags() will not reflect them. 💅

in effect, RSA_flags() behaves more like a hypothetical RSA_default_flags() function — which, as of right now, does not exist. and that's the whole mess, no notes. 🙈

---

and that's the page, bestie 🎀✨ this one came from the Debian build, dated June 8, 2025 — go off and pick your RSA methods wisely, mwah 💖🦋