RSA_SET_METHOD(3SSL)

💖 RSA_SET_METHOD — ok bestie, come learn how OpenSSL picks its RSA vibe 💖

So this is a page from the OpenSSL manual, section 3SSL (yep, the 3SSL is literally part of the name, that's the manual section, slay). It's all about choosing which method does your RSA math. Sit down, grab a strawberry 🍓, and let's get into it.

✨ NAME ✨

RSA_set_default_method, RSA_get_default_method, RSA_set_method, RSA_get_method, RSA_PKCS1_OpenSSL, RSA_flags, RSA_new_method — these are the little guys that let you select the RSA method. That's the whole job description.

✨ SYNOPSIS ✨

First things first — pull in the header:

#include <openssl/rsa.h>

Okay so important lowkey lore: every single one of these functions has been deprecated since OpenSSL 3.0. You can hide them entirely by defining OPENSSL_API_COMPAT with a suitable version value — check openssl_user_macros(7) for how that works. It's giving "we still ship it but we're side-eyeing it" 💀

void RSA_set_default_method(const RSA_METHOD *meth);

const RSA_METHOD *RSA_get_default_method(void);

int RSA_set_method(RSA *rsa, const RSA_METHOD *meth);

const RSA_METHOD *RSA_get_method(const RSA *rsa);

const RSA_METHOD *RSA_PKCS1_OpenSSL(void);

int RSA_flags(const RSA *rsa);

RSA *RSA_new_method(ENGINE *engine);

✨ DESCRIPTION ✨

OK real talk, bestie: all of the functions described on this page are deprecated. Applications should instead use the OSSL_PROVIDER APIs. No cap. That's the official word, I'm just the messenger 🫶

So. An RSA_METHOD is the thing that specifies which functions OpenSSL uses for RSA operations. And here's the fun part — by swapping the method, you can slot in alternative implementations, like hardware accelerators. Main character energy for your crypto hardware 💅

At the very start, the default RSA_METHOD is the OpenSSL internal implementation, the one you get back from RSA_PKCS1_OpenSSL().

• RSA_set_default_method() — makes meth the default method for every RSA structure created later. ⚠️ Big NB incoming: this function is not thread-safe and should not be called at the same time as other OpenSSL functions. Fr fr, don't do it.
• RSA_get_default_method() — hands you a pointer to the current default RSA_METHOD. Simple. Iconic. Does exactly what it says.
• RSA_set_method() — selects meth to perform all operations using the key rsa. This replaces the RSA_METHOD used by that RSA key. Real talk though: it's possible to have RSA keys that only work with certain RSA_METHOD implementations, and in those cases trying to change the RSA_METHOD for the key can have unexpected results. 😭
• RSA_get_method() — returns a pointer to the RSA_METHOD that rsa is using. Heads up: the return value can only be guaranteed to be valid as long as the RSA key itself is valid and nobody changed its implementation via RSA_set_method(). So don't hold onto it forever, bestie 🙈
• RSA_flags() — returns the flags that are set for rsa's current RSA_METHOD. See the BUGS section, because it's a whole thing (spoiler: it's messy).
• RSA_new_method() — allocates and initializes an RSA structure. Here engine must be NULL, and the RSA_METHOD controlled by RSA_set_default_method() is what gets used.

And then, because redundancy said "let me speak":

• RSA_flags() — returns the flags that are set for rsa's current method. (Yes, again. It's literally in here twice. I don't make the rules 💀)
• RSA_new_method() — allocates and initializes an RSA structure so that method will be used for the RSA operations. If method is NULL, the default method is used.

🦋 THE RSA_METHOD STRUCTURE 🦋

Okay here's the whole struct, verbatim, every field, every comment, no edits, because details matter:

typedef struct rsa_meth_st
{
    /* name of the implementation */
    const char *name;

    /* encrypt */
    int (*rsa_pub_enc)(int flen, unsigned char *from,
                       unsigned char *to, RSA *rsa, int padding);

    /* verify arbitrary data */
    int (*rsa_pub_dec)(int flen, unsigned char *from,
                       unsigned char *to, RSA *rsa, int padding);

    /* sign arbitrary data */
    int (*rsa_priv_enc)(int flen, unsigned char *from,
                        unsigned char *to, RSA *rsa, int padding);

    /* decrypt */
    int (*rsa_priv_dec)(int flen, unsigned char *from,
                        unsigned char *to, RSA *rsa, int padding);

    /* compute r0 = r0 ^ I mod rsa->n (May be NULL for some implementations) */
    int (*rsa_mod_exp)(BIGNUM *r0, BIGNUM *I, RSA *rsa);

    /* compute r = a ^ p mod m (May be NULL for some implementations) */
    int (*bn_mod_exp)(BIGNUM *r, BIGNUM *a, const BIGNUM *p,
                      const BIGNUM *m, BN_CTX *ctx, BN_MONT_CTX *m_ctx);

    /* called at RSA_new */
    int (*init)(RSA *rsa);

    /* called at RSA_free */
    int (*finish)(RSA *rsa);

    /*
     * RSA_FLAG_EXT_PKEY        - rsa_mod_exp is called for private key
     *                            operations, even if p,q,dmp1,dmq1,iqmp
     *                            are NULL
     * RSA_METHOD_FLAG_NO_CHECK - don't check pub/private match
     */
    int flags;

    char *app_data; /* ?? */

    int (*rsa_sign)(int type,
                    const unsigned char *m, unsigned int m_length,
                    unsigned char *sigret, unsigned int *siglen, const RSA *rsa);
    int (*rsa_verify)(int dtype,
                      const unsigned char *m, unsigned int m_length,
                      const unsigned char *sigbuf, unsigned int siglen,
                      const RSA *rsa);
    /* keygen. If NULL built-in RSA key generation will be used */
    int (*rsa_keygen)(RSA *rsa, int bits, BIGNUM *e, BN_GENCB *cb);

} RSA_METHOD;

Note the char *app_data; /* ?? */ — the OpenSSL authors left a literal ?? in there and honestly? Respect. Same energy as me pretending I know what's going on 🧚

✨ RETURN VALUES ✨

• RSA_PKCS1_OpenSSL(), RSA_PKCS1_null_method(), RSA_get_default_method() and RSA_get_method() — all return pointers to their respective RSA_METHODs. Cute.
• RSA_set_default_method() — returns no value. Just vibes.
• RSA_set_method() — returns 1 for success. And get this: it always succeeds. No failure arc, ever.
• RSA_new_method() — returns NULL and sets an error code (which you can grab with ERR_get_error(3)) if the allocation fails. Otherwise it returns a pointer to the newly allocated structure. 🎀

💀 BUGS 💀

Okay this is the messy part and I need you to pay attention.

The behaviour of RSA_flags() is a mis-feature that is left as-is for now, to avoid creating compatibility problems. Yes, they said "mis-feature." Out loud. In the manual.

Here's the tea: RSA functionality — stuff like the encryption functions — is controlled by the flags value in the RSA key itself, not by the flags value in the RSA_METHOD attached to the RSA key. And which one does this function return? The RSA_METHOD one. 🙃

So if the flags element of an RSA key is changed, the changes will be honoured by RSA functionality, but will not be reflected in the return value of RSA_flags(). In effect, RSA_flags() behaves more like an RSA_default_flags() function — which, and I quote, does not currently exist. So you're just out here with a function that lies to you a little. It's giving gaslight, gatekeep, girlboss ✨

👑 SEE ALSO 👑

• RSA_new(3)

🌸 HISTORY 🌸

• All of these functions were deprecated in OpenSSL 3.0.
• RSA_null_method(), which was a partial attempt to avoid patent issues, was replaced to always return NULL in OpenSSL 1.1.1. RIP to that era 🕯️

💜 COPYRIGHT 💜

Copyright 2000-2026 The OpenSSL Project Authors. All Rights Reserved.

Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliance with the License. You can obtain a copy in the file LICENSE in the source distribution or at <https://www.openssl.org/source/license.html>.

---

Okay and that's the whole page, bestie! This one shipped in OpenSSL 4.0.3, dated 2026-09-29, and it lives in the OpenSSL manual as RSA_SET_METHOD(3SSL). Go forth and don't call the non-thread-safe one from two threads at once, I'm begging you 🫶💖✨