RSA_SET_METHOD(3ossl)

📦
version 3.5.9 · 8 versions
, in fedora 43, fedora 44

🌸 NAME 🌸

okay bestie, welcome to RSA_SET_METHOD(3ossl) — the page about picking which RSA implementation OpenSSL is gonna use 💅

The squad on this page: RSA_set_default_method, RSA_get_default_method, RSA_set_method, RSA_get_method, RSA_PKCS1_OpenSSL, RSA_flags, and RSA_new_method. Together they're the "select RSA method" crew 🎀

✨ SYNOPSIS ✨

First things first, you gotta include the header:

#include <openssl/rsa.h>
⚠️ Real talk: the functions below have been deprecated since OpenSSL 3.0, and you can hide them entirely by defining OPENSSL_API_COMPAT with a suitable version value — see openssl_user_macros(7) for the deets 🫶
void RSA_set_default_method(const RSA_METHOD *meth);

const RSA_METHOD *RSA_get_default_method(void);

int RSA_set_method(RSA *rsa, const RSA_METHOD *meth);

const RSA_METHOD *RSA_get_method(const RSA *rsa);

const RSA_METHOD *RSA_PKCS1_OpenSSL(void);

int RSA_flags(const RSA *rsa);

RSA *RSA_new_method(ENGINE *engine);

💖 DESCRIPTION 💖

Okay so, heads up, and I mean this with love: every single function described on this page is deprecated. Like, all of them. If you're writing an app in this day and age you should be using the OSSL_PROVIDER APIs instead. Periodt. 💀

So what even is an RSA_METHOD? It's a lil struct that specifies the functions OpenSSL uses for RSA operations. And the way that's powerful? By swapping out the method, you can plug in alternative implementations — think hardware accelerators, the whole glow-up ✨

🚨 IMPORTANT: go read the NOTES section for important info about how these RSA API functions get affected by ENGINE API calls. Don't skip it. I'm serious. 🙈

At first, the default RSA_METHOD is the OpenSSL internal implementation, which is what RSA_PKCS1_OpenSSL() hands back 🎀

RSA_set_default_method() makes meth the default method for all RSA structures created later. NB bestie: this is only true whilst no ENGINE has been set as a default for RSA, which is exactly why this function is no longer recommended. Also?? It's not thread-safe — do not call it at the same time as other OpenSSL functions, I am begging 😭

RSA_get_default_method() returns a pointer to the current default RSA_METHOD. But how meaningful that result even is depends on whether the ENGINE API is being used, so… yeah, also no longer recommended. 🙈

RSA_set_method() selects meth to perform all operations using the key rsa. This replaces the RSA_METHOD used by the RSA key, and if the previous method was supplied by an ENGINE, the handle to that ENGINE gets released during the change. And here's the plot twist 👑: it's possible to have RSA keys that only work with certain RSA_METHOD implementations (like, say, from an ENGINE module that supports embedded hardware-protected keys), and in those cases trying to change the RSA_METHOD for the key can have unexpected results. So, like, be careful, okay?

RSA_get_method() returns a pointer to the RSA_METHOD being used by rsa. That method may or may not be supplied by an ENGINE implementation — but if it is, the return value can only be guaranteed valid as long as the RSA key itself is valid and doesn't have its implementation changed by RSA_set_method(). Fickle much? 💅

RSA_flags() returns the flags that are set for rsa's current RSA_METHOD. See the BUGS section. (Yes, really. Go look. It's a whole thing. 😭)

RSA_new_method() allocates and initializes an RSA structure so that engine will be used for the RSA operations. If engine is NULL, the default ENGINE for RSA operations is used — and if no default ENGINE is set, then the RSA_METHOD controlled by RSA_set_default_method() is used.

And then, bestie… the page says it again: 🌸

RSA_flags() returns the flags that are set for rsa's current method.

RSA_new_method() allocates and initializes an RSA structure so that method will be used for the RSA operations. If method is NULL, the default method is used.

(Yes, it genuinely repeats itself. Not me editing it out — that's just what the page says, and we keep the lore intact. 🫶)

🦋 THE RSA_METHOD STRUCTURE 🦋

Here's the whole thing, verbatim, because she is iconic and you deserve to see every field:

typedef struct rsa_meth_st
{
    /* name of the implementation */
    const char *name;

    /* encrypt */
    int (*rsa_pub_enc)(int flen, unsigned char *from,
                       unsigned char *to, RSA *rsa, int padding);

    /* verify arbitrary data */
    int (*rsa_pub_dec)(int flen, unsigned char *from,
                       unsigned char *to, RSA *rsa, int padding);

    /* sign arbitrary data */
    int (*rsa_priv_enc)(int flen, unsigned char *from,
                        unsigned char *to, RSA *rsa, int padding);

    /* decrypt */
    int (*rsa_priv_dec)(int flen, unsigned char *from,
                        unsigned char *to, RSA *rsa, int padding);

    /* compute r0 = r0 ^ I mod rsa->n (May be NULL for some implementations) */
    int (*rsa_mod_exp)(BIGNUM *r0, BIGNUM *I, RSA *rsa);

    /* compute r = a ^ p mod m (May be NULL for some implementations) */
    int (*bn_mod_exp)(BIGNUM *r, BIGNUM *a, const BIGNUM *p,
                      const BIGNUM *m, BN_CTX *ctx, BN_MONT_CTX *m_ctx);

    /* called at RSA_new */
    int (*init)(RSA *rsa);

    /* called at RSA_free */
    int (*finish)(RSA *rsa);

    /*
     * RSA_FLAG_EXT_PKEY        - rsa_mod_exp is called for private key
     *                            operations, even if p,q,dmp1,dmq1,iqmp
     *                            are NULL
     * RSA_METHOD_FLAG_NO_CHECK - don't check pub/private match
     */
    int flags;

    char *app_data; /* ?? */

    int (*rsa_sign)(int type,
                    const unsigned char *m, unsigned int m_length,
                    unsigned char *sigret, unsigned int *siglen, const RSA *rsa);
    int (*rsa_verify)(int dtype,
                      const unsigned char *m, unsigned int m_length,
                      const unsigned char *sigbuf, unsigned int siglen,
                      const RSA *rsa);
    /* keygen. If NULL built-in RSA key generation will be used */
    int (*rsa_keygen)(RSA *rsa, int bits, BIGNUM *e, BN_GENCB *cb);

} RSA_METHOD;

💜 RETURN VALUES 💜

  • RSA_PKCS1_OpenSSL(), RSA_PKCS1_null_method(), RSA_get_default_method() and RSA_get_method() — return pointers to the respective RSA_METHODs 🎀
  • RSA_set_default_method() — returns no value. Nothing. Nada. 💅
  • RSA_set_method() — returns 1 for success. And get this: it always succeeds. Slay, I guess. 👑
  • RSA_new_method() — returns NULL and sets an error code (which you can grab with ERR_get_error(3)) if the allocation fails. Otherwise it returns a pointer to the newly allocated structure ✨

💀 BUGS 💀

The behaviour of RSA_flags() is a mis-feature that's been left as-is for now, purely to avoid creating compatibility problems. 🙈

So here's the tea: RSA functionality — like, say, the encryption functions — is controlled by the flags value in the RSA key itself, not by the flags value in the RSA_METHOD attached to the RSA key (which is, unfortunately, what this function returns). So if you change the flags element of an RSA key, the changes will be honoured by RSA functionality… but they will not be reflected in the return value of RSA_flags().

In effect, RSA_flags() behaves more like an RSA_default_flags() function — which, bestie, does not currently exist. 😭

👑 SEE ALSO 👑

🌈 HISTORY 🌈

  • All of these functions were deprecated in OpenSSL 3.0.
  • RSA_null_method(), which was a partial attempt to avoid patent issues, was replaced to always return NULL in OpenSSL 1.1.1.

Copyright 2000-2026 The OpenSSL Project Authors. All Rights Reserved.

Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliance with the License. You can obtain a copy in the file LICENSE in the source distribution or at <https://www.openssl.org/source/license.html>.

and that's the whole page, bestie!! 🫶 this one shipped with OpenSSL 3.5.9, dated 2026-09-29 — so if you're reading this from the future, hiii, hope the deprecation warnings are treating you kindly 💖✨🌸 mwah 💅

📦 where this page comes from