BN_GENERATE_PRIME(3)

🫶 NAME 🫶

okay so bestie, welcome to BN_GENERATE_PRIME(3) — the libcrypto manual page for the whole prime-making / primality-checking friend group: BN_is_prime_ex, BN_is_prime_fasttest_ex, BN_generate_prime_ex, BN_GENCB_call, BN_GENCB_new, BN_GENCB_free, BN_GENCB_set, BN_GENCB_get_arg, and BN_GENCB_set_old. their collective vibe? generate primes and test for primality. periodt 💅

---

✨ SYNOPSIS ✨

library “libcrypto”
#include <openssl/bn.h>

int
BN_is_prime_ex(const BIGNUM *a, int nchecks, BN_CTX *ctx, BN_GENCB *cb);

int
BN_is_prime_fasttest_ex(const BIGNUM *a, int nchecks, BN_CTX *ctx,
    int do_trial_division, BN_GENCB *cb);

int
BN_generate_prime_ex(BIGNUM *ret, int bits, int safe,
    const BIGNUM *modulus, const BIGNUM *remainder, BN_GENCB *cb);

int
BN_GENCB_call(BN_GENCB *cb, int state_code, int serial_number);

BN_GENCB *
BN_GENCB_new(void);

void
BN_GENCB_free(BN_GENCB *cb);

void
BN_GENCB_set(BN_GENCB *cb, int (*cb_fp)(int, int, BN_GENCB *),
    void *cb_arg);

void *
BN_GENCB_get_arg(BN_GENCB *cb);

Deprecated:

void
BN_GENCB_set_old(BN_GENCB *cb, void (*cb_fp)(int, int, void *),
    void *cb_arg);

---

💅 DESCRIPTION 💅

so BN_is_prime_ex() and BN_is_prime_fasttest_ex() are the “is this number a prime?” besties. in LibreSSL they behave identically — they use the Baillie-Pomerance-Selfridge-Wagstaff algorithm plus a bunch of Miller-Rabin rounds based on checks. and the do_trial_division argument? literally ignored. 🙈

now here’s the tea: nobody knows whether any composite number exists that the Baillie-PSW algorithm would misclassify as prime. some people suspect there might be infinitely many such numbers, but not a single one is currently known. what is known is that no such number exists below 2^64. slay for small numbers, i guess 💀

to lower the chance of a composite sneaking past the primality tests in BN_is_prime_fasttest_ex() and BN_is_prime_ex(), a number of rounds of the probabilistic Miller-Rabin test gets performed. if checks is positive, that’s the number of rounds. if it’s zero or the special value BN_prime_checks, a suitable number of rounds is calculated from the bit length of a. iykyk ✨

if NULL is passed for the ctx argument, these functions allocate a BN_CTX object internally when they need one and free it before returning. alternatively — to save the overhead of allocating and freeing that object on every call — the caller can pre-allocate a BN_CTX object and pass it in the ctx argument. efficiency queen 👑

BN_generate_prime_ex() generates a pseudo-random prime number of at least bit length bits and places it in ret. primality of ret is tested internally using BN_is_prime_ex(). consequently, for bits larger than 64, it is theoretically possible that this function might place a composite number into ret; the probability of such an event is unknown but very small. not me being theoretical about it 🫠

the prime may have to fulfill additional requirements for use in Diffie-Hellman key exchange:

• if modulus is not NULL, a prime is generated that fulfills the condition ret % modulus = remainder. if the remainder argument is NULL, 1 is used as the desired remainder.
• if the safe argument is non-zero, a safe prime is generated, that is, (ret - 1)/2 is also prime.

if cb is not NULL, it gets used like this:

• BN_GENCB_call(cb, 0, serial_number) is called after generating a potential prime number.
• the state_code of 1 is reserved for callbacks during primality testing, but LibreSSL performs no such callbacks.
• when safe is non-zero and a safe prime has been found, BN_GENCB_call(cb, 2, serial_number) is called.
• the callers of BN_generate_prime_ex() may call BN_GENCB_call() with other values as described in their respective manual pages; see “SEE ALSO”.

in all cases, the serial_number is the number of candidates that have already been discarded for not being prime; that is, serial_number is 0 for the first candidate and then incremented whenever a new candidate is generated. tracking the flops, fr fr 📉

BN_GENCB_call() calls the callback function held in cb and passes the state_code and the serial_number as arguments. if cb is NULL or does not contain a callback function, no action occurs. no callback, no drama 💅

BN_GENCB_new() allocates a new BN_GENCB object. fresh out the box 🎀

BN_GENCB_free() frees cb. if cb is NULL, no action occurs. clean up after yourself, bestie 🧚

BN_GENCB_set() initialises cb to use the callback function pointer cb_fp and the additional callback argument cb_arg.

the deprecated function BN_GENCB_set_old() initialises cb to use the old-style callback function pointer cb_fp and the additional callback argument cb_arg. it’s giving legacy behavior, handle with care 🫶

---

🎀 RETURN VALUES 🎀

• BN_is_prime_ex() and BN_is_prime_fasttest_ex() return 0 if the number is composite, 1 if it is prime with a very small error probability, or -1 on error.
• BN_generate_prime_ex() returns 1 on success or 0 on error.
• BN_GENCB_call() returns 1 on success, including when cb is NULL or does not contain a callback function, or 0 on error.
• BN_GENCB_new() returns a pointer to the newly allocated BN_GENCB object or NULL if memory allocation fails.
• the callback functions pointed to by the cb_fp arguments are supposed to return 1 on success or 0 on error.
• BN_GENCB_get_arg() returns the cb_arg pointer that was previously stored in cb using BN_GENCB_set() or BN_GENCB_set_old().
• in some cases, error codes can be obtained by ERR_get_error(3).

---

🦋 SEE ALSO 🦋

• BN_new(3)
• DH_generate_parameters(3)
• DSA_generate_parameters_ex(3)
• RSA_generate_key(3)

---

🌸 HISTORY 🌸

BN_generate_prime_ex(), BN_is_prime_ex(), BN_is_prime_fasttest_ex(), BN_GENCB_call(), BN_GENCB_set_old(), and BN_GENCB_set() first appeared in OpenSSL 0.9.8 and have been available since OpenBSD 4.5.

BN_GENCB_new(), BN_GENCB_free(), and BN_GENCB_get_arg() first appeared in OpenSSL 1.1.0 and have been available since OpenBSD 6.3.

okay that’s the whole page, bestie — its footer credits Debian, dated September 1, 2026. go forth and generate primes responsibly 💖✨