EVP_PKEY_ENCAPSULATE(3ossl)

💖 EVP_PKEY_ENCAPSULATE(3ossl) 💖

okay bestie, welcome to the OpenSSL manual, the EVP_PKEY_ENCAPSULATE era. this is the one about key encapsulation using a KEM algorithm with a public key. we're so back.

---

✨ NAME ✨

EVP_PKEY_encapsulate_init, EVP_PKEY_auth_encapsulate_init, EVP_PKEY_encapsulate — key encapsulation using a KEM algorithm with a public key. yes that's the whole vibe. 🔐

---

🌸 SYNOPSIS 🌸

#include <openssl/evp.h>

int EVP_PKEY_encapsulate_init(EVP_PKEY_CTX *ctx, const OSSL_PARAM params[]);
int EVP_PKEY_auth_encapsulate_init(EVP_PKEY_CTX *ctx, EVP_PKEY *authpriv,
                                  const OSSL_PARAM params[]);
int EVP_PKEY_encapsulate(EVP_PKEY_CTX *ctx,
                         unsigned char *wrappedkey, size_t *wrappedkeylen,
                         unsigned char *genkey, size_t *genkeylen);

---

🦋 DESCRIPTION 🦋

so. EVP_PKEY_encapsulate_init() sets up a public key algorithm context ctx for an encapsulation operation, and then it sets the params on that context the exact same way calling EVP_PKEY_CTX_set_params(3) would. little heads up: ctx is usually produced using EVP_PKEY_CTX_new_from_pkey(3), where you specify the public key you wanna use. slay.

EVP_PKEY_auth_encapsulate_init() is basically the same thing as EVP_PKEY_encapsulate_init() but it also passes an authpriv authentication private key that gets used during encapsulation. extra key, extra ~flavor~.

then EVP_PKEY_encapsulate() actually does the public key encapsulation operation using ctx. the shared secret gets written to genkey, and you can use it as input for key derivation, typically for various symmetric algorithms. its size gets written to genkeylen, which must be initialised to the size of the provided buffer. no cap.

the ciphertext written to wrappedkey is an encapsulated form, and it's expected to be usable only by whoever holds the private key corresponding to the public key associated with ctx. this ciphertext then gets communicated to the private-key holder, who can use EVP_PKEY_decapsulate(3) to securely recover that same shared secret. it's giving handoff. 🤝

now the buffer-length lore:

• if wrappedkey is NULL, then the maximum size of the output buffer is written to the *wrappedkeylen parameter — unless wrappedkeylen is NULL, in which case the maximum size of the generated key buffer is written to *genkeylen instead, unless genkeylen is NULL. (yes, nested unlesses. iconic.)
• if wrappedkey is not NULL and the call succeeds, then the generated shared secret is written to genkey and its size is written to *genkeylen (which must be non-NULL). the encapsulated ciphertext is written to wrappedkey and its size is written to *wrappedkeylen (which must also be non-NULL). the value pointed to by wrappedlen initially holds the size of the unwrapped buffer, so the call can validate that size and make sure it's large enough to hold the result written to wrapped.

⚠️ and here's the big one: absent detailed prior knowledge of the internals of the specific KEM algorithm, callers SHOULD NOT assume that the returned shared secret and ciphertext are necessarily of the maximum possible length. the lengths returned via *wrappedkeylen and *genkeylen SHOULD be used to determine the actual lengths of the outputs. do not skip this. fr fr.

---

💅 NOTES 💅

after you call EVP_PKEY_encapsulate_init(), algorithm-specific parameters for the operation may be set or modified using EVP_PKEY_CTX_set_params(3). tune it, bestie. 🎛️

---

👑 RETURN VALUES 👑

EVP_PKEY_encapsulate_init(), EVP_PKEY_auth_encapsulate_init() and EVP_PKEY_encapsulate() all return 1 for success, and 0 or a negative value for failure. in particular, a return value of -2 means the operation is not supported by the public key algorithm. 💀

---

🌈 EXAMPLES 🌈

encapsulate an RSASVE key (for RSA keys). look at her go:

#include <openssl/evp.h>

/*
 * NB: assumes rsa_pub_key is an public key of another party.
 */

EVP_PKEY_CTX *ctx = NULL;
size_t secretlen = 0, outlen = 0;
unsigned char *out = NULL, *secret = NULL;

ctx = EVP_PKEY_CTX_new_from_pkey(libctx, rsa_pub_key, NULL);
if (ctx == NULL)
    /* Error */
if (EVP_PKEY_encapsulate_init(ctx, NULL) <= 0)
    /* Error */

/* Set the mode - only 'RSASVE' is currently supported */
 if (EVP_PKEY_CTX_set_kem_op(ctx, "RSASVE") <= 0)
    /* Error */
/* Determine buffer length */
if (EVP_PKEY_encapsulate(ctx, NULL, &outlen, NULL, &secretlen) <= 0)
    /* Error */

out = OPENSSL_malloc(outlen);
secret = OPENSSL_malloc(secretlen);
if (out == NULL || secret == NULL)
    /* malloc failure */

/*
 * The generated 'secret' can be used as key material.
 * The encapsulated 'out' can be sent to another party who can
 * decapsulate it using their private key to retrieve the 'secret'.
 */
if (EVP_PKEY_encapsulate(ctx, out, &outlen, secret, &secretlen) <= 0)
    /* Error */

---

🎀 SEE ALSO 🎀

EVP_PKEY_CTX_new_from_pkey(3), EVP_PKEY_decapsulate(3), EVP_KEM-RSA(7), EVP_KEM-X25519(7), EVP_KEM-EC(7), EVP_KEM-ML-KEM-512(7), EVP_KEM-ML-KEM-768(7), EVP_KEM-ML-KEM-1024(7)

---

⭐ HISTORY ⭐

the functions EVP_PKEY_encapsulate_init() and EVP_PKEY_encapsulate() were added in OpenSSL 3.0. the function EVP_PKEY_auth_encapsulate_init() was added in OpenSSL 3.2.

support for ML-KEM was added in OpenSSL 3.5. glow up. ✨

---

💜 COPYRIGHT 💜

Copyright 2020-2025 The OpenSSL Project Authors. All Rights Reserved.

Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliance with the License. You can obtain a copy in the file LICENSE in the source distribution or at <https://www.openssl.org/source/license.html>.

---

and that's the tea, bestie 🫶 this whole page is from OpenSSL 3.5.9, dated 2026-09-29, and yes it's literally titled EVP_PKEY_ENCAPSULATE(3ossl). go encapsulate something. mwah 💅🌸