EVP_PKEY_ENCAPSULATE(3ossl)

📦 version 3.5.0, in opensuse-leap 16.0

💖 EVP_PKEY_ENCAPSULATE(3ossl) 💖

okay so bestie, this is a page from the 3ossl manual — OpenSSL, the big one. Sit down, hydrate, and let me tell you about key encapsulation using a KEM algorithm with a public key. It's giving cryptography main character energy and honestly? Obsessed.

✨ NAME ✨

EVP_PKEY_encapsulate_init, EVP_PKEY_auth_encapsulate_init, EVP_PKEY_encapsulate — key encapsulation using a KEM algorithm with a public key. That's the whole vibe. 💅

🎀 SYNOPSIS 🎀

#include <openssl/evp.h>

int EVP_PKEY_encapsulate_init(EVP_PKEY_CTX *ctx, const OSSL_PARAM params[]);
int EVP_PKEY_auth_encapsulate_init(EVP_PKEY_CTX *ctx, EVP_PKEY *authpriv,
                                  const OSSL_PARAM params[]);
int EVP_PKEY_encapsulate(EVP_PKEY_CTX *ctx,
                         unsigned char *wrappedkey, size_t *wrappedkeylen,
                         unsigned char *genkey, size_t *genkeylen);

like, look at those signatures. so clean. 🫶

🌸 DESCRIPTION 🌸

So <pink>EVP_PKEY_encapsulate_init()</pink> is the starter — it initializes a public key algorithm context ctx for an encapsulation operation, and then sets the params on that context, literally the same way as calling EVP_PKEY_CTX_set_params(3). Also, tiny note: ctx is usually produced using EVP_PKEY_CTX_new_from_pkey(3), specifying the public key you wanna use. iykyk.

<purple>EVP_PKEY_auth_encapsulate_init()</purple> is the same deal, but it also passes an authpriv authentication private key that gets used during encapsulation. extra steps, extra power. 🔥

Then <cyan>EVP_PKEY_encapsulate()</cyan> is the moment — it performs the actual public key encapsulation operation using ctx.

  • The shared secret written to genkey can be used as an input for key derivation, typically for various symmetric algorithms. Its size gets written to genkeylen, which must be initialised to the size of the provided buffer. don't skip that step bestie, it's not optional. 😭
  • The ciphertext written to wrappedkey is an encapsulated form. It's expected to be usable only by the holder of the private key corresponding to the public key associated with ctx. That ciphertext gets communicated to the private-key holder, who can then use EVP_PKEY_decapsulate(3) to securely recover the same shared secret. it's a whole two-party slay. 🦋

okay, now the branchy part, pay attention:

  • If wrappedkey is NULL, then the maximum size of the output buffer is written to the *wrappedkeylen parameter — unless wrappedkeylen is NULL — and the maximum size of the generated key buffer is written to *genkeylen — unless genkeylen is NULL. so many unlesses. we love a conditional. 🌈
  • If wrappedkey is not NULL and the call succeeds, then the generated shared secret is written to genkey and its size is written to *genkeylen (which must be non-NULL). The encapsulated ciphertext is written to wrappedkey and its size is written to *wrappedkeylen (also must be non-NULL). The value pointed to by wrappedlen initially holds the size of the unwrapped buffer, so the call can validate its size, making sure it's large enough to hold the result written to wrapped.
real talk though: unless you have detailed prior knowledge of the internals of the specific KEM algorithm, callers SHOULD NOT assume that the returned shared secret and ciphertext are necessarily of the maximum possible length. The lengths returned via *wrappedkeylen and *genkeylen SHOULD be used to determine the actual lengths of the outputs. periodt. 💀

⭐ NOTES ⭐

After the call to EVP_PKEY_encapsulate_init(), algorithm-specific parameters for the operation may be set or modified using EVP_PKEY_CTX_set_params(3). easy. 🧚

👑 RETURN VALUES 👑

EVP_PKEY_encapsulate_init(), EVP_PKEY_auth_encapsulate_init() and EVP_PKEY_encapsulate() all return 1 for success and 0 or a negative value for failure. Specifically, a return value of -2 means the operation is not supported by the public key algorithm. so if you see -2, it's not you, it's the algorithm. 🙈

🍓 EXAMPLES 🍓

Encapsulate an RSASVE key (for RSA keys). iconic example, no notes:

#include <openssl/evp.h>

/*
 * NB: assumes rsa_pub_key is an public key of another party.
 */

EVP_PKEY_CTX *ctx = NULL;
size_t secretlen = 0, outlen = 0;
unsigned char *out = NULL, *secret = NULL;

ctx = EVP_PKEY_CTX_new_from_pkey(libctx, rsa_pub_key, NULL);
if (ctx == NULL)
    /* Error */
if (EVP_PKEY_encapsulate_init(ctx, NULL) <= 0)
    /* Error */

/* Set the mode - only 'RSASVE' is currently supported */
 if (EVP_PKEY_CTX_set_kem_op(ctx, "RSASVE") <= 0)
    /* Error */
/* Determine buffer length */
if (EVP_PKEY_encapsulate(ctx, NULL, &outlen, NULL, &secretlen) <= 0)
    /* Error */

out = OPENSSL_malloc(outlen);
secret = OPENSSL_malloc(secretlen);
if (out == NULL || secret == NULL)
    /* malloc failure */

/*
 * The generated 'secret' can be used as key material.
 * The encapsulated 'out' can be sent to another party who can
 * decapsulate it using their private key to retrieve the 'secret'.
 */
if (EVP_PKEY_encapsulate(ctx, out, &outlen, secret, &secretlen) <= 0)
    /* Error */

🦋 SEE ALSO 🦋

EVP_PKEY_CTX_new_from_pkey(3), EVP_PKEY_decapsulate(3), EVP_KEM-RSA(7), EVP_KEM-X25519(7), EVP_KEM-EC(7), EVP_KEM-ML-KEM-512(7), EVP_KEM-ML-KEM-768(7), EVP_KEM-ML-KEM-1024(7)

the whole KEM friend group, fr fr. 💜

💫 HISTORY 💫

  • EVP_PKEY_encapsulate_init() and EVP_PKEY_encapsulate() were added in OpenSSL 3.0.
  • EVP_PKEY_auth_encapsulate_init() was added in OpenSSL 3.2.
  • Support for ML-KEM was added in OpenSSL 3.5.

glow-up timeline. ✨

Copyright 2020-2025 The OpenSSL Project Authors. All Rights Reserved.

Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliance with the License. You can obtain a copy in the file LICENSE in the source distribution or at <https://www.openssl.org/source/license.html>.

okay that's the whole page bestie, you ate that 💅💖

version 3.5.0, dated 2026-09-25, from the EVP_PKEY_ENCAPSULATE(3ossl) manual — mwah, go encapsulate something 🎀

📦 where this page comes from