EVP_PKEY_ENCAPSULATE(3SSL)

📦
version 4.0.3 · 6 versions
, in ubuntu stonking

💖 EVP_PKEY_ENCAPSULATE 💖

okay bestie, this is OpenSSL, and today we're talking about key encapsulation — using a KEM algorithm with a public key. yes it's giving cryptography main character energy, and yes you're gonna slay. 🦋✨

The star lineup: EVP_PKEY_encapsulate_init, EVP_PKEY_auth_encapsulate_init, and EVP_PKEY_encapsulate.

✨ SYNOPSIS ✨

#include <openssl/evp.h>

int EVP_PKEY_encapsulate_init(EVP_PKEY_CTX *ctx, const OSSL_PARAM params[]);
int EVP_PKEY_auth_encapsulate_init(EVP_PKEY_CTX *ctx, EVP_PKEY *authpriv,
                                  const OSSL_PARAM params[]);
int EVP_PKEY_encapsulate(EVP_PKEY_CTX *ctx,
                         unsigned char *wrappedkey, size_t *wrappedkeylen,
                         unsigned char *genkey, size_t *genkeylen);

🌸 DESCRIPTION 🌸

So like, here's the tea on each one bestie. 💅

EVP_PKEY_encapsulate_init() — this one initializes a public key algorithm context ctx for an encapsulation operation, and then sets the params on the context the exact same way calling EVP_PKEY_CTX_set_params(3) does. Quick note, no cap: ctx is usually produced using EVP_PKEY_CTX_new_from_pkey(3), specifying the public key you wanna use. 🫶

EVP_PKEY_auth_encapsulate_init() — basically the same vibe as EVP_PKEY_encapsulate_init(), but it also passes an authpriv authentication private key that gets used during encapsulation. extra. we love her. 💜

EVP_PKEY_encapsulate() — this is the moment. ✨ It performs a public key encapsulation operation using ctx.

  • The shared secret lands in genkey, and it can be used as an input for key derivation, typically for various symmetric algorithms. Its size gets written to genkeylen, which must be initialised to the size of the provided buffer. 📏
  • The ciphertext lands in wrappedkey — it's an encapsulated form that's expected to be only usable by the holder of the private key corresponding to the public key associated with ctx. This ciphertext then gets communicated to the private-key holder, who can use EVP_PKEY_decapsulate(3) to securely recover the same shared secret. it's a whole handoff, iconic. 🎀

Now, buffer sizing, because details matter:

  • If wrappedkey is NULL, then the maximum size of the output buffer is written to the *wrappedkeylen parameter — unless wrappedkeylen is NULL — and the maximum size of the generated key buffer is written to *genkeylen — unless genkeylen is NULL.
  • If wrappedkey is not NULL and the call succeeds, then the generated shared secret is written to genkey and its size is written to *genkeylen (which must be non-NULL). The encapsulated ciphertext is written to wrappedkey and its size is written to *wrappedkeylen (must also be non-NULL). The value pointed to by wrappedlen initially holds the size of the unwrapped buffer, so that its size can be validated by the call, ensuring it's large enough to hold the result written to wrapped.
⚠️ Real talk bestie, this part matters: absent detailed prior knowledge of the internals of the specific KEM algorithm, callers SHOULD NOT assume that the returned shared secret and ciphertext are necessarily of the maximum possible length. The lengths returned via *wrappedkeylen and *genkeylen SHOULD be used to determine the actual lengths of the outputs. fr fr, read your lengths. 😭

📝 NOTES 📝

After the call to EVP_PKEY_encapsulate_init(), algorithm-specific parameters for the operation may be set or modified using EVP_PKEY_CTX_set_params(3). tweak away babe. 💖

🏆 RETURN VALUES 🏆

EVP_PKEY_encapsulate_init(), EVP_PKEY_auth_encapsulate_init() and EVP_PKEY_encapsulate() return 1 for success and 0 or a negative value for failure. And specifically? A return value of -2 indicates the operation is not supported by the public key algorithm. not supported, not invited, blocked. 💀

🌈 EXAMPLES 🌈

Encapsulate an RSASVE key (for RSA keys). here's the whole thing, verbatim bestie:

#include <openssl/evp.h>

/*
 * NB: assumes rsa_pub_key is an public key of another party.
 */

EVP_PKEY_CTX *ctx = NULL;
size_t secretlen = 0, outlen = 0;
unsigned char *out = NULL, *secret = NULL;

ctx = EVP_PKEY_CTX_new_from_pkey(libctx, rsa_pub_key, NULL);
if (ctx == NULL)
    /* Error */
if (EVP_PKEY_encapsulate_init(ctx, NULL) <= 0)
    /* Error */

/* Set the mode - only 'RSASVE' is currently supported */
 if (EVP_PKEY_CTX_set_kem_op(ctx, "RSASVE") <= 0)
    /* Error */
/* Determine buffer length */
if (EVP_PKEY_encapsulate(ctx, NULL, &outlen, NULL, &secretlen) <= 0)
    /* Error */

out = OPENSSL_malloc(outlen);
secret = OPENSSL_malloc(secretlen);
if (out == NULL || secret == NULL)
    /* malloc failure */

/*
 * The generated 'secret' can be used as key material.
 * The encapsulated 'out' can be sent to another party who can
 * decapsulate it using their private key to retrieve the 'secret'.
 */
if (EVP_PKEY_encapsulate(ctx, out, &outlen, secret, &secretlen) <= 0)
    /* Error */

👀 SEE ALSO 👀

🦋 HISTORY 🦋

  • The functions EVP_PKEY_encapsulate_init() and EVP_PKEY_encapsulate() were added in OpenSSL 3.0.
  • The function EVP_PKEY_auth_encapsulate_init() was added in OpenSSL 3.2.
  • Support for ML-KEM was added in OpenSSL 3.5.

glow-up after glow-up honestly. 👑

Copyright 2020-2025 The OpenSSL Project Authors. All Rights Reserved.

Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliance with the License. You can obtain a copy in the file LICENSE in the source distribution or at <https://www.openssl.org/source/license.html>.

and that's the whole page bestie! 🫶 this one's from OpenSSL 4.0.3, dated 2026-09-29, and it's the EVP_PKEY_ENCAPSULATE(3SSL) manual. go encapsulate something iconic. 🎀✨💅

📦 where this page comes from