CMS_DECRYPT(3ossl)

📦 version 3.5.0, in opensuse-leap 16.0

💌 CMS_DECRYPT — the OpenSSL 3ossl girlie's guide 💌

okay so this page is all about CMS_decrypt and its little set1_ squad, and it lives in the OpenSSL manual, section 3ossl. we're talking decrypting content from a CMS envelopedData structure and honestly? it's giving locked-treasure-chest energy 🔐✨

🌸 SYNOPSIS 🌸

like, the vibes are three headers and four functions, no biggie:

#include <openssl/cms.h>

int CMS_decrypt(CMS_ContentInfo *cms, EVP_PKEY *pkey, X509 *cert,
                BIO *dcont, BIO *out, unsigned int flags);
int CMS_decrypt_set1_pkey_and_peer(CMS_ContentInfo *cms,
                EVP_PKEY *pk, X509 *cert, X509 *peer);
int CMS_decrypt_set1_pkey(CMS_ContentInfo *cms, EVP_PKEY *pk, X509 *cert);
int CMS_decrypt_set1_password(CMS_ContentInfo *cms,
                              unsigned char *pass, ossl_ssize_t passlen);

✨ DESCRIPTION ✨

bestie, let's go one by one 💅

🎀 CMS_decrypt()

okay so CMS_decrypt() extracts the decrypted content from a CMS EnvelopedData or AuthEnvelopedData structure — like unzipping the vibe, yk?

it uses CMS_decrypt_set1_pkey() to decrypt the content with the recipient private key pkey — if pkey is not NULL. and in this case, the associated certificate is recommended to provide in cert (peep the NOTES below, iykyk 👀). out is a BIO to write the content to, and flags is an optional set of flags.

but wait — if pkey is NULL, the function assumes decryption was already done (e.g., using CMS_decrypt_set1_pkey() or CMS_decrypt_set1_password()) and just provides the content, unless cert, dcont, and out are NULL as well. lowkey efficient ✨

the dcont parameter is used in the rare case where the encrypted content is detached. it will normally be set to NULL.

🦋 CMS_decrypt_set1_pkey_and_peer()

this one decrypts the CMS_ContentInfo structure cms using the private key pkey, the corresponding certificate cert (which is recommended but may be NULL), and the (optional) originator certificate peer.

on success, it also records in cms the decryption key pkey, and then should be followed by "CMS_decrypt(cms, NULL, NULL, dcont, out, flags)". and heads up — this call deallocates any decryption key stored in cms. periodt. 💀

🌈 CMS_decrypt_set1_pkey()

literally just CMS_decrypt_set1_pkey_and_peer() with peer being NULL. same icon, smaller accessory. 🫶

🍓 CMS_decrypt_set1_password()

this one decrypts the CMS_ContentInfo structure cms using the secret pass of length passlen.

on success, it also records in cms the decryption key used, and then should be followed by "CMS_decrypt(cms, NULL, NULL, dcont, out, flags)". also — this call deallocates any decryption key stored in cms. so don't come crying later, bestie 😭

👑 NOTES 👑

ngl this part is the main character energy of the whole page. read it. fr fr.

although the recipient's certificate is not needed to decrypt the data, it is needed to locate the appropriate (of possible several) recipients in the CMS structure. so like, use it pls 💖

if cert is set to NULL, all possible recipients are tried. this case however is problematic. 🚨 to thwart the MMA attack (Bleichenbacher's attack on PKCS #1 v1.5 RSA padding) all recipients are tried whether they succeed or not. if no recipient succeeds, then a random symmetric key is used to decrypt the content: this will typically output garbage and may (but is not guaranteed to) ultimately return a padding error only.

😭 the reason is timing: if CMS_decrypt() just returned an error when all recipient encrypted keys failed to decrypt, an attacker could use this in a timing attack. so the paranoia is load-bearing.

if the special flag CMS_DEBUG_DECRYPT is set, then the above behaviour is modified and an error is returned if no recipient encrypted key can be decrypted without generating a random content encryption key. but bestie — applications should use this flag with extreme caution, especially in automated gateways, as it can leave them open to attack. 🔥 seriously. see EVP_PKEY_decrypt(3) for more details.

it is possible to determine the correct recipient key by other means (for example, looking them up in a database) and setting them in the CMS structure in advance using the CMS utility functions such as CMS_set1_pkey(), or use CMS_decrypt_set1_password() if the recipient has a symmetric key. in these cases both cert and pkey should be set to NULL. cute and clean ✨

to process KEKRecipientInfo types, CMS_set1_key() or CMS_RecipientInfo_set0_key() and CMS_RecipientInfo_decrypt() should be called before CMS_decrypt(), and cert and pkey set to NULL. order matters, bb 💅

🧚 the flags parameter

the following flags can be passed in the flags parameter:

  • CMS_TEXT — if this flag is set, MIME headers for type "text/plain" are deleted from the content. if the content is not of type "text/plain", then an error is returned. no mixing vibes 🚫

⭐ RETURN VALUES ⭐

CMS_decrypt(), CMS_decrypt_set1_pkey_and_peer(), CMS_decrypt_set1_pkey(), and CMS_decrypt_set1_password() all return either 1 for success or 0 for failure. and the error can be obtained from ERR_get_error(3). so when it flops, go ask ERR_get_error(3) what happened 🔍

💀 BUGS 💀

  • the set1_ part of these function names is misleading and should better read: with_. like, not me reading "set1_" and getting it wrong every time 😭 (we keep the names as-is, obviously — don't go renaming anything, bestie.)
  • the lack of single pass processing and the need to hold all data in memory, as mentioned in CMS_verify(), also applies to CMS_decrypt(). so yeah, memory go brrr 🙈

🌈 SEE ALSO 🌈

go read these next, they're the supporting cast:

🌸 HISTORY 🌸

CMS_decrypt_set1_pkey_and_peer() and CMS_decrypt_set1_password() were added in OpenSSL 3.0. newer icons, same slay 💅

Copyright 2008-2020 The OpenSSL Project Authors. All Rights Reserved.

Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliance with the License. You can obtain a copy in the file LICENSE in the source distribution or at <https://www.openssl.org/source/license.html>.

okay that's the whole page, bestie 🫶 mwah 💋 — this masterpiece dropped as part of OpenSSL 3.5.0, dated 2026-09-25, and yes the page itself is CMS_DECRYPT(3ossl) 🎀✨

📦 where this page comes from