PERLGUTS(1perl)

💖 NAME 💖

perlguts — Introduction to the Perl API ✨

---

🦋 DESCRIPTION 🦋

okay so bestie, this doc is literally trying to explain how to use the Perl API, plus give you some tea on how the Perl core actually works under the hood. 💅

ngl it is far from complete and probably contains many errors — the manpage said it, not me. 😭 Any questions or comments? Send them to the Perl 5 Porters <perl5-porters@perl.org>. 🫶

---

🌸 Variables 🌸

🎀 Datatypes 🎀

Perl has three typedefs that handle Perl's three main data types, and they are icons fr fr:

    SV  Scalar Value
    AV  Array Value
    HV  Hash Value

Each typedef has its own specific routines that manipulate the various data types. slay. ✨

👑 What is an "IV"? 👑

Perl uses a special typedef IV which is a simple signed integer type that is guaranteed to be large enough to hold a pointer, but may be larger, depending on how perl is Configured. And there's also the UV, which is just an unsigned IV. 💜

Perl also uses several special typedefs to declare variables to hold integers of (at least) a given size:

• Use I8, I16, I32, and I64 to declare a signed integer variable which has at least as many bits as the number in its name.
• These all evaluate to the native C type that is closest to the given number of bits, but no smaller than that number.
• For example, on many platforms, a short is 16 bits long, and if so, I16 will evaluate to a short. But on platforms where a short isn't exactly 16 bits, Perl will use the smallest type that contains 16 bits or more. 💅

U8, U16, U32, and U64 are to declare the corresponding unsigned integer types.

If the platform doesn't support 64-bit integers, both I64 and U64 will be undefined. Use IV and UV to declare the largest practicable, and "WIDEST_UTYPE" in perlapi for the absolute maximum unsigned — but that one may not be usable in all circumstances. ⚠️

A numeric constant can be specified with the C99 macros like INT16_C, "UINTMAX_C" in perlapi, and similar. 🍓

---

🧚 Working with SVs 🧚

An SV can be created and loaded with one command. There are five types of values that can be loaded:

• an integer value (IV)
• an unsigned integer value (UV)
• a double (NV)
• a string (PV)
• another scalar (SV)

so basically the whole friend group. 💅

(PV stands for "Pointer Value". You might think that it is misnamed because it is described as pointing only to strings. However, it is possible to have it point to other things — for example, it could point to an array of UVs. But using it for non-strings requires care, as the underlying assumption of much of the internals is that PVs are just for strings. Often, for example, a trailing NUL is tacked on automatically. The non-string use is documented only in this paragraph. iykyk. 🙈)

There are more than a few routines that create and initialize SVs, with new ones occasionally added. These are all documented in "SV Handling" in perlapi; each one's name begins with newSV. Here are the basic ones:

    SV*  newSVbool(boolean_value);
    SV*  newSViv(IV);
    SV*  newSVuv(UV);
    SV*  newSVnv(double);
    SV*  newSVpv(const char*, STRLEN);
    SV*  newSVpvf_nocontext(const char * const pat, ...)
    SV*  newSVpvn(const char*, STRLEN);
    SV*  newSVpvs("literal string")
    SV*  newSVpvz(STRLEN)
    SV*  newSVsv(SV*);

STRLEN is an integer type (Size_t, usually defined as size_t in config.h) guaranteed to be large enough to represent the size of any string that perl can handle. 💖

literal string is a string enclosed in double quote characters, and the created SV is of string type (a PV), initialized to contain a copy of the literal string parameter. newSVpvs("") creates an SV containing an empty (zero-length) string and returns a pointer to it. ✨

newSVpvz is like newSVpvs(""), but it reserves the number of bytes specified by its parameter for future growth, without having to reallocate memory. It's useful in situations where you don't know what the string eventually will be, but you have a pretty good idea of its eventual minimum size. 🌈

For an SV requiring more complex initialization, you can create an empty SV with newSV(len):

• If len is 0, an empty SV of type NULL is returned.
• Else, an SV of type PV is returned with len + 1 (for the NUL) bytes of storage allocated, accessible via SvPVX.
• In both cases the SV has the undef value.

    SV *sv = newSV(0);   /* no storage allocated  */
    SV *sv = newSV(10);  /* 10 (+1) bytes of uninitialised storage
                          * allocated */

To change the value of an already-existing SV, these are the basic routines (with more specified in "SV Handling" in perlapi):

    void  sv_setbool(SV*, bool);
    void  sv_setiv(SV*, IV);
    void  sv_setuv(SV*, UV);
    void  sv_setnv(SV*, double);
    void  sv_setpv(SV*, const char*);
    void  sv_setpvn(SV*, const char*, STRLEN)
    void  sv_setpvf(SV*, const char*, ...);
    void  sv_vsetpvfn(SV*, const char*, STRLEN, va_list *,
                                        SV **, Size_t, bool *);
    void  sv_setsv(SV*, SV*);

So notice, bestie: you can choose to specify the length of the string to be assigned by using sv_setpvn, newSVpvn, or newSVpv — or you may let Perl calculate the length by using sv_setpv or by specifying 0 as the second argument to newSVpv. 🎀

But be warned: Perl will determine the string's length by using strlen, which depends on the string terminating with a NUL character, and not otherwise containing NULs. 💀

The arguments of sv_setpvf are processed like sprintf, and the formatted output becomes the value. ✨

sv_vsetpvfn is an analogue of vsprintf, but it allows you to specify either a pointer to a variable argument list or the address and length of an array of SVs. The last argument points to a boolean; on return, if that boolean is true, then locale-specific information has been used to format the string, and the string's contents are therefore untrustworthy (see perlsec). This pointer may be NULL if that information is not important. Note that this function requires you to specify the length of the format. 💅

The sv_set*() functions are not generic enough to operate on values that have "magic". See "Magic Virtual Tables" later in this document. 🔮

---

All SVs that contain strings should be terminated with a NUL character. 🫶

If it is not NUL-terminated, there is a risk of core dumps and corruptions from code which passes the string to C functions or system calls which expect a NUL-terminated string. Perl's own functions typically add a trailing NUL for this reason. Nevertheless, you should be very careful when you pass a string stored in an SV to a C function or system call. 😭

---

To access the actual value that an SV points to, Perl's API exposes several macros that coerce the actual scalar type into an IV, UV, double, or string:

• SvIV(SV*) (IV) and SvUV(SV*) (UV)
• SvNV(SV*) (double)
• Strings are a bit complicated:
  • Byte string: SvPVbyte(SV*, STRLEN len) or SvPVbyte_nolen(SV*)
    • If the Perl string is "\xff\xff", then this returns a 2-byte char*.
    • This is suitable for Perl strings that represent bytes.
  • UTF-8 string: SvPVutf8(SV*, STRLEN len) or SvPVutf8_nolen(SV*)
    • If the Perl string is "\xff\xff", then this returns a 4-byte char*.
    • This is suitable for Perl strings that represent characters.
    • CAVEAT: That char* will be encoded via Perl's internal UTF-8 variant, which means that if the SV contains non-Unicode code points (e.g., 0x110000), then the result may contain extensions over valid UTF-8. See "is_strict_utf8_string" in perlapi for some methods Perl gives you to check the UTF-8 validity of these macros' returns. ⚠️
  • You can also use SvPV(SV*, STRLEN len) or SvPV_nolen(SV*) to fetch the SV's raw internal buffer. This is tricky, though; if your Perl string is "\xff\xff", then depending on the SV's internal encoding you might get back a 2-byte OR a 4-byte char*. Moreover, if it's the 4-byte string, that could come from either Perl "\xff\xff" stored UTF-8 encoded, or Perl "\xc3\xbf\xc3\xbf" stored as raw octets. To differentiate between these you MUST look up the SV's UTF8 bit (cf. SvUTF8) to know whether the source Perl string is 2 characters (SvUTF8 would be on) or 4 characters (SvUTF8 would be off).
    • IMPORTANT: Use of SvPV, SvPV_nolen, or similarly-named macros without looking up the SV's UTF8 bit is almost certainly a bug if non-ASCII input is allowed. 💀
    • When the UTF8 bit is on, the same CAVEAT about UTF-8 validity applies here as for SvPVutf8.
  • (See "How do I pass a Perl string to a C library?" for more details.)
  • In SvPVbyte, SvPVutf8, and SvPV, the length of the char* returned is placed into the variable len (these are macros, so you do not use &len). If you do not care what the length of the data is, use SvPVbyte_nolen, SvPVutf8_nolen, or SvPV_nolen instead. The global variable PL_na can also be given to SvPVbyte/SvPVutf8/SvPV in this case. But that can be quite inefficient because PL_na must be accessed in thread-local storage in threaded Perl. In any case, remember that Perl allows arbitrary strings of data that may both contain NULs and might not be terminated by a NUL. 🌀
  • Also remember that C doesn't allow you to safely say foo(SvPVbyte(s, len), len);. It might work with your compiler, but it won't work for everyone. Break this sort of statement up into separate assignments:

    SV *s;
    STRLEN len;
    char *ptr;
    ptr = SvPVbyte(s, len);
    foo(ptr, len);

---

If you want to know if the scalar value is TRUE, you can use:

    SvTRUE(SV*)

Although Perl will automatically grow strings for you, if you need to force Perl to allocate more memory for your SV, you can use the macro:

    SvGROW(SV*, STRLEN newlen)

which will determine if more memory needs to be allocated. If so, it will call the function sv_grow. 💖

Note that SvGROW can only increase, not decrease, the allocated memory of an SV — and that it does not automatically add space for the trailing NUL byte (perl's own string functions typically do SvGROW(sv, len + 1)). 👑

So like, if you wanna write into an SV's buffer that already exists and set its value to a string, you grab SvPVbyte_force() or one of its little siblings and use that to force the SV into being a PV. This yeets all the various flavours of non-stringness out of the SV while keeping the content of the SV in the PV, no cap. Super handy for, say, appending data from an API function onto a buffer without paying for an extra copy:

(void)SvPVbyte_force(sv, len);
s = SvGROW(sv, len + needlen + 1);
/* something that modifies up to needlen bytes at s+len, but
   modifies newlen bytes
     eg. newlen = read(fd, s + len, needlen);
   ignoring errors for these examples
 */
s[len + newlen] = '\0';
SvCUR_set(sv, len + newlen);
SvUTF8_off(sv);
SvSETMAGIC(sv);

If the data is already chilling in memory, or you just want to keep your code simple (valid 💅), you can use one of the sv_cat*() variants, like sv_catpvn(). And if you need to insert anywhere in the string, there's sv_insert() or sv_insert_flags() for that.

Don't need the existing content of the SV at all? Then you can dodge a bunch of copying like this:

SvPVCLEAR(sv);
s = SvGROW(sv, needlen + 1);
/* something that modifies up to needlen bytes at s, but modifies
   newlen bytes
     eg. newlen = read(fd, s, needlen);
 */
s[newlen] = '\0';
SvCUR_set(sv, newlen);
SvPOK_only(sv); /* also clears SVf_UTF8 */
SvSETMAGIC(sv);

And again — if you already have the data in memory, or you want to skip all that complexity, sv_setpvn() has your back.

Got a buffer allocated with Newx() and you want to make it the SV's value? That's sv_usepvn_flags(). But bestie, it has requirements if you want to stop perl from re-allocating the buffer just to fit that trailing NUL:

Newx(buf, somesize+1, char);
/* ... fill in buf ... */
buf[somesize] = '\0';
sv_usepvn_flags(sv, buf, somesize, SV_SMAGIC | SV_HAS_TRAILING_NUL);
/* buf now belongs to perl, don't release it */

If you've got an SV and you're dying to know what kind of data Perl thinks is stored in it, these macros will do a lil vibe check on the type of SV you're holding:

SvIOK(SV*)
SvNOK(SV*)
SvPOK(SV*)

⚠️ Heads up: grabbing the numeric value of an SV can set IOK or NOK on that SV, even when the SV started out as a string. Prior to Perl 5.36.0, retrieving the string value of an integer could set POK, but that can no longer occur. From 5.36.0 onwards this can be used to distinguish the original representation of an SV, and it's intended to make life simpler for serializers:

/* references handled elsewhere */
if (SvIsBOOL(sv)) {
    /* originally boolean */
    ...
}
else if (SvPOK(sv)) {
    /* originally a string */
    ...
}
else if (SvNIOK(sv)) {
    /* originally numeric */
    ...
}
else {
    /* something special or undef */
}

To get and set the current length of the string stored in an SV, these two macros are the moment:

SvCUR(SV*)
SvCUR_set(SV*, I32 val)

You can also snag a pointer to the end of the string stored in the SV with:

SvEND(SV*)

But bestie, plot twist — those last three macros are only valid if SvPOK() is true. Not negotiable.

Wanna append something to the end of the string stored in an SV*? These functions, take your pick:

void  sv_catpv(SV*, const char*);
void  sv_catpvn(SV*, const char*, STRLEN);
void  sv_catpvf(SV*, const char*, ...);
void  sv_vcatpvfn(SV*, const char*, STRLEN, va_list *, SV **,
                                                         I32, bool);
void  sv_catsv(SV*, SV*);

The first one figures out the length of the string to append by using strlen. In the second, you specify the length yourself. The third processes its arguments like sprintf and appends the formatted output. The fourth works like vsprintf — and fun fact, you can specify the address and length of an array of SVs instead of the va_list argument. The fifth extends the string stored in the first SV with the string stored in the second SV. It also forces that second SV to be interpreted as a string, no arguing.

The sv_cat*() functions are not generic enough to operate on values that have "magic". Peep "Magic Virtual Tables" later in this document for the tea. 🫖

Know the name of a scalar variable and want a pointer to its SV? Slay, just:

SV*  get_sv("package::varname", 0);

This returns NULL if the variable doesn't exist. Rude but honest.

Wanna know if this variable (or literally any other SV) is actually defined? Call:

SvOK(SV*)

The scalar undef value is stored in an SV instance called PL_sv_undef.

Its address can be used whenever an SV* is needed. But make sure you don't try to compare a random sv with &PL_sv_undef. For example, when interfacing Perl code, it'll work correctly for:

foo(undef);

But won't work when it's called as:

$x = undef;
foo($x);

So to say it once more for the people in the back: always use SvOK() to check whether an sv is defined. Periodt.

Also, be careful when using &PL_sv_undef as a value in AVs or HVs (see "AVs, HVs and undefined values").

There are also two more values, PL_sv_yes and PL_sv_no, which hold boolean TRUE and FALSE values respectively. Like PL_sv_undef, their addresses can be used whenever an SV* is needed. Iconic trio, honestly.

Do not get fooled into thinking that (SV *) 0 is the same as &PL_sv_undef. Look at this code:

SV* sv = (SV*) 0;
if (I-am-to-return-a-real-value) {
        sv = sv_2mortal(newSViv(42));
}
sv_setsv(ST(0), sv);

This code is trying to return a new SV (containing the value 42) if it should return a real value, or undef otherwise. Instead it has returned a NULL pointer which, somewhere down the line, will cause a segmentation violation, bus error, or just plain weird results. 😭 Change the zero to &PL_sv_undef in the first line and all will be well. That's the whole fix. That's it.

To free an SV that you've created, call SvREFCNT_dec(SV*). Normally this call is not necessary though (see "Reference Counts and Mortality").

---

✨ Offsets ✨

Perl provides the function sv_chop to efficiently remove characters from the beginning of a string. You hand it an SV and a pointer to somewhere inside the PV, and it discards everything before that pointer. The efficiency comes from a little hack: instead of actually removing the characters, sv_chop sets the flag OOK (offset OK) to signal to other functions that the offset hack is in effect, and it moves the PV pointer (called SvPVX) forward by the number of bytes chopped off, and adjusts SvCUR and SvLEN accordingly. (A portion of the space between the old and new PV pointers is used to store the count of chopped bytes. Sneaky. 🧚)

So at this point, the start of the buffer that we allocated lives at SvPVX(sv) - SvIV(sv) in memory, and the PV pointer is pointing into the middle of this allocated storage. It's giving "pointer into the void" but make it intentional.

This is best demonstrated by example. Normally copy-on-write will prevent the substitution operator from using this hack, but if you can craft a string for which copy-on-write is not possible, you can see it in play. In the current implementation, the final byte of a string buffer is used as a copy-on-write reference count. If the buffer isn't big enough, then copy-on-write is skipped. First, have a look at an empty string:

% ./perl -Ilib -MDevel::Peek -le '$a=""; $a .= ""; Dump $a'
SV = PV(0x7ffb7c008a70) at 0x7ffb7c030390
  REFCNT = 1
  FLAGS = (POK,pPOK)
  PV = 0x7ffb7bc05b50 ""\0
  CUR = 0
  LEN = 10

Notice here the LEN is 10. (It may differ on your platform, don't come for me.) Now extend the length of the string to one less than 10, and do a substitution:

% ./perl -Ilib -MDevel::Peek -le '$a=""; $a.="123456789"; $a=~s/.//; \
                                                            Dump($a)'
SV = PV(0x7ffa04008a70) at 0x7ffa04030390
  REFCNT = 1
  FLAGS = (POK,OOK,pPOK)
  OFFSET = 1
  PV = 0x7ffa03c05b61 ( "\1" . ) "23456789"\0
  CUR = 8
  LEN = 9

Here the number of bytes chopped off (1) is shown next as the OFFSET. The portion of the string between the "real" and the "fake" beginnings is shown in parentheses, and the values of SvCUR and SvLEN reflect the fake beginning, not the real one. (The first character of the string buffer happens to have changed to "\1" here, not "1", because the current implementation stores the offset count in the string buffer. This is subject to change. ⚠️)

Something similar to the offset hack is performed on AVs to enable efficient shifting and splicing off the beginning of the array. While AvARRAY points to the first element in the array that is visible from Perl, AvALLOC points to the real start of the C array. These are usually the same, but a shift operation can be carried out by increasing AvARRAY by one and decreasing AvFILL and AvMAX. Again, the location of the real start of the C array only comes into play when freeing the array. See "av_shift" in av.c.

---

🌈 What's Really Stored in an SV? 🌈

Okay so recall that the usual method of determining the type of scalar you have is to use Sv*OK macros. Because a scalar can be both a number and a string, usually these macros will always return TRUE, and calling the Sv*V macros will do the appropriate conversion of string to integer/double or integer/double to string. Multitasking behaviour, honestly.

If you really need to know if you have an integer, double, or string pointer in an SV, use these three macros instead:

SvIOKp(SV*)
SvNOKp(SV*)
SvPOKp(SV*)

These will tell you if you truly have an integer, double, or string pointer stored in your SV. The "p" stands for private. 🤫

There are various ways in which the private and public flags may differ. For example, in perl 5.16 and earlier, a tied SV may have a valid underlying value in the IV slot (so SvIOKp is true), but the data should be accessed via the FETCH routine rather than directly, so SvIOK is false. (In perl 5.18 onwards, tied scalars use the flags the same way as untied scalars.) Another way is when numeric conversion has occurred and precision has been lost: only the private flag is set on 'lossy' values. So when an NV is converted to an IV with loss, SvIOKp, SvNOKp and SvNOK will be set, while SvIOK won't be.

In general though, it's best to use the Sv*V macros.

---

💜 Read-Only Values 💜

In Perl 5.16 and earlier, copy-on-write (see the next section) shared a flag bit with read-only scalars. So the only way to test whether sv_setsv, etc., will raise a "Modification of a read-only value" error in those versions is:

SvREADONLY(sv) && !SvIsCOW(sv)

Okay so under Perl 5.18 and later, SvREADONLY only applies to read-only variables, and under 5.20, copy-on-write scalars can also be read-only — so the check above? Yeah, it's just wrong now 😭. What you actually want is:

SvREADONLY(sv)

And if you find yourself doing this check all the time, just define your own macro, like:

#if PERL_VERSION >= 18
# define SvTRULYREADONLY(sv) SvREADONLY(sv)
#else
# define SvTRULYREADONLY(sv) (SvREADONLY(sv) && !SvIsCOW(sv))
#endif

Or honestly, even better? Go read about the THINKFIRST macros down below. Trust. 💅

✨ Copy on Write ✨

Perl does this whole copy-on-write (COW) thing for scalars: when someone asks for a string copy, it does not make the copy right away — it just defers it until one or the other scalar actually changes. It's giving "we'll deal with it when it becomes a problem" 🫶. Most of the time this is completely transparent, but heads up: you must be careful not to modify string buffers that are shared between multiple SVs. That's the one rule, bestie.

Wanna know if an SV is doing the COW thing? Test it with SvIsCOW(sv).

Wanna force an SV to make its own actual copy of its string buffer? Call sv_force_normal(sv) or SvPV_force_nolen(sv).

And if you want the SV to just drop its string buffer instead? Use sv_force_normal_flags(sv, SV_COW_DROP_PV) or, like, simply sv_setsv(sv, NULL).

⚠️ All of these functions will croak on read-only scalars (peep the previous section for the whole story on those).

So how do you check that your code is actually behaving and not out there vandalizing COW buffers 😭? On systems that support mmap(2) — think Unix, Linux, BSDs, macOS — you can configure perl with -Accflags=-DPERL_DEBUG_READONLY_COW and it will turn buffer violations into full-on crashes. Very dramatic, very effective. It is also, and I quote, marvellously slow, so you might wanna skip perl's own tests. 💀

🦋 THINKFIRST before writing to a string buffer 🦋

You're more likely to see — and honestly you usually should see — the SvTHINKFIRST(sv) macro used to check whether an SV is even ready to be written to. It's a combined single check that tells you if the SV:

• Points to a COW buffer
• Is READONLY
• Contains a reference
• Has some relevant magic assigned

There are two related macros that do those checks and, if it's needed, also perform an action:

• SV_CHECK_THINKFIRST(sv) calls sv_force_normal(sv) to copy the string buffer, turning it into a mutable string.
• SV_CHECK_THINKFIRST_COW_DROP(sv) calls sv_force_normal_flags(sv, SV_COW_DROP_PV), which drops any string buffer pointed to by sv. This is usually used to avoid copying a COW string into a new buffer, right before writing some completely new string into sv.

👑 Working with AVs 👑

There are two main, longstanding ways to create and load an AV. The first one makes you an empty AV:

AV*  newAV();

The second one creates the AV and populates it with SVs right off the bat:

AV*  av_make(SSize_t num, SV **ptr);

That second argument points to an array containing num SV*'s. Once the AV exists, those SVs can be destroyed, if you feel like it. No pressure. 🎀

Perl v5.36 added two new ways to create an AV and allocate a SV** array without populating it. These are more efficient than a newAV() followed by an av_extend(). Slay.

/* Creates but does not initialize (Zero) the SV** array */
AV *av = newAV_alloc_x(1);
/* Creates and does initialize (Zero) the SV** array */
AV *av = newAV_alloc_xz(1);

That numerical argument is the number of array elements to allocate — not an array index — and it must be >0. Also, the first form must only ever be used when all the elements will be initialized before any read happens.

Serious bug alert 😭: reading a non-initialized SV* — like, treating a random memory address as an SV* — is a serious bug. Do not do that. Not even a little.

Once your AV is created, these operations are on the menu:

void  av_push(AV*, SV*);
SV*   av_pop(AV*);
SV*   av_shift(AV*);
void  av_unshift(AV*, SSize_t num);

These should all feel pretty familiar, with one exception: av_unshift. That routine adds num elements to the front of the array with the undef value. You then have to use av_store (described below) to actually assign values to those new elements. She's a two-step process, okay? 💅

Here are some other functions:

Size_t  av_count(AV*);
SSize_t av_top_index(AV*);
SV**    av_fetch(AV*, SSize_t key, I32 lval);
SV**    av_store(AV*, SSize_t key, SV* val);

av_count returns the number of elements in the array — empty slots (undefined ones) mixed in with the filled-in ones included, no erasure. The av_top_index function returns the highest index value in an array, just like $#array in Perl. If the array is empty, -1 gets returned. It is always equal to av_count() - 1. The av_fetch function returns the value at index key, but if lval is non-zero, then av_fetch will store an undef value at that index. The av_store function stores the value val at index key, and it does not increment the reference count of val. So the caller is the one responsible for handling that, and if av_store returns NULL, the caller will have to decrement the reference count to avoid a memory leak 🙈. Also note: av_fetch and av_store both return SV**'s, not SV*'s as their return value.

A few more:

void  av_clear(AV*);
void  av_undef(AV*);
void  av_extend(AV*, SSize_t key);

The av_clear function deletes all the elements in the AV* array, but does not actually delete the array itself. The av_undef function will delete all the elements in the array plus the array itself — she's thorough. The av_extend function extends the array so that it contains at least key+1 elements. If key+1 is less than the currently allocated length of the array, then nothing is done.

If you already know the name of an array variable, you can grab a pointer to its AV like this:

AV*  get_av("package::varname", 0);

This returns NULL if the variable doesn't exist. Rude, but factual.

See "Understanding the Magic of Tied Hashes and Arrays" for more info on how to use the array access functions on tied arrays.

🌸 More efficient working with new or vanilla AVs 🌸

Perl v5.36 and v5.38 introduced streamlined, inlined versions of some functions:

• av_store_simple
• av_fetch_simple
• av_push_simple

These are drop-in replacements ✨ — but they can only be used on straightforward AVs that meet the following criteria:

• are not magical
• are not readonly
• are "real" AVs (see the next section 👀)
• have an av_top_index value > -2

AVs created using newAV(), av_make, newAV_alloc_x, and newAV_alloc_xz are all compatible at the time of creation. It's only if they get declared readonly or unreal, have magic attached, or are otherwise configured unusually that they stop being compatible. She changed, bestie. 💀

Also, note that some interpreter functions may attach magic to an AV as part of normal operations. So, unless you're sure about the lifecycle of an AV, it is safest to only use these new functions close to the point of AV creation. 🫶

🧚 Real AVs — and those that are not 🧚

The standard or typical AV is sometimes referred to as being a "real" AV, a state you can check for with the AvREAL macro. What that state basically means is:

• Every accessible element in the array is either initialized but empty, or contains a pointer to a live SV.
• When a SV* is assigned to an array element, the reference count on the SV is incremented. Conversely, when the element is unset, or assigned a different SV*, the reference count of the expelled SV is decremented.

"Fake" AVs are intended for specific, limited use cases only. Using them outside of core perl is strongly discouraged. Like, don't. 🚫

For example, the interpreter's argument stack is implemented as an AV, but it is not AvREAL. Its elements were historically not reference counted, which gives rise to the whole "stack-not-refcounted" class of bugs — the kind where SVs get freed during the execution of a statement but are still needed later in that same statement 😭. Efforts are currently underway to move to a refcounted stack to fix this class of bugs: see the "Reference-counted argument stack" section in this document.

A "fake" AV can be converted into a "real" AV by av_reify. Conversion must occur if the AV has the SVpav_REIFY flag set, which you can check for using the AvREIFY() macro. Trying to store a SV* into, or delete from, an AV usually involves an AvREIFY() check and a conversion if needed. It's giving ✨due diligence✨.

See the comments in av.h and the functions in av.c if you really want to know more. Deep lore. 📚

💜 Working with HVs 💜

To create an HV, you use this routine:

HV*  newHV();

Once the HV has been created, these operations are possible on it:

SV**  hv_store(HV*, const char* key, U32 klen, SV* val, U32 hash);
SV**  hv_fetch(HV*, const char* key, U32 klen, I32 lval);

The klen parameter is the length of the key you're passing in — and note that you cannot pass 0 as klen to tell Perl to go measure the length of the key for you. That's not a thing. The val argument contains the SV pointer to the scalar being stored, and hash is the precomputed hash value (0 if you want hv_store to calculate it for you). The lval parameter tells us whether this fetch is actually part of a store operation — and if so, a new undefined value will be added to the HV with the supplied key, and hv_fetch will return as if the value had already existed. Sneaky but iconic. 💅

Remember that hv_store and hv_fetch return SV**'s and not just SV*. To get at the scalar value, you must first dereference the return value. But — and this is the important bit — you should check to make sure the return value is not NULL before dereferencing it. 🙏

The first of these two functions checks if a hash table entry exists, and the second deletes it.

bool  hv_exists(HV*, const char* key, U32 klen);
SV*   hv_delete(HV*, const char* key, U32 klen, I32 flags);

If flags does not include the G_DISCARD flag, then hv_delete will create and return a mortal copy of the deleted value. 🎀

And more miscellaneous functions:

void   hv_clear(HV*);
void   hv_undef(HV*);

Like their AV counterparts, hv_clear deletes all the entries in the hash table but does not actually delete the hash table itself. The hv_undef deletes both the entries and the hash table itself. She takes everything with her. 💀

Perl keeps the actual data in a linked list of structures with a typedef of HE. These contain the actual key and value pointers (plus some extra administrative overhead). The key is a string pointer; the value is an SV*. However, once you've got an HE*, to get at the actual key and value, you use the routines specified below.

           I32    hv_iterinit(HV*);
                   /* Prepares starting point to traverse hash table */
           HE*    hv_iternext(HV*);
                   /* Get the next entry, and return a pointer to a
                      structure that has both the key and value */
           char*  hv_iterkey(HE* entry, I32* retlen);
                   /* Get the key from an HE structure and also return
                      the length of the key string */
           SV*    hv_iterval(HV*, HE* entry);
                   /* Return an SV pointer to the value of the HE
                      structure */
           SV*    hv_iternextsv(HV*, char** key, I32* retlen);
                   /* This convenience routine combines hv_iternext,
                      hv_iterkey, and hv_iterval.  The key and retlen
                      arguments are return values for the key and its
                      length.  The value is returned in the SV* argument */

okay so walking a whole hash is literally just a little road trip 💅 hv_iterinit sets you up at the starting line, hv_iternext keeps handing you the next entry as an HE*, and then you grab the key with hv_iterkey (which also drops the key's length into retlen for you, bestie, so you're not out here calling strlen) and the value with hv_iterval. And if you can't be bothered to do all three steps yourself, hv_iternextsv is the combo meal: it does hv_iternext, hv_iterkey and hv_iterval in one go, giving you back the key and its length through the key and retlen arguments, and the value as the returned SV*. iconic, no notes ✨

If you already know the name of a hash variable and just want a pointer to its HV, here's the move 👇

    HV*  get_hv("package::varname", 0);

It hands you back NULL if that variable doesn't exist, so like... check it, bestie 🫶

The hash algorithm itself lives in the PERL_HASH macro:

    PERL_HASH(hash, key, klen)

⚠️ fr fr important: the exact implementation of this macro changes depending on your architecture and your version of perl, and the return value can literally change between invocations. So that value is only valid for the duration of a single perl process. Do not store it, do not pass it around, do not make it your whole personality. 💀

And if you're working with tied hashes, go read "Understanding the Magic of Tied Hashes and Arrays" for how to use the hash access functions on them. 🧚

✨ Hash API Extensions ✨

Starting from version 5.004, these functions said "we're joining the group chat too":

    HE*     hv_fetch_ent  (HV* tb, SV* key, I32 lval, U32 hash);
    HE*     hv_store_ent  (HV* tb, SV* key, SV* val, U32 hash);

    bool    hv_exists_ent (HV* tb, SV* key, U32 hash);
    SV*     hv_delete_ent (HV* tb, SV* key, I32 flags, U32 hash);

    SV*     hv_iterkeysv  (HE* entry);

the way these take SV* keys instead of plain strings is such a glow-up for anyone writing extension code that deals with hash structures 💖 It also means you can pass SV* keys straight to tie functions without being forced to stringify them first (unlike the previous batch of functions, which lowkey made you do the most).

They also return and accept whole hash entries (HE*), which is so much more efficient because the hash number for a given string doesn't have to be recalculated every single time. Head to perlapi for the detailed descriptions — iykyk. 😌

Now these macros are the only correct way to peek inside hash entries:

    HePV(HE* he, STRLEN len)
    HeVAL(HE* he)
    HeHASH(HE* he)
    HeSVKEY(HE* he)
    HeSVKEY_force(HE* he)
    HeSVKEY_set(HE* he, SV* sv)

🚨 bestie listen: the arguments to these macros have to be simple variables, because they may get evaluated more than once. No fancy expressions in there, okay? perlapi has the full tea on all of them.

And these two are the lower-level ones — they exist, sure, but only reach for them when you're dealing with keys that are not SV*s:

    HeKEY(HE* he)
    HeKLEN(HE* he)

Not me about to let you leak memory — okay so heads up: both hv_store and hv_store_ent do not bump the reference count of the val you stored. That's your job, caller. And if either of these functions returns a NULL value, you'll usually have to decrement the reference count of val yourself to avoid a memory leak. 👑 (We'll get to refcounts below, don't worry.)

🌸 AVs, HVs and undefined values 🌸

Sometimes you gotta store undefined values in AVs or HVs. Rare? Sure. Tricky? so tricky. Because your instinct is just to reach for &PL_sv_undef whenever you need an undefined SV, right?

Like, your brain goes: this XS code

    AV *av = newAV();
    av_store( av, 0, &PL_sv_undef );

is totally the same as this Perl code

    my @av;
    $av[0] = undef;

and unfortunately that's just not true 😭 Here's why:

• In perl 5.18 and earlier, AVs use &PL_sv_undef as a marker to say "this array element hasn't been initialized yet". So exists $av[0] would be true in the Perl code above, but false for the array the XS code made. Same syntax, different lore. 💀
• In perl 5.20, storing &PL_sv_undef creates a read-only element, because the scalar &PL_sv_undef itself gets stored — not a copy of it.

And the same kind of chaos happens when you stash &PL_sv_undef in an HV:

    hv_store( hv, "key", 3, &PL_sv_undef, 0 );

Yeah it makes the value undef, fine, but the second you try to modify key's value you get hit with:

    Modification of non-creatable hash value attempted

Also in perl 5.8.0, &PL_sv_undef was used to mark placeholders in restricted hashes, which meant those hash entries wouldn't show up when you iterated over the hash or checked for keys with hv_exists. rude, honestly.

You can trip over the same nonsense if you store &PL_sv_yes or &PL_sv_no into AVs or HVs. Trying to modify those gives you:

    Modification of a read-only value attempted

So like, to make a long story short 🫶 you can use the special variables &PL_sv_undef, &PL_sv_yes and &PL_sv_no with AVs and HVs — but only if you genuinely know what you're doing. Main character energy requires main character knowledge, periodt.

Generally, if you want an undefined value in an AV or HV, do not use &PL_sv_undef. Make a fresh undefined value with the newSV function instead, like:

    av_store( av, 42, newSV(0) );
    hv_store( hv, "foo", 3, newSV(0), 0 );

💜 References 💜

References are a special type of scalar that point to other data types — including other references. It's giving linked list but make it fashion. 🦋

To make one, use either of these:

    SV* newRV_inc((SV*) thing);
    SV* newRV_noinc((SV*) thing);

The thing argument can be an SV*, an AV*, or an HV*. These two functions are identical except that newRV_inc increments the reference count of thing and newRV_noinc does not. And for purely historical reasons, newRV is just a synonym for newRV_inc. (Legacy naming, we love her. 💅)

Once you've got a reference, dereference it with this macro:

    SvRV(SV*)

...then call whatever routine you need, casting the returned SV* to an AV* or an HV* if you have to.

Wanna know if an SV is a reference? Vibe check incoming:

    SvROK(SV*)

And to find out what type of value the reference points to, use this and then check what came back:

    SvTYPE(SvRV(SV*))

The most useful types you'll get back are:

• SVt_PVAV — Array
• SVt_PVHV — Hash
• SVt_PVCV — Code
• SVt_PVGV — Glob (possibly a file handle)

Any numerical value returned that's less than SVt_PVAV is some form of scalar. See "svtype" in perlapi for more details. ✨

👑 Blessed References and Class Objects 👑

References are also how perl does object-oriented programming. In perl's OO lexicon, an object is simply a reference that has been blessed into a package (or class). Once it's blessed, the programmer can use that reference to reach all the various methods in the class. That's it, that's the whole thing. 🎀

Bless a reference into a package with:

    SV* sv_bless(SV* sv, HV* stash);

The sv argument must be a reference value. The stash argument says which class that reference is gonna belong to. See "Stashes and Globs" for how to turn class names into stashes.

    /* Still under construction */

The following function upgrades rv to a reference if it isn't one already. It creates a new SV for rv to point to. If classname is non-null, the SV is blessed into the specified class. The SV is what gets returned.

        SV* newSVrv(SV* rv, const char* classname);

The following three functions copy an integer, an unsigned integer, or a double into an SV whose reference is rv. The SV is blessed if classname is non-null.

        SV* sv_setref_iv(SV* rv, const char* classname, IV iv);
        SV* sv_setref_uv(SV* rv, const char* classname, UV uv);
        SV* sv_setref_nv(SV* rv, const char* classname, NV iv);

The next function copies the pointer value — that's the address, not the string, bestie!! — into an SV whose reference is rv. The SV is blessed if classname is non-null.

        SV* sv_setref_pv(SV* rv, const char* classname, void* pv);

The following function copies a string into an SV whose reference is rv. Set length to 0 to let Perl work out the string length itself. The SV is blessed if classname is non-null.

    SV* sv_setref_pvn(SV* rv, const char* classname, char* pv,
                                                         STRLEN length);

Now, tests! 👇 The following function checks whether the SV is blessed into the specified class. Note that it does not check inheritance relationships.

        int  sv_isa(SV* sv, const char* name);

The following function checks whether the SV is a reference to a blessed object.

        int  sv_isobject(SV* sv);

The following function checks whether the SV is derived from the specified class. The SV can be either a reference to a blessed object or a string containing a class name. This is the function that implements the UNIVERSAL::isa functionality.

        bool sv_derived_from(SV* sv, const char* name);

So to check whether you've got an object derived from a specific class, you have to write it like this:

        if (sv_isobject(sv) && sv_derived_from(sv, class)) { ... }

🎀 Creating New Variables 🎀

Wanna create a brand new Perl variable with an undef value that your Perl script can actually get to? Use whichever of these matches the variable type you want:

    SV*  get_sv("package::varname", GV_ADD);
    AV*  get_av("package::varname", GV_ADD);
    HV*  get_hv("package::varname", GV_ADD);

Notice the GV_ADD in the second parameter spot, okay? That's the whole trick. Now your new variable can be set using whichever routines fit the data type. Slay. ✨

There are extra macros whose values you can bitwise OR with that GV_ADD argument to switch on certain bonus features. Those bits are:

• GV_ADDMULTI — marks the variable as multiply defined, which stops this warning from happening:

      Name <varname> used only once: possible typo

• GV_ADDWARN — throws this warning:

      Had to create <varname> unexpectedly

...if the variable didn't exist before the function was called.

And if you don't specify a package name at all, the variable gets created in the current package. 💖

💀 Reference Counts and Mortality 💀

Perl does garbage collection with a reference count-driven mechanism. SVs, AVs or HVs (we'll call 'em xV from here on) come into this world with a reference count of 1. If an xV's reference count ever drops to 0, it gets destroyed and its memory becomes available for reuse. That's the circle of life, bestie. 🦋

At the most basic internal level, you can mess with reference counts using these macros:

    int SvREFCNT(SV* sv);
    SV* SvREFCNT_inc(SV* sv);
    void SvREFCNT_dec(SV* sv);

(There are also suffixed versions of the increment and decrement macros, for situations where you're willing to trade away the full generality of these basic macros in exchange for a little performance boost. Speedrun strats. ⭐)

Okay so real talk, the refcount itself? Not the whole story. Ownership is the whole story. A reference to an xV can be owned by literally a whole cast of characters: another xV, the Perl interpreter itself, an XS data structure, a piece of running code, or a dynamic scope. And the xV? It has zero clue who owns it bestie. It just knows how many references there are (that's the reference count) and that's it. It's giving "I don't know her" energy.

So to keep reference counts correct, you gotta track what references your XS code is touching. Always know where a reference came from, who owns it, and stay aware of any references being created, destroyed, or having their ownership transferred. Because ownership isn't explicitly represented in the xV data structures, only the reference count actually has to be maintained by the code — which means all that ownership knowledge you're carrying around? Not actually evident in the code. 🙈 Example: transferring ownership of a reference from one owner to another doesn't change the reference count at all, so it can be achieved with literally no code. (The transferring code doesn't touch the referenced object, but it does need to make sure the former owner knows it no longer owns the reference, and the new owner knows it does.)

An xV that's visible at the Perl level should not become unreferenced and get destroyed. Normally an object only becomes unreferenced when it's no longer visible, often by the same means that makes it invisible. For example: a Perl reference value (RV) owns a reference to its referent, so if the RV gets overwritten, that reference gets destroyed, and the no-longer-reachable referent may get destroyed as a result. 💀

Lots of functions have some kind of reference manipulation as part of their whole deal. Sometimes that's documented in terms of ownership of references, and sometimes it's (less helpfully, ngl) documented in terms of changes to reference counts. Example: newRV_inc() is documented to create a new RV (with reference count 1) and increment the reference count of the referent that was supplied by the caller. Best way to understand it: you're creating a new reference to the referent, owned by the created RV, and returning to the caller ownership of the sole reference to the RV. Meanwhile newRV_noinc() does not increment the reference count of the referent, but the RV still ends up owning a reference to the referent. So the implication is that the caller of newRV_noinc() is relinquishing a reference to the referent — making this conceptually a more complicated operation even though it does less to the data structures. The audacity, honestly. 😭

Okay imagine this: you want to return a reference from an XSUB function. Inside the XSUB routine, you create an SV which initially has just a single reference, owned by the XSUB routine. That reference needs to be disposed of before the routine is complete — otherwise it will leak, preventing the SV from ever being destroyed. So to create an RV referencing the SV, it's most convenient to pass the SV to newRV_noinc(), which consumes that reference. Now the XSUB routine no longer owns a reference to the SV, but it does own a reference to the RV, which in turn owns a reference to the SV. Then the ownership of the reference to the RV gets transferred by the process of returning the RV from the XSUB. Clean. Slay. ✨

There are some convenience functions that can help with destroying xVs, and these introduce the concept of "mortality". A lot of documentation talks about an xV itself being mortal, but that's misleading bestie. It's really a reference to an xV that's mortal, and there can be more than one mortal reference to a single xV. For a reference to be mortal means it's owned by the temps stack — one of perl's many internal stacks — which will destroy that reference "a short time later". Usually that "short time later" is the end of the current Perl statement. But it gets messier around dynamic scopes: there can be multiple sets of mortal references hanging around at the same time, with different death dates. 👀 Internally, the actual determinant for when mortal xV references get destroyed depends on two macros, SAVETMPS and FREETMPS. See perlcall and perlxs and "Temporaries Stack" below for more on these macros.

Mortal references are mainly used for xVs that get placed on perl's main stack. And the stack is problematic for reference tracking, because it holds a lot of xV references but doesn't own those references — they are not counted. Currently there are many bugs resulting from xVs being destroyed while referenced by the stack, because the stack's uncounted references aren't enough to keep the xVs alive. 🚩 So when you put an (uncounted) reference on the stack, it is vitally important to make sure there will be a counted reference to the same xV that lasts at least as long as the uncounted reference. But it's also important that counted reference gets cleaned up at an appropriate time, and doesn't unduly prolong the xV's life. The best way to satisfy this requirement is often for there to be a mortal reference — especially if the xV was created specifically to be put on the stack and would otherwise be unreferenced.

To create a mortal reference, use these functions:

SV*  sv_newmortal()
SV*  sv_mortalcopy(SV*)
SV*  sv_2mortal(SV*)

sv_newmortal() creates an SV (with the undefined value) whose sole reference is mortal. sv_mortalcopy() creates an xV whose value is a copy of a supplied xV and whose sole reference is mortal. sv_2mortal() mortalises an existing xV reference: it transfers ownership of a reference from the caller to the temps stack. And because sv_newmortal gives the new SV no value, you normally have to hand it one via sv_setpv, sv_setiv, etc.:

SV *tmp = sv_newmortal();
sv_setiv(tmp, an_integer);

Since that's multiple C statements, you'll see this idiom alllll the time instead:

SV *tmp = sv_2mortal(newSViv(an_integer));

And heads up, the mortal routines aren't just for SVs — AVs and HVs can be made mortal too by passing their address (type-casted to SV*) to the sv_2mortal or sv_mortalcopy routines. 🫶

---

✨ Stashes and Globs ✨

A stash is a hash that contains all the variables defined within a package. Each key of the stash is a symbol name (shared by all the different types of objects that have the same name), and each value in the hash table is a GV (Glob Value). That GV in turn contains references to the various objects of that name, including (but not limited to) the following:

Scalar Value
Array Value
Hash Value
I/O Handle
Format
Subroutine

There's a single stash called PL_defstash that holds the items existing in the main package. To get at the items in other packages, append the string :: to the package name. The items in the Foo package live in the stash Foo:: inside PL_defstash. The items in the Bar::Baz package live in the stash Baz:: in Bar::'s stash.

To get the stash pointer for a particular package, use the function:

HV*  gv_stashpv(const char* name, I32 flags)
HV*  gv_stashsv(SV*, I32 flags)

The first one takes a literal string, the second uses the string stored in the SV. Remember: a stash is just a hash table, so you get back an HV*. The flags flag will create a new package if it's set to GV_ADD.

The name that gv_stash*v wants is the name of the package whose symbol table you want. The default package is called main. If you have multiply nested packages, pass their names to gv_stash*v, separated by :: just like in the Perl language itself.

Alternatively, if you have an SV that's a blessed reference, you can find out the stash pointer by using:

HV*  SvSTASH(SvRV(SV*));

then use this to get the package name itself:

char*  HvNAME(HV* stash);

And if you need to bless or re-bless an object you can use this function:

SV*  sv_bless(SV*, HV* stash)

where the first argument, an SV*, must be a reference, and the second argument is a stash. The returned SV* can now be used in the same way as any other SV. 💖

For more info on references and blessings, go consult perlref.

---

🌸 I/O Handles 🌸

Like AVs and HVs, IO objects are another type of non-scalar SV which may contain input and output PerlIO objects, or a DIR * from opendir().

You can create a new IO object:

IO*  newIO();

But unlike other SVs, a new IO object is automatically blessed into the IO::File class. Iconic behaviour, honestly. 👑

The IO object contains an input and output PerlIO handle:

PerlIO *IoIFP(IO *io);
PerlIO *IoOFP(IO *io);

Typically, if the IO object has been opened on a file, the input handle is always present, but the output handle is only present if the file is open for output. For a file, if both are present they will be the same PerlIO object.

Distinct input and output PerlIO objects are created for sockets and character devices.

The IO object also contains other data associated with Perl I/O handles:

IV IoLINES(io);                /* $. */
IV IoPAGE(io);                 /* $% */
IV IoPAGE_LEN(io);             /* $= */
IV IoLINES_LEFT(io);           /* $- */
char *IoTOP_NAME(io);          /* $^ */
GV *IoTOP_GV(io);              /* $^ */
char *IoFMT_NAME(io);          /* $~ */
GV *IoFMT_GV(io);              /* $~ */
char *IoBOTTOM_NAME(io);
GV *IoBOTTOM_GV(io);
char IoTYPE(io);
U8 IoFLAGS(io);

Most of these are involved with formats.

IoFLAGs() may contain a combination of flags, the most interesting of which are IOf_FLUSH ($|) for autoflush and IOf_UNTAINT, settable with IO::Handle's untaint() method. 🔥

The IO object may also contain a directory handle:

DIR *IoDIRP(io);

suitable for use with PerlDir_read() etc.

And okay, important one: all of these accessors macros are lvalues — there are no distinct _set() macros to modify the members of the IO object. So don't go looking for them.

---

🦋 Double-Typed SVs 🦋

Scalar variables normally contain only one type of value — an integer, double, pointer, or reference. And Perl will automatically convert the actual scalar data from the stored type into the requested type. ✨

But some scalar variables contain more than one type of scalar data. For example, the variable $! contains either the numeric value of errno or its string equivalent from either strerror or sys_errlist[].

To force multiple data values into an SV, you gotta do two things: use the sv_set*v routines to add the additional scalar type, then set a flag so that Perl will believe it contains more than one type of data. The four macros to set the flags are:

               SvIOK_on
               SvNOK_on
               SvPOK_on
               SvROK_on

Okay so which one of these do you actually reach for? It depends entirely on which sv_set*v routine you called first, bestie. Every single sv_set*v routine flips on the bit for the type of data it's setting and shuts off all the others. So whichever you did last is the only type the SV thinks it is. Rude, honestly, but that's the deal.

Say you want a fresh Perl variable called dberror holding both the numeric error code and the worded error string — iconic dual-wield. You'd do:

    extern int  dberror;
    extern char *dberror_list;

    SV* sv = get_sv("dberror", GV_ADD);
    sv_setiv(sv, (IV) dberror);
    sv_setpv(sv, dberror_list[dberror]);
    SvIOK_on(sv);

And if you'd flipped the order of sv_setiv and sv_setpv? Then you'd have to call SvPOK_on instead of SvIOK_on. Order matters, periodt. 💅

---

🪄 Magic Variables 🪄

[This section still under construction. Ignore everything here. Post no bills. Everything not permitted is forbidden.]

Any SV can be magical — meaning it's got special features a boring-ass normal SV doesn't have. These features live in the SV structure as a linked list of struct magics, typedef'ed to MAGIC.

    struct magic {
        MAGIC*      mg_moremagic;
        MGVTBL*     mg_virtual;
        U16         mg_private;
        char        mg_type;
        U8          mg_flags;
        I32         mg_len;
        SV*         mg_obj;
        char*       mg_ptr;
    };

Heads up bestie: this is current as of patchlevel 0 and it could change at literally any time. 🫠

---

✨ Assigning Magic ✨

Perl hands magic to an SV with the sv_magic function:

  void sv_magic(SV* sv, SV* obj, int how, const char* name, I32 namlen);

The sv argument is a pointer to the SV that's about to get a brand-new magical feature. Main character upgrade, fr.

If sv isn't already magical, Perl uses the SvUPGRADE macro to convert sv to type SVt_PVMG. Then it keeps going and adds the new magic to the beginning of the linked list of magical features. Any previous entry of the same type of magic gets deleted. Small caveat tho: that can be overridden, so multiple instances of the same type of magic can be attached to one SV. Sharing is caring 🫶

The name and namlen arguments are how you attach a string to the magic — usually the name of a variable. namlen gets stored in the mg_len field, and if name is non-null then either a savepvn copy of name or name itself lands in the mg_ptr field — which one depends on whether namlen is greater than zero or equal to zero, respectively. And as a special little treat: if (name && namlen == HEf_SVKEY) then name is assumed to hold an SV* and gets stored as-is with its REFCNT incremented.

Then there's how, which tells sv_magic which — if any — predefined "Magic Virtual Table" should go in the mg_virtual field. (Scroll down for the "Magic Virtual Tables" section, it's right there.) The how argument also gets stored in the mg_type field, and you should pick its value from the set of PERL_MAGIC_foo macros in perl.h. Fun history moment: before those macros existed, Perl internals just raw-dogged character literals, so you might stumble across ancient code or docs talking about 'U' magic instead of PERL_MAGIC_uvar. Vintage 💀

The obj argument goes into the mg_obj field of the MAGIC structure. If it's not the same as the sv argument, the reference count of obj gets incremented. But if it is the same, or if how is PERL_MAGIC_arylen, PERL_MAGIC_regdatum, PERL_MAGIC_regdata, or if it's a NULL pointer, then obj is just stored — no reference count bump. No extra credit, no tax.

Also go look at sv_magicext in perlapi if you want a more flexible way to add magic to an SV. Options, bestie. ✨

And there's a function for adding magic to an HV too:

    void hv_magic(HV *hv, GV *gv, int how);

It literally just calls sv_magic and coerces the gv argument into an SV. Efficiency queen.

Want to remove magic from an SV? Call sv_unmagic:

    int sv_unmagic(SV *sv, int type);

The type argument should equal the how value from back when the SV first got made magical.

BUT — important — sv_unmagic removes all magic of a given type from the SV. If you only want to remove certain magic of a type based on the magic virtual table, you want sv_unmagicext instead:

    int sv_unmagicext(SV *sv, int type, MGVTBL *vtbl);

---

🌈 Magic Virtual Tables 🌈

The mg_virtual field in the MAGIC structure is a pointer to an MGVTBL, which is a structure full of function pointers — it stands for "Magic Virtual Table" and it handles all the various operations that might get applied to that variable. It's giving middle management but make it sorcery.

The MGVTBL has five (or sometimes eight) pointers to these routine types:

    int  (*svt_get)  (pTHX_ SV* sv, MAGIC* mg);
    int  (*svt_set)  (pTHX_ SV* sv, MAGIC* mg);
    U32  (*svt_len)  (pTHX_ SV* sv, MAGIC* mg);
    int  (*svt_clear)(pTHX_ SV* sv, MAGIC* mg);
    int  (*svt_free) (pTHX_ SV* sv, MAGIC* mg);

    int  (*svt_copy) (pTHX_ SV *sv, MAGIC* mg, SV *nsv,
                                          const char *name, I32 namlen);
    int  (*svt_dup)  (pTHX_ MAGIC *mg, CLONE_PARAMS *param);
    int  (*svt_local)(pTHX_ SV *nsv, MAGIC *mg);

This MGVTBL structure gets set at compile-time in perl.h, and there are currently 32 types. Each of these different structures points at various routines that do extra stuff depending on which function is being called.

So here's the breakdown, bestie:

• svt_get — do something before the value of the SV is retrieved.
• svt_set — do something after the SV is assigned a value.
• svt_len — report on the SV's length.
• svt_clear — clear something the SV represents.
• svt_free — free any extra storage associated with the SV.
• svt_copy — copy tied variable magic to a tied element.
• svt_dup — duplicate a magic structure during thread cloning.
• svt_local — copy magic to local value during 'local'.

For instance, the MGVTBL structure called vtbl_sv (which corresponds to an mg_type of PERL_MAGIC_sv) contains:

    { magic_get, magic_set, magic_len, 0, 0 }

So when an SV gets flagged as magical and its type is PERL_MAGIC_sv, and a get operation is happening, the routine magic_get gets called. Every single one of the routines for the various magical types starts with magic_. NOTE: the magic routines are not considered part of the Perl API, and may not be exported by the Perl library. So don't come for them. 🙈

Those last three slots are a recent addition, and for source code compatibility they're only checked for if one of the three flags MGf_COPY, MGf_DUP, or MGf_LOCAL is set in mg_flags. Which means most code can keep declaring a vtable as a good old 5-element value with zero drama. Those three are currently used exclusively by the threading code, and are highly subject to change. Don't get attached, ngl. 😭

The current kinds of Magic Virtual Tables are:

…and here's the rest of the char / magic macro / vtbl list, one per line so your eyeballs don't just give up on you 💅

• \0 · PERL_MAGIC_sv · vtbl vtbl_sv — ✨ Special scalar variable ✨
• # · PERL_MAGIC_arylen · vtbl vtbl_arylen — array length, aka $#ary
• % · PERL_MAGIC_rhash · (none) — extra data for restricted hashes
• * · PERL_MAGIC_debugvar · vtbl vtbl_debugvar — $DB::single, signal, trace vars
• . · PERL_MAGIC_pos · vtbl vtbl_pos — pos() lvalue
• : · PERL_MAGIC_symtab · (none) — extra data for symbol tables
• < · PERL_MAGIC_backref · vtbl vtbl_backref — for weak ref data
• @ · PERL_MAGIC_arylen_p · (none) — to move arylen out of XPVAV
• B · PERL_MAGIC_bm · vtbl vtbl_regexp — Boyer-Moore (fast string search) 🔥
• c · PERL_MAGIC_overload_table · vtbl vtbl_ovrld — holds overload table (AMT) on stash
• D · PERL_MAGIC_regdata · vtbl vtbl_regdata — regex match position data (@+ and @- vars)
• d · PERL_MAGIC_regdatum · vtbl vtbl_regdatum — regex match position data element
• E · PERL_MAGIC_env · vtbl vtbl_env — the %ENV hash
• e · PERL_MAGIC_envelem · vtbl vtbl_envelem — one %ENV hash element
• f · PERL_MAGIC_fm · vtbl vtbl_regexp — formline, the 'compiled' format
• g · PERL_MAGIC_regex_global · vtbl vtbl_mglob — m//g target
• H · PERL_MAGIC_hints · vtbl vtbl_hints — the %^H hash
• h · PERL_MAGIC_hintselem · vtbl vtbl_hintselem — one %^H hash element
• I · PERL_MAGIC_isa · vtbl vtbl_isa — the @ISA array
• i · PERL_MAGIC_isaelem · vtbl vtbl_isaelem — one @ISA array element
• k · PERL_MAGIC_nkeys · vtbl vtbl_nkeys — scalar(keys()) lvalue
• L · PERL_MAGIC_dbfile · (none) — debugger %_<filename
• l · PERL_MAGIC_dbline · vtbl vtbl_dbline — debugger %_<filename element
• N · PERL_MAGIC_shared · (none) — shared between threads
• n · PERL_MAGIC_shared_scalar · (none) — shared between threads
• o · PERL_MAGIC_collxfrm · vtbl vtbl_collxfrm — locale transformation
• P · PERL_MAGIC_tied · vtbl vtbl_pack — tied array or hash
• p · PERL_MAGIC_tiedelem · vtbl vtbl_packelem — tied array or hash element
• q · PERL_MAGIC_tiedscalar · vtbl vtbl_packelem — tied scalar or handle
• r · PERL_MAGIC_qr · vtbl vtbl_regexp — precompiled qr// regex
• S · PERL_MAGIC_sig · vtbl vtbl_sig — the %SIG hash
• s · PERL_MAGIC_sigelem · vtbl vtbl_sigelem — one %SIG hash element
• t · PERL_MAGIC_taint · vtbl vtbl_taint — taintedness
• U · PERL_MAGIC_uvar · vtbl vtbl_uvar — available for use by extensions
• u · PERL_MAGIC_uvar_elem · (none) — reserved for use by extensions
• V · PERL_MAGIC_vstring · (none) — SV was a vstring literal
• v · PERL_MAGIC_vec · vtbl vtbl_vec — vec() lvalue
• w · PERL_MAGIC_utf8 · vtbl vtbl_utf8 — cached UTF-8 information
• X · PERL_MAGIC_destruct · vtbl vtbl_destruct — destruct callback
• x · PERL_MAGIC_substr · vtbl vtbl_substr — substr() lvalue
• Y · PERL_MAGIC_nonelem · vtbl vtbl_nonelem — array element that does not exist
• y · PERL_MAGIC_defelem · vtbl vtbl_defelem — shadow "foreach" iterator variable / smart parameter vivification
• Z · PERL_MAGIC_hook · vtbl vtbl_hook — the %{^HOOK} hash
• z · PERL_MAGIC_hookelem · vtbl vtbl_hookelem — one %{^HOOK} hash element
• \ · PERL_MAGIC_lvref · vtbl vtbl_lvref — lvalue reference constructor
• ] · PERL_MAGIC_checkcall · vtbl vtbl_checkcall — inlining/mutation of call to this CV
• ^ · PERL_MAGIC_extvalue · (none) — value magic available for use by extensions
• ~ · PERL_MAGIC_ext · (none) — variable magic available for use by extensions

---

okay so the case thing 🔤 — when an uppercase AND a lowercase letter both show up in that table, the uppercase one is typically the whole composite thing (a list or a hash) and the lowercase one is one element of that composite. Some internals code straight up leans on this case relationship, iykyk. BUT — and this is important bestie — 'v' and 'V' (that's vec and v-string) are in no way related. Not even a little. Don't ship them. 💀

The PERL_MAGIC_ext, PERL_MAGIC_extvalue and PERL_MAGIC_uvar magic types are defined specifically for use by extensions, and perl itself will not use them. Extensions can use PERL_MAGIC_ext or PERL_MAGIC_extvalue magic to 'attach' private information to variables (typically objects). This is especially useful because there's genuinely no way for normal perl code to corrupt that private info (unlike, say, using extra elements of a hash object, which is giving chaos 🙈). PERL_MAGIC_extvalue is value magic (unlike PERL_MAGIC_ext and PERL_MAGIC_uvar), which means that on localization the new value will not be magical.

Similarly, PERL_MAGIC_uvar magic can be used much like tie() to call a C function any time a scalar's value is used or changed. The MAGIC's mg_ptr field points to a ufuncs structure:

struct ufuncs {
    I32 (*uf_val)(pTHX_ IV, SV*);
    I32 (*uf_set)(pTHX_ IV, SV*);
    IV uf_index;
};

When the SV is read from or written to, the uf_val or uf_set function will be called with uf_index as the first arg and a pointer to the SV as the second. A simple example of how to add PERL_MAGIC_uvar magic is shown below. Note that the ufuncs structure is copied by sv_magic, so you can safely allocate it on the stack. 🫶

void
Umagic(sv)
    SV *sv;
PREINIT:
    struct ufuncs uf;
CODE:
    uf.uf_val   = &my_get_fn;
    uf.uf_set   = &my_set_fn;
    uf.uf_index = 0;
    sv_magic(sv, 0, PERL_MAGIC_uvar, (char*)&uf, sizeof(uf));

Attaching PERL_MAGIC_uvar to arrays is permissible but has no effect. (permissible and useless, we love a low-effort icon 🎀)

For hashes there is a specialized hook that gives control over hash keys (but not values). This hook calls PERL_MAGIC_uvar 'get' magic if the "set" function in the ufuncs structure is NULL. The hook is activated whenever the hash is accessed with a key specified as an SV through the functions hv_store_ent, hv_fetch_ent, hv_delete_ent, and hv_exists_ent. Accessing the key as a string through the functions without the ..._ent suffix circumvents the hook. See "GUTS" in Hash::Util::FieldHash for a detailed description. 👑

Note that because multiple extensions may be using PERL_MAGIC_ext or PERL_MAGIC_uvar magic, it is important for extensions to take extra care to avoid conflict. Typically only using the magic on objects blessed into the same class as the extension is sufficient. For PERL_MAGIC_ext magic, it is usually a good idea to define an MGVTBL, even if all its fields will be 0, so that individual MAGIC pointers can be identified as a particular kind of magic using their magic virtual table. mg_findext provides an easy way to do that: ✨

static MGVTBL my_vtbl = { 0, 0, 0, 0, 0, 0, 0, 0 };

MAGIC *mg;
if ((mg = mg_findext(sv, PERL_MAGIC_ext, &my_vtbl))) {
    /* this is really ours, not another module's PERL_MAGIC_ext */
    my_priv_data_t *priv = (my_priv_data_t *)mg->mg_ptr;
    ...
}

Also note that the sv_set*() and sv_cat*() functions described earlier do not invoke 'set' magic on their targets. This must be done by the user, either by calling the SvSETMAGIC() macro after calling these functions, or by using one of the sv_set*_mg() or sv_cat*_mg() functions. Similarly, generic C code must call the SvGETMAGIC() macro to invoke any 'get' magic if they use an SV obtained from external sources in functions that don't handle magic. See perlapi for a description of these functions. For example, calls to the sv_cat*() functions typically need to be followed by SvSETMAGIC(), but they don't need a prior SvGETMAGIC() since their implementation handles 'get' magic. 💅

---

🔍 Finding Magic 🔍

MAGIC *mg_find(SV *sv, int type); /* Finds the magic pointer of that
                                   * type */

This routine returns a pointer to a MAGIC structure stored in the SV. If the SV does not have that magical feature, NULL is returned. If the SV has multiple instances of that magical feature, the first one will be returned. mg_findext can be used to find a MAGIC structure of an SV based on both its magic type and its magic virtual table: 🌈

MAGIC *mg_findext(SV *sv, int type, MGVTBL *vtbl);

Also, if the SV passed to mg_find or mg_findext is not of type SVt_PVMG, Perl may core dump. 😭

int mg_copy(SV* sv, SV* nsv, const char* key, STRLEN klen);

This routine checks to see what types of magic sv has. If the mg_type field is an uppercase letter, then the mg_obj is copied to nsv, but the mg_type field is changed to be the lowercase letter.

---

🎀 Understanding the Magic of Tied Hashes and Arrays 🎀

Tied hashes and arrays are magical beasts of the PERL_MAGIC_tied magic type. 🦋

🚨 WARNING: As of the 5.004 release, proper usage of the array and hash access functions requires understanding a few caveats. Some of these caveats are actually considered bugs in the API, to be fixed in later releases, and are bracketed with [MAYCHANGE] below. If you find yourself actually applying such information in this section, be aware that the behavior may change in the future, umm, without warning. 🙈

The perl tie function associates a variable with an object that implements the various GET, SET, etc methods. To perform the equivalent of the perl tie function from an XSUB, you must mimic this behaviour. The code below carries out the necessary steps — firstly it creates a new hash, and then creates a second hash which it blesses into the class which will implement the tie methods. Lastly it ties the two hashes together, and returns a reference to the new tied hash. Note that the code below does NOT call the TIEHASH method in the MyTie class — see "Calling Perl Routines from within C Programs" for details on how to do this. 💜

           SV*
           mytie()
           PREINIT:
               HV *hash;
               HV *stash;
               SV *tie;
           CODE:
               hash = newHV();
               tie = newRV_noinc((SV*)newHV());
               stash = gv_stashpv("MyTie", GV_ADD);
               sv_bless(tie, stash);
               hv_magic(hash, (GV*)tie, PERL_MAGIC_tied);
               SvREFCNT_dec(tie); /* hv_magic() increases tie ref count */
               RETVAL = newRV_noinc(hash);
           OUTPUT:
               RETVAL

ok so real talk about av_store 💅 when you hand it a tied array it does NOT store anything, it literally just copies the magic off the array onto the value being "stored" via mg_copy. Sneaky. It might also hand you back NULL, which is its lil way of saying "bestie, this value genuinely did not need to live in that array". [MAYCHANGE] so after you call av_store on a tied array you'll usually need to hit mg_set(val) to actually trigger the Perl-level STORE method on the TIEARRAY object. And if av_store gave you NULL? you also gotta call SvREFCNT_dec(val) or you're leaking memory, no cap. [/MAYCHANGE]

and everything I just said? applies verbatim to tied hashes too, via hv_store and hv_store_ent. same energy, different container. 💀

av_fetch and its hash besties hv_fetch and hv_fetch_ent actually return an undefined mortal value whose magic got initialized by mg_copy. ✨ key thing: that returned value does NOT need deallocating because it's already mortal, slay. [MAYCHANGE] but you will need to call mg_get() on what you got back to actually invoke the Perl-level FETCH method on the underlying TIE object. And you can also call mg_set() on the return value after assigning something suitable to it with sv_setsv, which will invoke the STORE method on the TIE object. [/MAYCHANGE]

[MAYCHANGE] in other words: the array/hash fetch/store functions don't actually fetch or store real values when you're dealing with tied arrays and hashes. they just call mg_copy to slap magic onto the values that were meant to be stored or fetched. the actual TIE-method-invoking happens later, whenever mg_get and mg_set get called. so yeah, the magic system currently does like... lazy access to arrays and hashes. very demure, very mindful. 🧚

Currently (as of perl version 5.004), using the hash and array access functions means you MUST know whether you're working with "normal" hashes/arrays or their tied variants. The API may get changed to give more transparent access to both tied and normal data types in future versions. [MAYCHANGE] [/MAYCHANGE]

ngl bestie you'd really want to understand this: the TIEARRAY and TIEHASH interfaces are just sugar for invoking Perl method calls while using the uniform hash/array syntax. And sugar has a cost — typically about two to four extra opcodes per FETCH/STORE operation, plus all the mortal variables needed to invoke those methods. if your TIE methods are substantial, that overhead is comparatively small. but if they're only a few statements long? that overhead is NOT insignificant. 💀

---

💜 Localizing changes

Perl has this genuinely iconic construction:

{
  local $var = 2;
  ...
}

which is approximately equivalent to:

{
  my $oldvar = $var;
  $var = 2;
  ...
  $var = $oldvar;
}

The biggest difference: the first one reinstates $var's original value no matter how control leaves the block — goto, return, die/eval, whatever. it's a lil bit more efficient too. 💅

And you can do the same thing from C via the Perl API: build a pseudo-block and set it so that changes automatically get undone at the end of it, whether it ends explicitly or via a non-local exit (aka die()). You make a block-ish construct with a pair of ENTER/LEAVE macros (see "Returning a Scalar" in perlcall). You can build one specifically for some important localized task, OR reuse an existing one (like the boundaries of an enclosing Perl subroutine/block, or an existing pair for freeing TMPs). (In that second case, the overhead of the extra localization has to be basically negligible.) Oh and heads up: every XSUB is automatically enclosed in an ENTER/LEAVE pair. free real estate. 🎀

Inside such a pseudo-block, all this is available to you:

• SAVEINT(int i)
• SAVEIV(IV i)
• SAVEI32(I32 i)
• SAVEI8(I8 i)
• SAVEI16(I16 i)
• SAVEBOOL(int i)
• SAVESTRLEN(STRLEN i)

— these macros set things up so the value of the integer variable i gets restored at the end of the enclosing pseudo-block. ✨

• SAVESPTR(s)
• SAVEPPTR(p)

— these macros set things up so the values of pointers s and p get restored. s has to be a pointer of a type that survives conversion to SV* and back; p should be able to survive conversion to char* and back.

• SAVERCPV(char **ppv)

— arranges to restore the value of a char * variable that was allocated with a call to rcpv_new() to its previous state when the current pseudo block finishes. The pointer stored in *ppv at the time of the call gets refcount incremented and stored on the save stack. Later, when the current pseudo-block completes, the value stored in *ppv gets refcount decremented, and the previous value gets restored from the savestack (which also gets refcount decremented).

This is the "RCPV" equivalent of SAVEGENERICSV().

• SAVEGENERICSV(SV **psv)

— arranges to restore the value of an SV * variable to its previous state when the current pseudo block finishes. The pointer stored in *psv at the time of the call gets refcount incremented and stored on the save stack. Later, when the current pseudo-block completes, the value stored in *ppv gets refcount decremented, and the previous value gets restored from the savestack (which also gets refcount decremented). This is the C equivalent of local $sv. 👑

• SAVEFREESV(SV *sv)

— the refcount of sv gets decremented at the end of the pseudo-block. This is similar to sv_2mortal in that it's also a mechanism for a delayed SvREFCNT_dec. BUT — and this part matters — while sv_2mortal extends sv's lifetime until the beginning of the next statement, SAVEFREESV extends it until the end of the enclosing scope. Those lifetimes can be wildly different. 😭

Also compare SAVEMORTALIZESV.

• SAVEMORTALIZESV(SV *sv)

— just like SAVEFREESV, except it mortalizes sv at the end of the current scope instead of decrementing its reference count. Usually the effect is keeping sv alive until the statement that called the currently live scope finishes executing.

• SAVEFREEOP(OP *op)

— the OP * gets op_free()ed at the end of the pseudo-block.

• SAVEFREEPV(p)

— the chunk of memory pointed to by p gets Safefree()ed at the end of the current pseudo-block.

• SAVEFREERCPV(char *pv)

— makes sure a char * created by a call to rcpv_new() gets rcpv_free()ed at the end of the current pseudo-block.

This is the RCPV equivalent of SAVEFREESV().

• SAVECLEARSV(SV *sv)

— clears a slot in the current scratchpad that corresponds to sv at the end of the pseudo-block.

• SAVEDELETE(HV *hv, char *key, I32 length)

— the key key of hv gets deleted at the end of the pseudo-block. The string pointed to by key gets Safefree()ed. If you've got a key sitting in short-lived storage, you can reallocate the string like this:

SAVEDELETE(PL_defstash, savepv(tmpbuf), strlen(tmpbuf));

• SAVEDESTRUCTOR(DESTRUCTORFUNC_NOCONTEXT_t f, void *p)

— at the end of the pseudo-block, function f gets called with the single argument p, which may be NULL.

• SAVEDESTRUCTOR_X(DESTRUCTORFUNC_t f, void *p)

— at the end of the pseudo-block, function f gets called with the implicit context argument (if any), and p, which may be NULL.

⚠️ heads up: the end of the current pseudo-block may happen much later than the end of the current statement. You might wanna peek at the MORTALSVFUNC_X() macro instead.

• MORTALSVFUNC_X(SVFUNC_t f, SV *sv)

— at the end of the current statement, function f gets called with the implicit context argument (if any), and sv, which may be NULL.

⚠️ be aware: the parameter argument to the destructor function differs from the related SAVEDESTRUCTOR_X() in that it MUST be either NULL or an SV*. no vibes, no exceptions. 💅

⚠️ also: the end of the current statement may happen much before the end of the current pseudo-block. You might wanna peek at the SAVEDESTRUCTOR_X() macro instead.

• MORTALDESTRUCTOR_SV(SV *coderef, SV *args)

— at the end of the current statement, the Perl function inside coderef gets called with the arguments provided (if any) in args. See the docs for mortal_destructor_sv() for the details on how the args parameter is handled.

⚠️ again: the end of the current statement may happen much before the end of the current pseudo-block. If you wanna call a Perl function at the end of the current pseudo block, use the SAVEDESTRUCTOR_X() API instead — which will mean you gotta write a C wrapper to call the Perl function. sorry bestie, that's just how it is. 🙈

• SAVESTACK_POS()

— the current offset on the Perl internal stack (cf. SP) gets restored at the end of the pseudo-block.

---

okay so the API list below is made of functions, which means you have to hand over pointers to the modifiable data explicitly (either C pointers, or Perlish GV *s). And wherever the macros above take int, the equivalent function takes int *. got it? ✨

some of the other macros up there do have functions implementing them, but honestly it's probably best to just use the macro, and NOT those functions or the ones coming below. 💅

• SV* save_scalar(GV *gv)

— equivalent to the Perl code local $gv.

• AV* save_ary(GV *gv)
• HV* save_hash(GV *gv)

— similar to save_scalar, but they localize @gv and %gv.

• void save_item(SV *item)

— duplicates the current value of SV. On exit from the current ENTER/LEAVE pseudo-block, the value of SV gets restored using the stored value. It does not handle magic. Use save_scalar if magic is affected. 🩷

• SV* save_svref(SV **sptr)

— similar to save_scalar, but it reinstates an SV *.

• void save_aptr(AV **aptr)
• void save_hptr(HV **hptr)

— similar to save_svref, but they localize AV * and HV *.

The Alias module implements localization of the basic types within the caller's scope. If you're curious about how to localize things in the containing scope, go take a look there too. 🦋

🔥 SUBROUTINES 🔥

okay bestie, this is the part where Perl gets its hands dirty with C. it's giving backstage-pass energy 💅

✨ XSUBs and the Argument Stack ✨

so XSUBs are literally just a cute lil way for Perl programs to reach into C subroutines. fr fr. an XSUB routine gets handed a stack that's carrying all the arguments from the Perl program, plus a whole mapping situation that translates Perl data structures into their C equivalents. iconic crossover event 🫶

you get at those stack arguments with the ST(n) macro, which hands you back the n'th stack argument. and heads up: argument 0 is the first argument passed in the Perl subroutine call, not the second, don't get it twisted. these arguments are all SV*, and you can use them anywhere an SV* is welcome. slay.

most of the time, the output from your C routine can just be handled by the RETVAL and OUTPUT directives, super chill. BUT. there are cases where the argument stack isn't already long enough to hold all the return values, and that's a whole vibe check fail. the classic example? the POSIX tzname() call — it takes no arguments but returns two: the local time zone's standard and summer time abbreviations. two for the price of zero, periodt.

to handle that lil situation, you use the PPCODE directive and extend the stack with this macro:

EXTEND(SP, num);

where SP is the macro representing the local copy of the stack pointer, and num is how many elements you want the stack extended by. ✨

now that there's room on the stack (finally, room to breathe 🧚), you push values onto it with the PUSHs macro. and ngl the pushed values will often need to be "mortal" (go read "Reference Counts and Mortality" for that whole saga):

PUSHs(sv_2mortal(newSViv(an_integer)))
PUSHs(sv_2mortal(newSVuv(an_unsigned_integer)))
PUSHs(sv_2mortal(newSVnv(a_double)))
PUSHs(sv_2mortal(newSVpv("Some String",0)))
/* Although the last example is better written as the more
 * efficient: */
PUSHs(newSVpvs_flags("Some String", SVs_TEMP))

and now when the Perl program calls tzname, those two values get assigned like this:

($standard_abbrev, $summer_abbrev) = POSIX::tzname;

there's also an alternate method for pushing values on the stack, possibly simpler, and it's this macro:

XPUSHs(SV*)

this one automatically adjusts the stack for you if it needs to. so like, you do not need to call EXTEND to extend the stack. ate and left no crumbs 💅

okay so real talk — even though earlier versions of this document were kinda suggesting it, the (X)PUSH[iunp] macros are not suited to XSUBs that return multiple results. for that, either stick with the (X)PUSHs macros shown above, or use the newer m(X)PUSH[iunp] macros instead; peep "Putting a C value on Perl stack".

want more? go consult perlxs and perlxstut. 💖

✨ Autoloading with XSUBs ✨

if an AUTOLOAD routine is an XSUB, then — just like with regular Perl subroutines — Perl drops the fully-qualified name of the autoloaded subroutine into the $AUTOLOAD variable of the XSUB's package. cute, consistent, love that for us.

but wait, there's more: it also stashes that same info in certain fields of the XSUB itself 👑

HV *stash           = CvSTASH(cv);
const char *subname = SvPVX(cv);
STRLEN name_length  = SvCUR(cv); /* in bytes */
U32 is_utf8         = SvUTF8(cv);

SvPVX(cv) holds just the sub name itself, no package prefix, none of that. and for an AUTOLOAD routine in UNIVERSAL or one of its superclasses, CvSTASH(cv) returns NULL during a method call on a nonexistent package. (mysterious behavior, we love a little drama 🙈)

Note bestie: setting $AUTOLOAD stopped working in 5.6.1, which didn't support XS AUTOLOAD subs at all. Perl 5.8.0 introduced using fields in the XSUB itself. Perl 5.16.0 restored the setting of $AUTOLOAD. so if you need to support 5.8–5.14, use the XSUB's fields. iykyk 📝

✨ Calling Perl Routines from within C Programs ✨

there are four routines you can use to call a Perl subroutine from inside a C program. these four:

I32  call_sv(SV*, I32);
I32  call_pv(const char*, I32);
I32  call_method(const char*, I32);
I32  call_argv(const char*, I32, char**);

the one you'll reach for most often is call_sv. the SV* argument holds either the name of the Perl subroutine you wanna call, or a reference to that subroutine — dealer's choice. the second argument is a pile of flags that control: what context the subroutine is called in, whether or not the sub is being passed arguments, how errors should be trapped, and how return values get treated. so many choices, main character decision energy ✨

all four routines return the number of arguments the subroutine returned on the Perl stack.

fun fact: before Perl v5.6.0 these were called perl_call_sv, etc., but those names are deprecated now — macros of the same name are provided for compatibility though, so older code doesn't just die.

when you use any of these routines (except call_argv), you the programmer must manipulate the Perl stack yourself. the squad of macros and functions involved:

dSP
SP
PUSHMARK()
PUTBACK
SPAGAIN
ENTER
SAVETMPS
FREETMPS
LEAVE
XPUSH*()
POP*()

for the full detailed description of calling conventions from C to Perl, go read perlcall. 💜

✨ Putting a C value on Perl stack ✨

a lot of opcodes (an opcode = an elementary operation in the internal perl stack machine) put an SV* onto the stack. but as an optimization, that corresponding SV is (usually) not recreated every single time. the opcodes reuse specially assigned SVs — called targets — which, as a lil corollary, are not constantly being freed and created. sustainable queen 🌱

each target is created only once (though see "Scratchpads and recursion" below, it's not that simple), and when an opcode needs to put an integer, a double, or a string on the stack, it just sets the corresponding parts of its target and puts that target on the stack. efficient, lowkey genius.

the macro that puts this target on the stack is PUSHTARG, and it gets used directly in some opcodes, plus indirectly in zillions of others which use it via (X)PUSH[iunp].

⚠️ because the target is reused, you have to be careful when pushing multiple values on the stack. this code will not do what you think it does:

XPUSHi(10);
XPUSHi(20);

this translates to: "set TARG to 10, push a pointer to TARG onto the stack; set TARG to 20, push a pointer to TARG onto the stack". and at the end of the operation, the stack does not contain the values 10 and 20 — it actually contains two pointers to TARG, which we've set to 20. two copies of the same girl. 💀

if you need to push multiple different values, then you should either use the (X)PUSHs macros, or else use the new m(X)PUSH[iunp] macros — neither of which make use of TARG. the (X)PUSHs macros simply push an SV* on the stack, which, as noted under "XSUBs and the Argument Stack", will often need to be "mortal". and the new m(X)PUSH[iunp] macros make that a little easier by creating a new mortal for you (via (X)PUSHmortal), pushing that onto the stack (extending it if necessary in the case of the mXPUSH[iunp] macros), and then setting its value. so instead of writing this to "fix" the example above:

XPUSHs(sv_2mortal(newSViv(10)))
XPUSHs(sv_2mortal(newSViv(20)))

you can simply write:

mXPUSHi(10)
mXPUSHi(20)

glow up tbh 💅

on a related note: if you do use (X)PUSH[iunp], then you're gonna need a dTARG in your variable declarations so that the *PUSH* macros can make use of the local variable TARG. see also "dTARGET" in perlapi and dXSTARG.

be careful with TARG, bestie: each non-recursive call to your OP, or XSUB for dXSTARG, will use the same SV for TARG for each call, so you can't just assume the SV is pristine — the results from the previous call to your OP or XSUB may still be sitting in TARG. 😭

other state stuff may also be set, like a cached numeric version of a PV or the cached UTF-8 length. you can dodge all of that drama by using the normal sv_set*() APIs instead of trying to optimize by directly manipulating the SV. do less, honestly.

for OPs that implement the OA_TARGLEX optimization, TARG may be special in other ways: there may be references to it, it might be blessed, it might be tied, or magical in other ways. so many secrets.

and you must not use TARG to return a reference, because that reference will be live until the next time the OP or XSUB executes non-recursively, or until perl exits. no cap. that's a real leak-flavored situation.

✨ Scratchpads ✨

okay so the question that's just been vibing in the air: when do the SVs that are targets for opcodes actually get created? and the answer is — they get created when the current unit gets compiled. that "current unit" being a subroutine, or a file (for opcodes for statements outside of subroutines). during that compilation time, a special anonymous Perl array gets created, and it's called a scratchpad for the current unit. ✨

a scratchpad keeps the SVs that are lexicals for the current unit and that are targets for opcodes.

a previous version of this document claimed you could deduce that an SV lives on a scratchpad by peeking at its flags: lexicals have SVs_PADMY set, and targets have SVs_PADTMP set. but ngl that has never been fully true. SVs_PADMY could be set on a variable that no longer resides in any pad. and while targets do have SVs_PADTMP set, it can also be set on variables that have never lived in a pad but nonetheless act like targets. confusing? yeah. as of perl 5.21.5, the SVs_PADMY flag is no longer used and is defined as 0. SvPADMY() now returns true for anything that doesn't have SVs_PADTMP. we love a simplification era 🎀

the correspondence between OPs and targets is not 1-to-1. different OPs in the compile tree of the unit can use the same target, as long as it wouldn't conflict with the expected life of the temporary. sharing is caring but responsibly.

✨ Scratchpads and recursion ✨

and actually, it's not 100% true that a compiled unit contains a pointer to the scratchpad AV. in fact it contains a pointer to an AV of (initially) one element, and that one element is the scratchpad AV. so why do we need this extra level of indirection, you ask? 🦋

the answer is recursion, and maybe threads. both of these can create several execution pointers going into the same subroutine. and so that the subroutine-child doesn't write over the temporaries for the subroutine-parent (whose lifespan covers the call to the child), the parent and the child should have different scratchpads. (and honestly the lexicals should be separate anyway! don't share!)

so every subroutine is born with an array of scratchpads (of length 1). on each entry to the subroutine, it's checked that the current depth of the recursion isn't more than the length of this array, and if it is, a new scratchpad is created and pushed into the array. growth mindset 💖

the targets on this scratchpad are undefs, but they're already marked with the correct flags. prepared and pretty.

🧠 Memory Allocation 💾

✨ Allocation ✨

okay bestie listen UP because this is the part where people get themselves into trouble 💀 All memory that's meant to be used with the Perl API functions should be manipulated using the macros described in this section. These macros give you the necessary transparency between the differences in the actual malloc implementation that's used inside perl.

🚨 NEVER pass pointers between Newx, Renew, Safefree and the libc equivalents malloc, realloc, free. They are not from the same memory pool or allocator. You'll get either an instant or a delayed SEGV, or subtle memory leaks, or subtle heap corruption. fr fr, do not do this.

The following three macros are used to initially allocate memory:

Newx(pointer, number, type);
Newxc(pointer, number, type, cast);
Newxz(pointer, number, type);

The first argument pointer should be the name of a variable that will point to the newly allocated memory. 🎯

The second and third arguments number and type specify how many of the specified type of data structure should be allocated. The argument type is passed to sizeof. The final argument to Newxc, cast, should be used if the pointer argument is different from the type argument. 💅

Unlike the Newx and Newxc macros, the Newxz macro calls memzero to zero out all the newly allocated memory. ✨ clean slate energy ✨

✨ Reallocation ✨

Renew(pointer, number, type);
Renewc(pointer, number, type, cast);
Safefree(pointer)

These three macros are used to change a memory buffer size or to free a piece of memory no longer needed. The arguments to Renew and Renewc match those of New and Newc with the exception of not needing the "magic cookie" argument. 🍪 (no cookie for you)

✨ Moving ✨

Move(source, dest, number, type);
Copy(source, dest, number, type);
Zero(dest, number, type);

These three macros are used to move, copy, or zero out previously allocated memory. The source and dest arguments point to the source and destination starting points. Perl will move, copy, or zero out number instances of the size of the type data structure (using the sizeof function). 🚚📦

---

🌊 PerlIO 🌊

The most recent development releases of Perl have been experimenting with removing Perl's dependency on the "normal" standard I/O suite and allowing other stdio implementations to be used. 👀 This involves creating a new abstraction layer that then calls whichever implementation of stdio Perl was compiled with. All XSUBs should now use the functions in the PerlIO abstraction layer and not make any assumptions about what kind of stdio is being used. no cap.

For a complete description of the PerlIO abstraction, consult perlapio. 📖

---

🛠️ Compiled code 🛠️

🌳 Code tree 🌳

Here we describe the internal form your code is converted to by Perl. Start with a simple example:

$a = $b + $c;

This is converted to a tree similar to this one:

             assign-to
           /           \
          +             $a
        /   \
      $b     $c

(but slightly more complicated). This tree reflects the way Perl parsed your code, but has nothing to do with the execution order. There is an additional "thread" going through the nodes of the tree which shows the order of execution of the nodes. In our simplified example above it looks like:

     $b ---> $c ---> + ---> $a ---> assign-to

But with the actual compile tree for $a = $b + $c it is different: some nodes optimized away. As a corollary, though the actual tree contains more nodes than our simplified example, the execution order is the same as in our example. 😌

🔍 Examining the tree 🔍

If you have your perl compiled for debugging (usually done with -DDEBUGGING on the Configure command line), you may examine the compiled tree by specifying -Dx on the Perl command line. The output takes several lines per node, and for $b+$c it looks like this:

    5           TYPE = add  ===> 6
                TARG = 1
                FLAGS = (SCALAR,KIDS)
                {
                    TYPE = null  ===> (4)
                      (was rv2sv)
                    FLAGS = (SCALAR,KIDS)
                    {
    3                   TYPE = gvsv  ===> 4
                        FLAGS = (SCALAR)
                        GV = main::b
                    }
                }
                {
                    TYPE = null  ===> (5)
                      (was rv2sv)
                    FLAGS = (SCALAR,KIDS)
                    {
    4                   TYPE = gvsv  ===> 5
                        FLAGS = (SCALAR)
                        GV = main::c
                    }
                }

This tree has 5 nodes (one per TYPE specifier), only 3 of them are not optimized away (one per number in the left column). The immediate children of the given node correspond to {} pairs on the same level of indentation, thus this listing corresponds to the tree:

                   add
                 /     \
               null    null
                |       |
               gvsv    gvsv

The execution order is indicated by ===> marks, thus it is 3 4 5 6 (node 6 is not included into above listing), i.e., gvsv gvsv add whatever. 🔥

Each of these nodes represents an op, a fundamental operation inside the Perl core. The code which implements each operation can be found in the pp*.c files; the function which implements the op with type gvsv is pp_gvsv, and so on. As the tree above shows, different ops have different numbers of children: add is a binary operator, as one would expect, and so has two children. To accommodate the various different numbers of children, there are various types of op data structure, and they link together in different ways. 🧩

The simplest type of op structure is OP: this has no children. Unary operators, UNOPs, have one child, and this is pointed to by the op_first field. Binary operators (BINOPs) have not only an op_first field but also an op_last field. The most complex type of op is a LISTOP, which has any number of children. In this case, the first child is pointed to by op_first and the last child by op_last. The children in between can be found by iteratively following the OpSIBLING pointer from the first child to the last (but see below). 👑

There are also some other op types: a PMOP holds a regular expression, and has no children, and a LOOP may or may not have children. If the op_children field is non-zero, it behaves like a LISTOP. To complicate matters, if a UNOP is actually a null op after optimization (see "Compile pass 2: context propagation") it will still have children in accordance with its former type. 😵‍💫

Finally, there is a LOGOP, or logic op. Like a LISTOP, this has one or more children, but it doesn't have an op_last field: so you have to follow op_first and then the OpSIBLING chain itself to find the last child. Instead it has an op_other field, which is comparable to the op_next field described below, and represents an alternate execution path. Operators like and, or and ? are LOGOPs. Note that in general, op_other may not point to any of the direct children of the LOGOP. 🌀

Starting in version 5.21.2, perls built with the experimental define -DPERL_OP_PARENT add an extra boolean flag for each op, op_moresib. When not set, this indicates that this is the last op in an OpSIBLING chain. This frees up the op_sibling field on the last sibling to point back to the parent op. Under this build, that field is also renamed op_sibparent to reflect its joint role. The macro OpSIBLING(o) wraps this special behaviour, and always returns NULL on the last sibling. With this build the op_parent(o) function can be used to find the parent of any op. Thus for forward compatibility, you should always use the OpSIBLING(o) macro rather than accessing op_sibling directly. 💡

Another way to examine the tree is to use a compiler back-end module, such as B::Concise. 🎀

🧪 Compile pass 1: check routines 🧪

The tree is created by the compiler while yacc code feeds it the constructions it recognizes. Since yacc works bottom-up, so does the first pass of perl compilation. ⬆️

What makes this pass interesting for perl developers is that some optimization may be performed on this pass. This is optimization by so-called "check routines". The correspondence between node names and corresponding check routines is described in opcode.pl (do not forget to run make regen_headers if you modify this file). 📝

A check routine is called when the node is fully constructed except for the execution-order thread. Since at this time there are no back-links to the currently constructed node, one can do most any operation to the top-level node, including freeing it and/or creating new nodes above/below it. 🪄

The check routine returns the node which should be inserted into the tree (if the top-level node was not modified, check routine returns its argument).

By convention, check routines have names ck_*. They are usually called from new*OP subroutines (or convert) (which in turn are called from perly.y). 🔧

✨ Compile pass 1a: constant folding ✨

Immediately after the check routine is called the returned node is checked for being compile-time executable. If it is (the value is judged to be constant) it is immediately executed, and a constant node with the "return value" of the corresponding subtree is substituted instead. The subtree is deleted. 🗑️

If constant folding was not performed, the execution-order thread is created.

🔄 Compile pass 2: context propagation 🔄

When a context for a part of compile tree is known, it is propagated down through the tree. At this time the context can have 5 values (instead of 2 for runtime context): void, boolean, scalar, list, and lvalue. In contrast with the pass 1 this pass is processed from top to bottom: a node's context determines the context for its children. ⬇️

Additional context-dependent optimizations are performed at this time. Since at this moment the compile tree contains back-references (via thread pointers), nodes cannot be free()d now. To allow optimized-away nodes at this stage, such nodes are null()ified instead of free()ing (i.e. their type is changed to OP_NULL). 🕳️

👀 Compile pass 3: peephole optimization 👀

After the compile tree for a subroutine (or for an eval or a file) is created, an additional pass over the code is performed. This pass is neither top-down nor bottom-up, but in the execution order (with additional complications for conditionals). Optimizations performed at this stage are subject to the same restrictions as in the pass 2. 🎯

Okay so peephole optimizations?? They happen when perl calls whatever function the global variable PL_peepp is pointing at, bestie. Out of the box, PL_peepp is literally just a lil relay that calls the function PL_rpeepp points at. And PL_rpeepp, also out of the box, does some basic op fixups and optimisations along the execution-order op chain, and then recursively calls PL_rpeepp for every single side chain of ops (the ones that come from conditionals). Extensions can slide in and provide extra optimisations or fixups — you can hook into either the per-subroutine stage or the recursive stage, like this:

    static peep_t prev_peepp;
    static void my_peep(pTHX_ OP *o)
    {
        /* custom per-subroutine optimisation goes here */
        prev_peepp(aTHX_ o);
        /* custom per-subroutine optimisation may also go here */
    }
    BOOT:
        prev_peepp = PL_peepp;
        PL_peepp = my_peep;

    static peep_t prev_rpeepp;
    static void my_rpeep(pTHX_ OP *first)
    {
        OP *o = first, *t = first;
        for(; o = o->op_next, t = t->op_next) {
            /* custom per-op optimisation goes here */
            o = o->op_next;
            if (!o || o == t) break;
            /* custom per-op optimisation goes AND here */
        }
        prev_rpeepp(aTHX_ orig_o);
    }
    BOOT:
        prev_rpeepp = PL_rpeepp;
        PL_rpeepp = my_rpeep;

---

🏃 Pluggable runops 🏃

The compile tree gets executed inside a runops function. There are two of these besties: one in run.c and one in dump.c. Perl_runops_debug is the one that gets used when you're DEBUGGING, and Perl_runops_standard is the one for every other day. But if you want fine control over how your compile tree actually runs, you can 100% provide your own runops function. Main character move, honestly.

Best tip ever: just copy one of the existing runops functions and remix it to suit your needs. Then, in the BOOT section of your XS file, add this line:

  PL_runops = my_runops;

This function should be as efficient as possible to keep your programs running as fast as possible. No pressure, but like. Efficiency era. 💅

---

🪝 Compile-time scope hooks 🪝

As of perl 5.14 — so we've had this for a minute — it's possible to hook into the compile-time lexical scope mechanism using Perl_blockhook_register. It goes like this:

    static void my_start_hook(pTHX_ int full);
    static BHK my_hooks;

    BOOT:
        BhkENTRY_set(&my_hooks, bhk_start, my_start_hook);
        Perl_blockhook_register(aTHX_ &my_hooks);

This arranges for my_start_hook to get called at the start of compiling every lexical scope. The hooks you can use are:

• void bhk_start(pTHX_ int full) — called right after a new lexical scope starts. Heads up: Perl code like

    if ($x) { ... }

creates two scopes — the first starts at the ( and has full == 1, and the second starts at the { and has full == 0. Both of them end at the }, so calls to start and pre/post_end will match up. Anything you push onto the save stack from this hook gets popped just before the scope ends (between the pre_ and post_end hooks, in fact).

• void bhk_pre_end(pTHX_ OP **o) — called at the end of a lexical scope, just before unwinding the stack. o is the root of the optree representing the scope; it's a double pointer so you can replace the OP if you need to. Go off.
• void bhk_post_end(pTHX_ OP **o) — called at the end of a lexical scope, just after unwinding the stack. o is as above. Ngl, calls to pre_ and post_end can nest, if there's something on the save stack that calls string eval.
• void bhk_eval(pTHX_ OP *const o) — called just before starting to compile an eval STRING, do FILE, require or use, after the eval has been set up. o is the OP that requested the eval, and it'll normally be an OP_ENTEREVAL, OP_DOFILE or OP_REQUIRE.

Once you've got your hook functions, you need a BHK structure to put them in. Allocate that thing statically, please — there's no way to free it once it's registered, so dynamic allocation is giving trap. The function pointers go into the structure with the BhkENTRY_set macro, which also sets flags saying which entries are valid. And if you do need to allocate your BHK dynamically for some reason, make sure you zero it before you start.

Once registered, there's no mechanism to switch these hooks off, so if you need that, you're doing it yourself, babes. An entry in %^H is probably the best way, so the effect stays lexically scoped; that said, you can also use the BhkDISABLE and BhkENABLE macros to temporarily switch entries on and off. Also, be aware that generally speaking at least one scope will have opened before your extension is loaded — so you will see some pre/post_end pairs that didn't have a matching start. It's not a bug, it's just the timeline. 🦋

---

🔍 Examining internal data structures with the dump functions 🔍

To aid debugging, the source file dump.c contains a bunch of functions that produce formatted output of internal data structures.

The one you'll see most is Perl_sv_dump — it dumps SVs, AVs, HVs, and CVs. The Devel::Peek module calls sv_dump to produce debugging output from Perl-space, so if you use that module you already know this format. Iconic.

Perl_op_dump dumps an OP structure or any of its derivatives, and its output looks a lot like perl -Dx; in fact, Perl_dump_eval will dump the main root of the code being evaluated, exactly like -Dx.

Other useful ones: Perl_dump_sub, which turns a GV into an op tree, Perl_dump_packsubs, which calls Perl_dump_sub on all the subroutines in a package like so (and thankfully, these are all xsubs, so there's no op tree):

(gdb) print Perl_dump_packsubs(PL_defstash)

SUB attributes::bootstrap = (xsub 0x811fedc 0)

SUB UNIVERSAL::can = (xsub 0x811f50c 0)

SUB UNIVERSAL::isa = (xsub 0x811f304 0)

SUB UNIVERSAL::VERSION = (xsub 0x811f7ac 0)

SUB DynaLoader::boot_DynaLoader = (xsub 0x805b188 0)

...and Perl_dump_all, which dumps all the subroutines in the stash and the op tree of the main root. The full sweep. 💖

---

👀 Tracing the interpreter execution 👀

Throughout the source code you'll see statements like

 DEBUG_L(PerlIO_printf(Perl_debug_log, "%s\n", msg));
 DEBUG_l(deb("Entering new RUNOPS level: depth=%d\n", depth));
 DEBUG_r(if (!PL_colorset) reginitcolors());
 DEBUG_y(sv_dump(sv));

Each of these executes the code within the outermost pair of parentheses, but only under certain circumstances:

• perl must have been compiled with -DDEBUGGING — if it wasn't, each such statement compiles to absolutely nothing. Zero bytes. Ghosted.
• The $^D variable must have the appropriate bit set. — so to actually execute, DEBUG_y needs the y bit set. Same energy for the L, l, and r bits.

Each bit handles a different aspect of interpreter execution. L is for locale handling, l is for context (loop) stack processing, r is for regular expressions, and y is for tr///.

All possible bits and their meanings are listed under -Dletters in perlrun, or in the perl.h source under, like, DEBUG_L_FLAG.

Setting these is most often done from the command line that started this execution of the interpreter, using -D... options, but a Perl program can change the value of this variable at will. Your call, bestie.

So for example, when perl has been compiled with -DEBUGGING, and turned on by passing the -Dy option to perl at startup, any time the execution encounters a C language source DEBUG_y(sv_dump(sv)); statement, sv_dump gets called. (See "sv_dump" in perlapi for what that does.)

Lines like these are strategically placed in the C code to help find problems when turned on, but they occupy no space at all without -DDEBUGGING. Free real estate when it's off. ✨

There's also a v modifier bit which requests extra verbosity for the other set bits. In statements like

 DEBUG_Lv(PerlIO_printf(Perl_debug_log, "Entering function(%s);", ...))

PerlIO_printf is executed only when perl has been compiled with -DDEBUGGING and BOTH the L and v bits are enabled, such as with -DLv or -DvL.... Only a few main bits are affected by v, and currently there's only one level of extra verbosity. -DLvvv and -LDv do exactly the same thing. Yes really.

The most commonly used form of these is like

 DEBUG_L(PerlIO_printf(Perl_debug_log, "%s", msg));

When enabled as described above, this statement causes the contents of the C language character string msg to be displayed to stderr whenever the statement is encountered during the execution of a Perl program. After Perl_debug_log must be a format string, and then any number of arguments to that format, just as-if this were a printf().

deb() as the interior does the same thing, but automatically notes the Perl source code line that was the ultimate cause of this C statement getting executed. This is an easy way to associate the Perl code with its corresponding C translation. See "deb" in perlapi. Its use in our source should actually be more widespread than it is. Patches Welcome! 🫶

Most of these statements are essentially permanently placed in the source code, to be turned on, perhaps in the far distant future, as-needed. But there's one bit that is reserved for temporary placement during development or handling special situations. You can add DEBUG_U (for "Unofficial") statements for whatever you need. But these should be removed after the code is debugged, or be converted to use other bits. Clean up after yourself, periodt.

You can write more complex conditions to more finely tune if output will be generated, using the DEBUG_U_TEST and DEBUG_Uv_TEST macros. They return true if and only if a debug statement of the corresponding type would display. There are corresponding macros for each of the bits.

---

👑 How multiple interpreters and concurrency are supported 👑

🧵 Background and MULTIPLICITY 🧵

The Perl interpreter can be regarded as a closed box: it has an API for feeding it code or otherwise making it do things, but it also has functions for its own use. Ngl, this smells a lot like an object — and there's a way for you to build Perl so that you can have multiple interpreters, with one interpreter represented either as a C structure, or inside a thread-specific structure. These structures contain all the context, the state of that interpreter. It's giving object-oriented, fr fr.

The macro that controls the major Perl build flavor is MULTIPLICITY. The MULTIPLICITY build has a C structure that packages all the interpreter state, which gets passed to various perl functions as a "hidden" first argument. MULTIPLICITY is what makes multi-threaded perls possible (with the ithreads threading model, related to the macro USE_ITHREADS).

PERL_IMPLICIT_CONTEXT is a legacy synonym for MULTIPLICITY.

To see whether you have non-const data you can use a BSD (or GNU) compatible nm:

nm libperl.a | grep -v ' [TURtr] '

okay so if that pops off and shows you any D or d symbols (or possibly C or c)... bestie, you have non-const data 😭. The symbols that the grep just yeeted out are, for the record: Tt are text, aka code, the Rr are read-only (const) data, and the U is <undefined>, meaning external symbols that are referred to.

The test t/porting/libperl.t does this kind of symbol sanity checking on libperl.a. automated vibe check, we love to see it ✨

All this obviously requires a way for the Perl internal functions to be either subroutines taking some kind of structure as the first argument, or subroutines taking nothing as the first argument. To enable these two very different ways of building the interpreter, the Perl source — as it does in so many other situations, and honestly iconic of it 💅 — makes heavy use of macros and subroutine naming conventions.

First problem: deciding which functions will be public API functions and which will be private. All functions whose names begin S_ are private (think "S" for "secret" or "static"). All other functions begin with Perl_, but just because a function begins with Perl_ does not mean it is part of the API. (See "Internal Functions".) The easiest way to be sure a function is part of the API is to find its entry in perlapi. If it exists in perlapi, it's part of the API. If it doesn't, and you think it should be (i.e., you need it for your extension), submit an issue at <https://github.com/Perl/perl5/issues> explaining why you think it should be.

Second problem: there must be a syntax so that the same subroutine declarations and calls can pass a structure as their first argument, or pass nothing. To solve this, the subroutines are named and declared in a particular way. Here's a typical start of a static function used within the Perl guts:

static void
S_incline(pTHX_ char *s)

A public function (i.e. part of the internal API, but not necessarily sanctioned for use in extensions) begins like this:

void
Perl_sv_setiv(pTHX_ SV* dsv, IV num)

pTHX_ is one of a number of macros (in perl.h) that hide the details of the interpreter's context. THX stands for "thread", "this", or "thingy", as the case may be. (And no, George Lucas is not involved. :-) The first character could be p for a prototype, a for argument, or d for declaration, so we have pTHX, aTHX and dTHX, and their variants.

When Perl is built without options that set MULTIPLICITY, there is no first argument containing the interpreter's context. The trailing underscore in the pTHX_ macro indicates that the macro expansion needs a comma after the context argument because other arguments follow it. If MULTIPLICITY is not defined, pTHX_ will be ignored, and the subroutine is not prototyped to take the extra argument. The form of the macro without the trailing underscore is used when there are no additional explicit arguments.

When a core function calls another, it must pass the context. This is normally hidden via macros. Consider sv_setiv. It expands into something like this:

#ifdef MULTIPLICITY
  #define sv_setiv(a,b)      Perl_sv_setiv(aTHX_ a, b)
  /* can't do this for vararg functions, see below */
#else
  #define sv_setiv           Perl_sv_setiv
#endif

This works well, and means that XS authors can gleefully write:

sv_setiv(foo, bar);

and still have it work under all the modes Perl could have been compiled with. gleefully!! as they should 🫶

This doesn't work so cleanly for varargs functions, though, as macros imply that the number of arguments is known in advance. Instead we either need to spell them out fully, passing aTHX_ as the first argument (the Perl core tends to do this with functions like Perl_warner), or use a context-free version.

The context-free version of Perl_warner is called Perl_warner_nocontext, and does not take the extra argument. Instead it does dTHX; to get the context from thread-local storage. We #define warner Perl_warner_nocontext so that extensions get source compatibility at the expense of performance. (Passing an arg is cheaper than grabbing it from thread-local storage.)

You can ignore [pad]THXx when browsing the Perl headers/sources. Those are strictly for use within the core. Extensions and embedders need only be aware of [pad]THX.

---

🪦 So what happened to dTHR?

dTHR was introduced in perl 5.005 to support the older thread model. The older thread model now uses the THX mechanism to pass context pointers around, so dTHR is not useful any more. Perl 5.6.0 and later still have it for backward source compatibility, but it is defined to be a no-op. rest easy queen, you did your service 💜

🧚 How do I use all this in extensions?

See also "Dealing with embedded perls and threads" in perlclib.

When Perl is built with MULTIPLICITY, extensions that call any functions in the Perl API will need to pass the initial context argument somehow. The kicker is that you will need to write it in such a way that the extension still compiles when Perl hasn't been built with MULTIPLICITY enabled.

There are three ways to do this. First, the easy but inefficient way, which is also the default, in order to maintain source compatibility with extensions: whenever XSUB.h is #included, it redefines the aTHX and aTHX_ macros to call a function that will return the context. Thus, something like:

sv_setiv(sv, num);

in your extension will translate to this when MULTIPLICITY is in effect:

Perl_sv_setiv(Perl_get_context(), sv, num);

or to this otherwise:

Perl_sv_setiv(sv, num);

You don't have to do anything new in your extension to get this; since the Perl library provides Perl_get_context(), it will all just work. effortless, zero notes 👑

The second, more efficient way is to use the following template for your Foo.xs:

#define PERL_NO_GET_CONTEXT     /* we want efficiency */
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

static void my_private_function(int arg1, int arg2);

static void
my_private_function(int arg1, int arg2)
{
    dTHX;       /* fetch context */
    ... call many Perl API functions ...
}

[... etc ...]

MODULE = Foo            PACKAGE = Foo

/* typical XSUB */

void
my_xsub(arg)
        int arg
    CODE:
        my_private_function(arg, 10);

Note that the only two changes from the normal way of writing an extension is the addition of a #define PERL_NO_GET_CONTEXT before including the Perl headers, followed by a dTHX; declaration at the start of every function that will call the Perl API. (You'll know which functions need this, because the C compiler will complain that there's an undeclared identifier in those functions.) No changes are needed for the XSUBs themselves, because the XS() macro is correctly defined to pass in the implicit context if needed.

The third, even more efficient way is to ape how it is done within the Perl guts:

#define PERL_NO_GET_CONTEXT     /* we want efficiency */
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

/* pTHX_ only needed for functions that call Perl API */
static void my_private_function(pTHX_ int arg1, int arg2);

static void
my_private_function(pTHX_ int arg1, int arg2)
{
    /* dTHX; not needed here, because THX is an argument */
    ... call Perl API functions ...
}

[... etc ...]

MODULE = Foo            PACKAGE = Foo

/* typical XSUB */

void
my_xsub(arg)
        int arg
    CODE:
        my_private_function(aTHX_ arg, 10);

This implementation never has to fetch the context using a function call, since it is always passed as an extra argument. Depending on your needs for simplicity or efficiency, you may mix the previous two approaches freely. pick your fighter, or don't, mix them, it's your world 🌈

Never add a comma after pTHX yourself — always use the form of the macro with the underscore for functions that take explicit arguments, or the form without the argument for functions with no explicit arguments. fr fr, do not freestyle this one 🙈

🧵 Should I do anything special if I call perl from multiple threads?

If you create interpreters in one thread and then proceed to call them in another, you need to make sure perl's own Thread Local Storage (TLS) slot is initialized correctly in each of those threads.

The perl_alloc and perl_clone API functions will automatically set the TLS slot to the interpreter they created, so that there is no need to do anything special if the interpreter is always accessed in the same thread that created it, and that thread did not create or call any other interpreters afterwards. If that is not the case, you have to set the TLS slot of the thread before calling any functions in the Perl API on that particular interpreter. This is done by calling the PERL_SET_CONTEXT macro in that thread as the first thing you do:

/* do this before doing anything else with some_perl */
PERL_SET_CONTEXT(some_perl);

... other Perl API calls on some_perl go here ...

(You can always get the current context via PERL_GET_CONTEXT.)

🔮 Future Plans and PERL_IMPLICIT_SYS

Just as MULTIPLICITY provides a way to bundle up everything that the interpreter knows about itself and pass it around, so too are there plans to allow the interpreter to bundle up everything it knows about the environment it's running on. This is enabled with the PERL_IMPLICIT_SYS macro. Currently it only works with USE_ITHREADS on Windows.

This allows the ability to provide an extra pointer (called the "host" environment) for all the system calls. This makes it possible for all the system stuff to maintain their own state, broken down into seven C structures. These are thin wrappers around the usual system calls (see win32/perllib.c) for the default perl executable, but for a more ambitious host (like the one that would do fork() emulation) all the extra work needed to pretend that different interpreters are actually different "processes", would be done here.

The Perl engine/interpreter and the host are orthogonal entities. There could be one or more interpreters in a process, and one or more "hosts", with free association between them. seven structures, free association, orthogonal entities — it's giving architecture 💖

💖 Internal Functions 💖

okay bestie so literally every single internal function in Perl that's gonna be exposed to the outside world gets a little glow-up prefix: Perl_ 💅 that's so it doesn't beef with XS functions or any function in some program that's embedding Perl. same energy for globals — they all start with PL_. (and by convention, static functions start with S_, cuties stay lowkey.)

Inside the Perl core (like when PERL_CORE is defined), you can reach the functions with or without the Perl_ prefix — iconic — thanks to a whole bunch of defines living in embed.h. But listen: extension code should NOT set PERL_CORE. bestie no. that exposes the full perl internals and is basically guaranteed to break your XS on every new perl release. 🙈 don't do it.

embed.h is auto-generated from embed.pl and embed.fnc — the way that embed.pl also makes the prototyping header files, the documentation, and a whole lotta other bits and pieces. so when you add a new function to the core or change an existing one, it is so important that you update the table in embed.fnc too. here's a lil sample entry from that table:

Apd |SV**   |av_fetch   |AV* ar|I32 key|I32 lval

first column = the flags 🚩, second = the return type, third = the name, and everything after that = the arguments. the flags are documented at the top of embed.fnc, go look.

if you edit embed.pl or embed.fnc, you gotta run:

make regen_headers

to force a rebuild of embed.h and the other auto-generated files. periodt. ⭐

✨ Formatted Printing of IVs, UVs, and NVs ✨

if you're printing IVs, UVs, or NVs instead of using stdio(3)-style formatting codes like %d, %ld, %f, you should use these macros for portability, queen:

IVdf            IV in decimal
UVuf            UV in decimal
UVof            UV in octal
UVxf            UV in hexadecimal
NVef            NV %e-like
NVff            NV %f-like
NVgf            NV %g-like

these take care of 64-bit integers and long doubles, no cap. like for example:

printf("IV is %" IVdf "\n", iv);

the IVdf will expand to whatever's the correct format for the IVs. now — important — the spaces are required around the format, in case the code gets compiled with C++, to keep compliance with its standard. don't skip the spaces bestie.

also note: there are different "long doubles" out here. Perl uses whatever the compiler has. 💫

if you're printing addresses of pointers, use %p or UVxf combined with PTR2UV().

🌸 Formatted Printing of SVs 🌸

the contents of SVs can be printed using the SVf format, like so:

Perl_croak(aTHX_ "This croaked because: %" SVf "\n", SVfARG(err_msg))

where err_msg is an SV.

not all scalar types are printable, okay, that's the vibe check. simple values definitely are: one of IV, UV, NV, or PV. also, if the SV is a reference to some value, either it gets dereferenced and the value printed, or you get info about the type of that value and its address. printing any other type of SV? undefined results and it'll probably crash the interpreter. 😭 NVs get printed using a %g-ish format.

and again — the spaces are required around the SVf, in case it's compiled with C++, for standards compliance.

also note: any filehandle being printed to under UTF-8 has to be expecting UTF-8 to get good results and avoid Wide-character warnings. one way to do that for typical filehandles is to invoke perl with the -C parameter. (See -C [number/list] in perlrun.)

you can use this to concatenate two scalars, so slay:

SV *var1 = get_sv("var1", GV_ADD);
SV *var2 = get_sv("var2", GV_ADD);
SV *var3 = newSVpvf("var1=%" SVf " and var2=%" SVf,
                    SVfARG(var1), SVfARG(var2));

SVf_QUOTEDPREFIX is similar to SVf except it restricts the number of characters printed, showing at most the first PERL_QUOTEDPREFIX_LEN characters of the argument, and rendering it with double quotes and escaping the contents using double-quoted string escaping rules. if the string is longer than that, ellipses ... get appended after the trailing quote. this is intended for error messages where the string is assumed to be a class name.

HvNAMEf and HvNAMEf_QUOTEDPREFIX are similar to SVf except they extract the string, length, and utf8 flags from the argument given to HvNAMEfARG() using the HvNAME(), HvNAMELEN(), HvNAMEUTF8() macros. this is for stringifying a class name straight from a stash HV. 🎀

🦋 Formatted Printing of Strings 🦋

if you literally just want bytes printed in a 7bit NUL-terminated string, you can just use %s (assuming they're all really only 7bit). but if there's a possibility the value will be encoded as UTF-8 or contains bytes above 0x7F (and therefore 8bit), you should use the UTF8f format instead. and as its parameter, use the UTF8fARG() macro:

chr * msg;

/* U+2018: \xE2\x80\x98 LEFT SINGLE QUOTATION MARK
   U+2019: \xE2\x80\x99 RIGHT SINGLE QUOTATION MARK */
if (can_utf8)
  msg = "\xE2\x80\x98Uses fancy quotes\xE2\x80\x99";
else
  msg = "'Uses simple quotes'";

Perl_croak(aTHX_ "The message is: %" UTF8f "\n",
                 UTF8fARG(can_utf8, strlen(msg), msg));

first parameter to UTF8fARG is a boolean: 1 if the string is in UTF-8; 0 if the string is in native byte encoding (Latin1). second parameter is the number of bytes in the string to print. third and final parameter is a pointer to the first byte in the string. easy. ✨

and again, note: any filehandle being printed to under UTF-8 must be expecting UTF-8 to get good results and avoid Wide-character warnings. one way to do this for typical filehandles is invoking perl with the -C parameter. (See -C [number/list] in perlrun.)

💜 Formatted Printing of Size_t and SSize_t 💜

the most general way to do this is to cast them to a UV or IV, and print as in the previous section.

but if you're using PerlIO_printf(), it's less typing and less visual clutter to use the %z length modifier (for siZe):

PerlIO_printf("STRLEN is %zu\n", len);

this modifier is not portable, so only use it with PerlIO_printf(). that's the rule, bestie. 🌈

⭐ Formatted Printing of Ptrdiff_t, intmax_t, short and other special sizes ⭐

there are modifiers for these special situations if you're using PerlIO_printf(). See "size" in perlfunc.

💅 Pointer-To-Integer and Integer-To-Pointer 💅

because pointer size doesn't necessarily equal integer size, use these follow macros to do it right:

PTR2UV(pointer)
PTR2IV(pointer)
PTR2NV(pointer)
INT2PTR(pointertotype, integer)

for example:

IV  iv = ...;
SV *sv = INT2PTR(SV*, iv);

and:

AV *av = ...;
UV  uv = PTR2UV(av);

there are also:

PTR2nat(pointer)   /* pointer to integer of PTRSIZE */
PTR2ul(pointer)    /* pointer to unsigned long (unsafe) */

PTR2ul() is unsafe, since a pointer may be larger than unsigned long. do not play with it unless you know what you're doing. 💀

and there's PTRV, which gives the native type for an integer the same size as pointers — like unsigned or unsigned long.

🔥 Exception Handling 🔥

there are a couple of macros to do very basic exception handling in XS modules. you have to define NO_XSLOCKS before including XSUB.h to be able to use these macros:

#define NO_XSLOCKS
#include "XSUB.h"

you can use these macros if you call code that may croak, but you need to do some cleanup before giving control back to Perl. for example:

dXCPT;    /* set up necessary variables */

XCPT_TRY_START {
  code_that_may_croak();
} XCPT_TRY_END

XCPT_CATCH
{
  /* do cleanup here */
  XCPT_RETHROW;
}

note: you always have to rethrow an exception that has been caught. using these macros, it is not possible to just catch the exception and ignore it. if you have to ignore the exception, you have to use the call_* function.

the advantage of using these macros is that you don't have to set up an extra function for call_*, and that using these macros is faster than using call_*. ate. 👑

🎀 Source Documentation 🎀

there's an effort going on to document the internal functions and automatically produce reference manuals from them — perlapi is one such manual, which details all the functions available to XS writers. perlintern is the autogenerated manual for the functions that aren't part of the API and are supposedly for internal use only.

source documentation is created by putting POD comments into the C source, like this:

/*
=for apidoc sv_setiv

Copies an integer into the given SV.  Does not handle 'set' magic.  See
L<perlapi/sv_setiv_mg>.

=cut
*/

please try and supply some documentation if you add functions to the Perl core, bestie. it's the little things. 🫶

🌈 Backwards compatibility 🌈

the Perl API changes over time. new functions get added, or the interfaces of existing functions get changed. the Devel::PPPort module tries to provide compatibility code for some of these changes, so XS writers don't have to code it themselves when supporting multiple versions of Perl. iconic behavior honestly.

Devel::PPPort generates a C header file ppport.h that can also be run as a Perl script. to generate ppport.h, run:

perl -MDevel::PPPort -eDevel::PPPort::WriteFile

besides checking existing XS code, the script can also be used to retrieve compatibility information for various API calls using the --api-info command line switch. for example:

% perl ppport.h --api-info=sv_magicext

for details, see "perldoc ppport.h". 💖

---

🌸 Unicode Support 🌸

Perl 5.6.0 introduced Unicode support. it's important for porters and XS writers to understand this support and make sure the code they write doesn't corrupt Unicode data. no pressure bestie but also yes pressure.

✨ What is Unicode, anyway? ✨

in the olden, less enlightened times, we all used to use ASCII. most of us did, anyway. the big problem with ASCII is that it's American. well — no, that's not actually the problem; the problem is that it's not particularly useful for people who don't use the Roman alphabet. what used to happen was that particular languages would stick their own alphabet in the upper range of the sequence, between 128 and 255. and of course, we then ended up with plenty of variants that weren't quite ASCII, and the whole point of it being a standard was lost. 😭

worse still — if you've got a language like Chinese or Japanese that has hundreds or thousands of characters, then you really can't fit them into a mere 256, so they had to forget about ASCII altogether and build their own systems using pairs of numbers to refer to one character. slay for them honestly, they had to invent their own thing because ASCII was not gonna cut it. 🦋

To fix this, some people formed Unicode, Inc. and produced a new character set containing all the characters you can possibly think of and more. There are several ways of representing these characters, and the one Perl uses is called UTF-8. UTF-8 uses a variable number of bytes to represent a character. You can learn more about Unicode and Perl's Unicode model in perlunicode.

(On EBCDIC platforms, Perl uses instead UTF-EBCDIC, which is a form of UTF-8 adapted for EBCDIC platforms. Below, we just talk about UTF-8. UTF-EBCDIC is like UTF-8, but the details are different. The macros hide the differences from you, just remember that the particular numbers and bit patterns presented below will differ in UTF-EBCDIC.)

✨ How can I recognise a UTF-8 string? ✨

You can't. It is not giving what you want it to give, bestie. 😭 This is because UTF-8 data is stored in bytes just like non-UTF-8 data. The Unicode character 200, (0xC8 for you hex types) capital E with a grave accent, is represented by the two bytes "v196.172". Unfortunately, the non-Unicode string "chr(196).chr(172)" has that byte sequence as well. So you can't tell just by looking — this is what makes Unicode input an interesting problem.

In general, you either have to know what you're dealing with, or you have to guess. The API function is_utf8_string can help; it'll tell you if a string contains only valid UTF-8 characters, and the chances of a non-UTF-8 string looking like valid UTF-8 become very small very quickly with increasing string length. On a character-by-character basis, isUTF8_CHAR will tell you whether the current character in a string is valid UTF-8.

✨ How does UTF-8 represent Unicode characters? ✨

As mentioned above, UTF-8 uses a variable number of bytes to store a character. Characters with values 0...127 are stored in one byte, just like good ol' ASCII. Character 128 is stored as "v194.128"; this continues up to character 191, which is "v194.191". Now we've run out of bits (191 is binary 10111111) so we move on; character 192 is "v195.128". And so it goes on, moving to three bytes at character 2048. "Unicode Encodings" in perlunicode has pictures of how this works. 🖼️✨

Assuming you know you're dealing with a UTF-8 string, you can find out how long the first character in it is with the UTF8SKIP macro:

char *utf = "\305\233\340\240\201";
I32 len;

len = UTF8SKIP(utf); /* len is 2 here */
utf += len;
len = UTF8SKIP(utf); /* len is 3 here */

Another way to skip over characters in a UTF-8 string is to use "utf8_hop_safe" in perlapi, which takes a string and a number of characters to skip over.

All bytes in a multi-byte UTF-8 character will have the high bit set, so you can test if you need to do something special with this character like this (the UTF8_IS_INVARIANT() is a macro that tests whether the byte is encoded as a single byte even in UTF-8):

U8 *utf;     /* Initialize this to point to the beginning of the
                sequence to convert */
U8 *utf_end; /* Initialize this to 1 beyond the end of the sequence
                pointed to by 'utf' */
UV uv;       /* Returned code point; note: a UV, not a U8, not a
                char */
STRLEN len; /* Returned length of character in bytes */

if (!UTF8_IS_INVARIANT(*utf)) {
    /* Must treat this as UTF-8 */
    if (! utf8_to_uv(utf, utf_end, &uv, &len)) {
        /* handle error */
    }
}
else
    /* OK to treat this character as a byte */
    uv = *utf;

You can also see in that example that we use utf8_to_uv to get the value of the character; the inverse function uv_to_utf8 is available for putting a UV into UTF-8:

if (!UVCHR_IS_INVARIANT(uv))
    /* Must treat this as UTF8 */
    utf8 = uv_to_utf8(utf8, uv);
else
    /* OK to treat this character as a byte */
    *utf8++ = uv;

You must convert characters to UVs using the above functions if you're ever in a situation where you have to match UTF-8 and non-UTF-8 characters. You may not skip over UTF-8 characters in this case. If you do this, you'll lose the ability to match hi-bit non-UTF-8 characters; for instance, if your UTF-8 string contains "v196.172", and you skip that character, you can never match a chr(200) in a non-UTF-8 string. So don't do that! 🙈

(Note that we don't have to test for invariant characters in the examples above. The functions work on any well-formed UTF-8 input. It's just that it's faster to avoid the function overhead when it's not needed.)

✨ How does Perl store UTF-8 strings? ✨

Currently, Perl deals with UTF-8 strings and non-UTF-8 strings slightly differently. A flag in the SV, SVf_UTF8, indicates that the string is internally encoded as UTF-8. Without it, the byte value is the codepoint number and vice versa. This flag is only meaningful if the SV is SvPOK or immediately after stringification via SvPV or a similar macro. You can check and manipulate this flag with the following macros:

SvUTF8(sv)
SvUTF8_on(sv)
SvUTF8_off(sv)

This flag has an important effect on Perl's treatment of the string: if UTF-8 data is not properly distinguished, regular expressions, length, substr and other string handling operations will have undesirable (wrong) results. 💀

The problem comes when you have, for instance, a string that isn't flagged as UTF-8, and contains a byte sequence that could be UTF-8 — especially when combining non-UTF-8 and UTF-8 strings.

Never forget that the SVf_UTF8 flag is separate from the PV value; you need to be sure you don't accidentally knock it off while you're manipulating SVs. More specifically, you cannot expect to do this:

SV *sv;
SV *nsv;
STRLEN len;
char *p;

p = SvPV(sv, len);
frobnicate(p);
nsv = newSVpvn(p, len);

The char* string does not tell you the whole story, and you can't copy or reconstruct an SV just by copying the string value. Check if the old SV has the UTF8 flag set (after the SvPV call), and act accordingly:

p = SvPV(sv, len);
is_utf8 = SvUTF8(sv);
frobnicate(p, is_utf8);
nsv = newSVpvn(p, len);
if (is_utf8)
    SvUTF8_on(nsv);

In the above, your frobnicate function has been changed to be made aware of whether or not it's dealing with UTF-8 data, so that it can handle the string appropriately.

Since just passing an SV to an XS function and copying the data of the SV is not enough to copy the UTF8 flags, even less right is just passing a char * to an XS function.

For full generality, use the DO_UTF8 macro to see if the string in an SV is to be treated as UTF-8. This takes into account if the call to the XS function is being made from within the scope of use bytes. If so, the underlying bytes that comprise the UTF-8 string are to be exposed, rather than the character they represent. But this pragma should only really be used for debugging and perhaps low-level testing at the byte level. Hence most XS code need not concern itself with this, but various areas of the perl core do need to support it.

And this isn't the whole story. Starting in Perl v5.12, strings that aren't encoded in UTF-8 may also be treated as Unicode under various conditions (see "ASCII Rules versus Unicode Rules" in perlunicode). This is only really a problem for characters whose ordinals are between 128 and 255, and their behavior varies under ASCII versus Unicode rules in ways that your code cares about (see "The "Unicode Bug"" in perlunicode). There is no published API for dealing with this, as it is subject to change, but you can look at the code for pp_lc in pp.c for an example as to how it's currently done.

✨ How do I pass a Perl string to a C library? ✨

A Perl string, conceptually, is an opaque sequence of code points. Many C libraries expect their inputs to be "classical" C strings, which are arrays of octets 1-255, terminated with a NUL byte. Your job when writing an interface between Perl and a C library is to define the mapping between Perl and that library. 🫶

Generally speaking, SvPVbyte and related macros suit this task well. These assume that your Perl string is a "byte string", i.e., is either raw, undecoded input into Perl or is pre-encoded to, e.g., UTF-8.

Alternatively, if your C library expects UTF-8 text, you can use SvPVutf8 and related macros. This has the same effect as encoding to UTF-8 then calling the corresponding SvPVbyte-related macro.

Some C libraries may expect other encodings (e.g., UTF-16LE). To give Perl strings to such libraries you must either do that encoding in Perl then use SvPVbyte, or use an intermediary C library to convert from however Perl stores the string to the desired encoding.

Take care also that NULs in your Perl string don't confuse the C library. If possible, give the string's length to the C library; if that's not possible, consider rejecting strings that contain NUL bytes.

💅 What about SvPV, SvPV_nolen, etc.? 💅

Consider a 3-character Perl string $foo = "\x64\x78\x8c". Perl can store these 3 characters either of two ways:

• bytes: 0x64 0x78 0x8c
• UTF-8: 0x64 0x78 0xc2 0x8c

Now let's say you convert $foo to a C string thus:

STRLEN strlen;
char *str = SvPV(foo_sv, strlen);

At this point str could point to a 3-byte C string or a 4-byte one.

Generally speaking, we want str to be the same regardless of how Perl stores $foo, so the ambiguity here is undesirable. SvPVbyte and SvPVutf8 solve that by giving predictable output: use SvPVbyte if your C library expects byte strings, or SvPVutf8 if it expects UTF-8.

If your C library happens to support both encodings, then SvPV — always in tandem with lookups to SvUTF8! — may be safe and (slightly) more efficient.

TESTING TIP: Use utf8's upgrade and downgrade functions in your tests to ensure consistent handling regardless of Perl's internal encoding. 🧪✨

✨ How do I convert a string to UTF-8? ✨

If you're mixing UTF-8 and non-UTF-8 strings, it is necessary to upgrade the non-UTF-8 strings to UTF-8. If you've got an SV, the easiest way to do this is:

sv_utf8_upgrade(sv);

However, you must not do this, for example:

if (!SvUTF8(left))
    sv_utf8_upgrade(left);

If you do this in a binary operator, you will actually change one of the strings that came into the operator, and, while it shouldn't be noticeable by the end user, it can cause problems in deficient code. 💀

Instead, bytes_to_utf8 hands you a UTF-8-encoded copy of its string argument. ✨ Lowkey perfect when you wanna keep that data around for comparisons and stuff without doing anything rude to the original SV. There's also utf8_to_bytes for going the other way, but bestie, naturally it just fails the moment the string has any characters above 255 that can't fit in a single byte. No cap.

💖 How do I compare strings?

sv_cmp in perlapi and sv_cmp_flags in perlapi do a lexicographic comparison of two SVs, and they handle UTF-8ness properly. 🫶 But heads up: Unicode specifies a much fancier collation mechanism, and you get that via the Unicode::Collate module.

Just wanna know if two strings are equal or not equal? You can literally just use memEQ() and memNE() like always — except the strings have to both be UTF-8 or both not be UTF-8 encoded. 👑

Case-insensitive comparison? Use foldEQ_utf8() — those strings don't have to have the same UTF-8ness, so slay. 💅

🌸 Is there anything else I need to know?

Honestly? Not really. Just remember these things, bestie:

• There's no way to tell if a char * or U8 * string is UTF-8 or not. 😭 But you can tell if an SV is to be treated as UTF-8 by calling DO_UTF8 on it, after stringifying it with SvPV or a similar macro. And you can tell if an SV is actually UTF-8 (even if it's not meant to be treated as such) by peeking at its SvUTF8 flag (again, after stringifying it). Don't forget to set the flag if something should be UTF-8. Treat the flag as part of the PV, even though it technically isn't — so if you pass on the PV to somewhere, pass on the flag too. 🎀
• If a string is UTF-8, always use utf8_to_uv to get at the value, unless UTF8_IS_INVARIANT(*s) in which case you can just use *s. ✨
• When writing a character UV to a UTF-8 string, always use uv_to_utf8, unless UVCHR_IS_INVARIANT(uv)) in which case you can use *s = uv. 🌈
• Mixing UTF-8 and non-UTF-8 strings is tricky. Use bytes_to_utf8 to get a new string that's UTF-8 encoded, and then combine them. 💜

---

👑 Custom Operators

Custom operator support is an experimental feature that lets you define your own ops. It's mainly there so you can build interpreters for other languages inside the Perl core, but it also lets you do optimizations by creating "macro-ops" — ops that do the jobs of several ops that usually run together, like gvsv, gvsv, add. 🔥

This feature is implemented as a new op type, OP_CUSTOM. The Perl core doesn't "know" anything special about this op type, so it won't be involved in any optimizations. Which also means you can define your custom ops to be any op structure you want — unary, binary, list, whatever. Main character energy, honestly.

Okay but it's important to know what custom operators won't do for you. They won't let you add new syntax to Perl, directly. They won't even let you add new keywords, directly. Fr fr, they won't change the way Perl compiles a program at all. You have to do those changes yourself, after Perl has compiled the program. You do this either by manipulating the op tree using a CHECK block and the B::Generate module, or by adding a custom peephole optimizer with the optimize module. 💅

When you do this, you replace ordinary Perl ops with custom ops by creating ops with the type OP_CUSTOM and the op_ppaddr of your own PP function. This should be defined in XS code, and should look like the PP ops in pp_*.c. You are responsible for making sure your op takes the appropriate number of values from the stack, and you are responsible for adding stack marks if necessary. 👑

You should also "register" your op with the Perl interpreter so it can produce sensible error and warning messages. Since it's possible to have multiple custom ops within the one "logical" op type OP_CUSTOM, Perl uses the value of o->op_ppaddr to figure out which custom op it's dealing with. You should create an XOP structure for each ppaddr you use, set the properties of the custom op with XopENTRY_set, and register the structure against the ppaddr using Perl_custom_op_register. A trivial example might look like:

static XOP my_xop;
static OP *my_pp(pTHX);

BOOT:
    XopENTRY_set(&my_xop, xop_name, "myxop");
    XopENTRY_set(&my_xop, xop_desc, "Useless custom op");
    Perl_custom_op_register(aTHX_ my_pp, &my_xop);

The available fields in the structure are:

• xop_name — A short name for your op. It'll show up in some error messages, and it's also returned as $op->name by the B module, so it'll appear in the output of a module like B::Concise. 💖
• xop_desc — A short description of what the op does.
• xop_class — Which of the various *OP structures this op uses. This should be one of the OA_* constants from op.h, namely:
  • OA_BASEOP
  • OA_UNOP
  • OA_BINOP
  • OA_LOGOP
  • OA_LISTOP
  • OA_PMOP
  • OA_SVOP
  • OA_PADOP
  • OA_PVOP_OR_SVOP — This should be interpreted as "PVOP" only. The _OR_SVOP is there because one core PVOP, OP_TRANS, can sometimes be a SVOP instead. 🙈
  • OA_LOOP
  • OA_COP
  • OA_UNOP_AUX

The other OA_* constants should not be used. 🚫

• xop_peep — This member is of type Perl_cpeep_t, which expands to void (*Perl_cpeep_t)(aTHX_ OP *o, OP *oldop). If it's set, this function gets called from Perl_rpeep whenever ops of this type are encountered by the peephole optimizer. o is the OP that needs optimizing; oldop is the previous OP optimized, whose op_next points to o. ✨
• xop_dump — This member is a pointer to a function of type void (pTHX_ const OP *, struct Perl_OpDumpContext *). If set, this function is called by op_dump() when dumping a custom operator of this type, after the op's basic fields have been printed. This function may make use of opdump_printf() to emit additional output that may be useful for debugging. 🐛

The opaque structure pointer passed in as its final argument should be passed directly into opdump_printf().

B::Generate directly supports the creation of custom ops by name. 💅

---

🦋 Stacks

The descriptions above occasionally refer to "the stack", but there are in fact many stack-like data structures within the perl interpreter. When otherwise unqualified, "the stack" usually means the value stack.

The various stacks have different purposes and operate in slightly different ways. Their differences are noted below. 👇

💜 Value Stack

This stack stores the values that regular perl code is operating on — usually intermediate values of expressions within a statement. The stack itself is formed of an array of SV pointers.

The base of this stack is pointed to by the interpreter variable PL_stack_base, of type SV **.

The head of the stack is PL_stack_sp, and it points to the most recently-pushed item. ✨

Items get pushed to the stack using the PUSHs() macro or its variants described above: XPUSHs(), mPUSHs(), mXPUSHs() and the typed versions. Note carefully that the non-X versions of these macros don't check the size of the stack and just assume it's big enough. These must be paired with a suitable check of the stack's size, like the EXTEND macro, to make sure it's large enough. For example:

EXTEND(SP, 4);
mPUSHi(10);
mPUSHi(20);
mPUSHi(30);
mPUSHi(40);

This is slightly more performant than making four separate checks in four separate mXPUSHi() calls. 💅

As a further performance optimisation, the various PUSH macros all operate using a local variable SP, rather than the interpreter-global variable PL_stack_sp. This variable is declared by the dSP macro — though it's normally implied by XSUBs and similar, so it's rare you have to think about it directly. Once declared, the PUSH macros will operate only on this local variable, so before invoking any other perl core functions you must use the PUTBACK macro to return the value from the local SP variable back to the interpreter variable. Similarly, after calling a perl core function that may have had reason to move the stack or push/pop values on it, you must use the SPAGAIN macro, which refreshes the local SP value back from the interpreter one. 🫶

Items get popped from the stack using the POPs macro or its typed versions. There's also a macro TOPs that inspects the topmost item without removing it. 👑

Note specifically that SV pointers on the value stack do not contribute to the overall reference count of the xVs being referred to. If newly-created xVs are being pushed to the stack, you must arrange for them to be destroyed at a suitable time — usually by using one of the mPUSH* macros or sv_2mortal() to mortalise the xV. 💀

🎀 Mark Stack

The value stack stores individual perl scalar values as temporaries between expressions. Some perl expressions operate on entire lists; for that purpose we need to know where on the stack each list begins. That's the whole job of the mark stack.

The mark stack stores integers as I32 values, which are the height of the value stack at the time before the list began; so the mark itself actually points to the value stack entry one before the list. The list itself starts at mark + 1. ✨

The base of this stack is pointed to by the interpreter variable PL_markstack, of type I32 *.

The head of the stack is PL_markstack_ptr, and it points to the most recently-pushed item.

Items get pushed to the stack using the PUSHMARK() macro. Even though the stack itself stores (value) stack indices as integers, the PUSHMARK macro should be given a stack pointer directly; it'll calculate the index offset by comparing to the PL_stack_sp variable. So almost always the code to do this is:

PUSHMARK(SP);

Items get popped from the stack by the POPMARK macro. There's also a macro TOPMARK that inspects the topmost item without removing it. These macros return I32 index values directly. There's also the dMARK macro, which declares a new SV double-pointer variable called mark, which points at the marked stack slot; this is the usual macro that C code will use when operating on lists given on the stack. 💅

As noted above, the mark variable itself will point at the most recently pushed value on the value stack before the list begins, so the list itself starts at mark + 1. The values of the list may be iterated by code such as:

for(SV **svp = mark + 1; svp <= PL_stack_sp; svp++) {
  SV *item = *svp;
  ...
}

Note specifically, in the case that the list is already empty, mark will equal PL_stack_sp. 🦋

Okay so real talk about that mark variable 🫶 — because mark gets converted into a pointer on the value stack, you have to be so careful if EXTEND or any of the XPUSH macros get invoked anywhere inside the function 💀 because the stack might need to be moved to extend it, and the moment that happens your existing pointer is just... invalid. dead. gone. no cap. If that might be a problem, one possible fix is to track the mark offset as an integer instead, and then go find the mark itself later, after the stack has done its little relocation move ✨:

    I32 markoff = POPMARK;

    ...

    SP **mark = PL_stack_base + markoff;

---

✨ Temporaries Stack ✨

So, as we established, xV references sitting on the main value stack do not contribute to the reference count of that xV. Which means perl needs a whole other mechanism to keep track of when temporary values living on the stack have to be released 👑 and that, bestie, is literally the entire job of the temporaries stack.

The temporaries stack stores pointers to xVs whose reference counts are going to get decremented soon 💅.

• The base of this stack is pointed to by the interpreter variable PL_tmps_stack, of type SV **.
• The head of the stack is indexed by PL_tmps_ix, an integer that stores the index in the array of the most recently-pushed item.

There's no public API to push items directly onto the temporaries stack 🚫. Instead, the API function sv_2mortal() is what you use to mortalize an xV — it adds its address to the temporaries stack.

Same energy on the reading side: there's no public API to read values off the temporaries stack either 🙈. Instead you use the macros SAVETMPS and FREETMPS. The SAVETMPS macro establishes the base levels of the temporaries stack by capturing the current value of PL_tmps_ix into PL_tmps_floor, and saving the previous value onto the save stack 🎀. After that, whenever FREETMPS is invoked, all the temporaries that have been pushed since that level get reclaimed. iconic.

Now, yes, you'll see these two macros paired up inside an ENTER/LEAVE pair basically everywhere, but ngl you do not have to match them — it's permitted to invoke FREETMPS multiple times since the most recent SAVETMPS 🌸 (think: a loop iterating over the elements of a list). And while you can invoke SAVETMPS multiple times inside a scope pair, it's unlikely to be useful fr fr — subsequent invocations just move the temporaries floor further up, which effectively traps the existing temporaries so they only get released at the end of the scope. highkey a trap, lowkey on purpose.

---

💾 Save Stack 💾

The save stack is what perl uses to implement the local keyword and other similar behaviours — basically any cleanup operations that need to happen when you leave the current scope 🧹. Items pushed onto this stack generally capture the current value of some internal variable or state, which then gets restored when the scope is unwound due to leaving, return, die, goto or other reasons. slay.

Here's where it's different from its siblings: other perl internal stacks store individual items that are all the same type (usually SV pointers or integers), but the save stack? It's storing many different types, with multiple fields each 😳. For example, the SAVEt_INT type needs to store both the address of the int variable to restore, and the value to restore it to. Sure, you could have stored all that info using fields of a struct, but it would have to be big enough to hold three pointers in the largest case — which would waste a ton of space in most of the smaller cases 🙄.

So instead, the stack stores information in a variable-length encoding of ANY structures ✨. The final value pushed gets stored in the UV field, and that encodes the kind of item held by the preceding items — the count and types of which depend entirely on what kind of item is being stored. The kind field is pushed last specifically because that makes it the first field to be popped when unwinding items off the stack. main character field energy, honestly.

• The base of this stack is pointed to by the interpreter variable PL_savestack, of type ANY *.
• The head of the stack is indexed by PL_savestack_ix, an integer which stores the index in the array at which the next item should be pushed. (Note that this is different to most other stacks, which reference the most recently-pushed item 👀.)

Items get pushed onto the save stack using the various SAVE...() macros 💅. Lots of these macros take a variable and store both its address and its current value on the save stack, guaranteeing that value gets restored on scope exit:

    SAVEI8(i8)
    SAVEI16(i16)
    SAVEI32(i32)
    SAVEINT(i)
    ...

There are also a whole variety of other special-purpose macros that save particular types or values of interest 🌈. SAVETMPS already got its moment above. Others include SAVEFREEPV, which arranges for a PV (i.e. a string buffer) to be freed, or SAVEDESTRUCTOR, which arranges for a given function pointer to be invoked on scope exit. A full list of such macros can be found in scope.h 📄.

There is no public API for popping individual values or items off the save stack 🙅. Instead, via the scope stack, the ENTER and LEAVE pair forms the way to start and stop nested scopes. Leaving a nested scope via LEAVE will restore all of the saved values that had been pushed since the most recent ENTER 👑.

---

🪜 Scope Stack 🪜

Just like the mark stack pairs with the value stack, the scope stack forms a pair with the save stack 💞. The scope stack stores the height of the save stack at which nested scopes begin, and that lets the save stack be unwound back to that point when the scope is left. chef's kiss.

When perl is built with debugging enabled 🐛, there's a second part to this stack storing human-readable string names describing the type of stack context. Each push operation saves the name as well as the height of the save stack, and each pop operation checks the topmost name against what's expected — causing an assertion failure if the name doesn't match 😱. it's giving vibe check, honestly.

• The base of this stack is pointed to by the interpreter variable PL_scopestack, of type I32 *. If enabled, the scope stack names are stored in a separate array pointed to by PL_scopestack_name, of type const char **.
• The head of the stack is indexed by PL_scopestack_ix, an integer which stores the index of the array (or arrays) at which the next item should be pushed. (Again — note that this is different to most other stacks, which reference the most recently-pushed item 👀.)

Values are pushed onto the scope stack using the ENTER macro, which begins a new nested scope 🌱. Any items pushed to the save stack are then restored at the next nested invocation of the LEAVE macro. periodt.

---

🌀 Dynamic Scope and the Context Stack 🌀

Note: this section describes a non-public internal API that is subject to change without notice. ⚠️

🥰 Introduction to the context stack

In Perl, dynamic scoping refers to the runtime nesting of things like subroutine calls, evals, etc., as well as the entering and exiting of block scopes 🦋. For example, the restoring of a localised variable is determined by the dynamic scope.

Perl tracks the dynamic scope with a data structure called the context stack — an array of PERL_CONTEXT structures, which is itself a big union for all the types of context 👑. Whenever a new scope is entered (a block, a for loop, a subroutine call, whatever), a new context entry gets pushed onto the stack. Similarly, when leaving a block or returning from a subroutine call etc., a context gets popped. And since the context stack represents the current dynamic scope, it can be searched 🔍. For example:

• next LABEL searches back through the stack looking for a loop context that matches the label
• return pops contexts until it finds a sub or eval context or similar
• caller examines sub contexts on the stack

Every single context entry is labelled with a context type, cx_type 🏷️. Typical context types are CXt_SUB, CXt_EVAL etc., plus CXt_BLOCK and CXt_NULL, which represent a basic scope (as pushed by pp_enter) and a sort block. The type is what determines which part of the context union is actually valid.

The main division inside the context struct is between a substitution scope (CXt_SUBST) and block scopes, which is literally everything else 💀. The former is used only while executing s///e, and won't be discussed any further here.

All the block scope types share a common base, which corresponds to CXt_BLOCK ✨. This stores the old values of various scope-related variables like PL_curpm, as well as info about the current scope such as gimme. On scope exit, those old variables get restored.

Particular block scope types store extra per-type information 📦. For example, CXt_SUB stores the currently executing CV, while the various for loop types might hold the original loop variable SV. On scope exit, the per-type data gets processed — like the CV having its reference count decremented, and the original loop variable being restored.

The macro cxstack returns the base of the current context stack, while cxstack_ix is the index of the current frame within that stack 🫶.

And here's a fun one: the context stack is actually part of a stack-of-stacks system 😳 — whenever something unusual happens, like calling a DESTROY or tie handler, a new stack gets pushed, then popped at the end. no cap.

Also important: the API described here changed considerably in perl 5.24 👀. Before that, big macros like PUSHBLOCK and POPSUB were used; in 5.24 they were replaced by the inline static functions described below. Plus, the ordering and detail of how these macros/functions work changed in many ways, often subtly. In particular, they didn't handle saving the savestack and temps stack positions, and they required additional ENTER, SAVETMPS and LEAVE compared to the new functions. The old-style macros will not be described any further here. moving on bestie 💅.

📥 Pushing contexts

For pushing a new context, the two basic pieces are:

• cx = cx_pushblock(), which pushes a new basic context block and returns its address
• a whole family of similar functions with names like cx_pushsub(cx), which populate the additional type-dependent fields in the cx struct

And note that CXt_NULL and CXt_BLOCK don't have their own push functions 🚫, because they don't store any data beyond what's pushed by cx_pushblock.

The fields of the context struct and the arguments to the cx_* functions are subject to change between perl releases — they just represent whatever is convenient or efficient for that particular release 🤷.

A typical context stack pushing shows up in pp_entersub; here's a simplified and stripped-down example of a non-XS call, with comments showing roughly what each function does ✨:

 dMARK;
 U8 gimme      = GIMME_V;
 bool hasargs  = cBOOL(PL_op->op_flags & OPf_STACKED);
 OP *retop     = PL_op->op_next;
 I32 old_ss_ix = PL_savestack_ix;
 CV *cv        = ....;

 /* ... make mortal copies of stack args which are PADTMPs here ... */

 /* ... do any additional savestack pushes here ... */

 /* Now push a new context entry of type 'CXt_SUB'; initially just
  * doing the actions common to all block types: */

 cx = cx_pushblock(CXt_SUB, gimme, MARK, old_ss_ix);

            /* this does (approximately):
                CXINC;              /* cxstack_ix++ (grow if necessary) */
                cx = CX_CUR();      /* and get the address of new frame */
                cx->cx_type        = CXt_SUB;
                cx->blk_gimme      = gimme;
                cx->blk_oldsp      = MARK - PL_stack_base;
                cx->blk_oldsaveix  = old_ss_ix;
                cx->blk_oldcop     = PL_curcop;
                cx->blk_oldmarksp  = PL_markstack_ptr - PL_markstack;
                cx->blk_oldscopesp = PL_scopestack_ix;
                cx->blk_oldpm      = PL_curpm;
                cx->blk_old_tmpsfloor = PL_tmps_floor;

                PL_tmps_floor        = PL_tmps_ix;
            */

then the new frame gets its sub-specific glow-up:

        /* then update the new context frame with subroutine-specific info,
         * such as the CV about to be executed: */

        cx_pushsub(cx, cv, retop, hasargs);

            /* this does (approximately):
                cx->blk_sub.cv          = cv;
                cx->blk_sub.olddepth    = CvDEPTH(cv);
                cx->blk_sub.prevcomppad = PL_comppad;
                cx->cx_type            |= (hasargs) ? CXp_HASARGS : 0;
                cx->blk_sub.retop       = retop;
                SvREFCNT_inc_simple_void_NN(cv);
            */

okay so heads up bestie 💖 cx_pushblock() sets two new floors for you: one for the args stack (set to MARK) and one for the temps stack (set to PL_tmps_ix). While you're executing at this scope level, every nextstate (amongst others) will reset the args and tmps stack levels right back down to these floors. And here's the spicy bit: since cx_pushblock uses the current value of PL_tmps_ix rather than having it passed in as an arg, that alone dictates at what point cx_pushblock should be called. Translation: any new mortals that should be freed only on scope exit (rather than at the next nextstate) gotta be created first. No cap. ✨

Most callers of cx_pushblock just set the new args stack floor to the top of the previous stack frame, but CXt_LOOP_LIST is built different 🦋 — it stores the items being iterated over on the stack, so it sets blk_oldsp to the top of those items instead. And note, contrary to its name, blk_oldsp doesn't always represent the value to restore PL_stack_sp to on scope exit. Iconic but misleading, we love that for it.

Also peep the early capture of PL_savestack_ix into old_ss_ix, which later gets passed as an arg to cx_pushblock. Why, bestie? In the case of pp_entersub, it's because — even though most values needing saving live in fields of the context struct — one extra value needs saving only when the debugger is running, and it does not make sense to bloat the struct for this rare case. So it goes on the savestack instead. And since this value gets calculated and saved before the context is pushed, you have to pass the old value of PL_savestack_ix to cx_pushblock to make sure that saved value gets freed during scope exit. For most users of cx_pushblock, where nothing needs pushing on the save stack, PL_savestack_ix is just passed directly as an arg to cx_pushblock. Easy 💅

Where possible, values should be saved in the context struct rather than on the save stack; it's much faster that way. Periodt.

Normally cx_pushblock should be immediately followed by the appropriate cx_pushfoo, with nothing between them. Why? Because if code in-between could die (e.g. a warning upgraded to fatal 😭), then the context stack unwinding code in dounwind would see (in the example above) a CXt_SUB context frame, but without all the subroutine-specific fields set — and crashes would soon ensue. It's giving disaster.

Where the two must be separate, initially set the type to CXt_NULL or CXt_BLOCK, and later change it to CXt_foo when doing the cx_pushfoo. That is exactly what pp_enteriter does, once it's determined which type of loop it's pushing. Main character behavior fr fr 👑

---

Popping contexts

Contexts get popped using cx_popsub() etc. and cx_popblock(). But heads up — unlike cx_pushblock, neither of these functions actually decrements the current context stack index. That part is done separately using CX_POP(). Teamwork, babe 🫶

There are two main ways that contexts are popped:

1. During normal execution as scopes are exited, functions like pp_leave, pp_leaveloop and pp_leavesub process and pop just one context using cx_popfoo and cx_popblock.
2. On the other hand, things like pp_return and next may have to pop back several scopes until a sub or loop context is found, and exceptions (such as die) need to pop back contexts until an eval context is found. Both of these are accomplished by dounwind(), which is capable of processing and popping all contexts above the target one.

Here's a typical example of context popping, as found in pp_leavesub (simplified slightly):

    U8 gimme;
    PERL_CONTEXT *cx;
    SV **oldsp;
    OP *retop;

    cx = CX_CUR();

    gimme = cx->blk_gimme;
    oldsp = PL_stack_base + cx->blk_oldsp; /* last arg of previous frame */

    if (gimme == G_VOID)
        PL_stack_sp = oldsp;
    else
        leave_adjust_stacks(oldsp, oldsp, gimme, 0);

    CX_LEAVE_SCOPE(cx);
    cx_popsub(cx);
    cx_popblock(cx);
    retop = cx->blk_sub.retop;
    CX_POP(cx);

    return retop;

Those steps are in a very specific order, designed to be the reverse order of when the context was pushed. 🔄

Step one: copy and/or protect any return arguments, and free any temps in the current scope. Scope exits like an rvalue sub normally return a mortal copy of their return args (as opposed to lvalue subs). It is important to make this copy before the save stack is popped or variables are restored — otherwise bad things like the following can happen:

    sub f { my $x =...; $x }  # $x freed before we get to copy it
    sub f { /(...)/;    $1 }  # PL_curpm restored before $1 copied

😭 not the freed-before-copy situation.

Although we want to free any temps at the same time, we have to be careful not to free any temps which are keeping return args alive — nor to free the temps we just created while mortal copying return args. Fortunately, leave_adjust_stacks() is capable of making mortal copies of return args, shifting args down the stack, and only processing those entries on the temps stack that are safe to do so. Multitasking queen 🌸

In void context no args are returned, so it's more efficient to skip calling leave_adjust_stacks(). Also in void context, a nextstate op is likely to be imminently called which will do a FREETMPS, so there's no need to do that either. Efficiency slay ✨

Next step: pop savestack entries. CX_LEAVE_SCOPE(cx) is just defined as LEAVE_SCOPE(cx->blk_oldsaveix). And note — during the popping, it's possible for perl to call destructors, call STORE to undo localisations of tied vars, and so on. Any of these can die or call exit(). In this case, dounwind() will be called, and the current context stack frame will be re-processed. Thus it is vital that all steps in popping a context are done in such a way to support reentrancy. The other alternative — decrementing cxstack_ix before processing the frame — would lead to leaks and the like if something died halfway through, or overwriting of the current frame. We do not do that. We are not that girl. 💅

CX_LEAVE_SCOPE itself is safely re-entrant: if only half the savestack items have been popped before dying and getting trapped by eval, then the CX_LEAVE_SCOPEs in dounwind or pp_leaveeval will continue where the first one left off. Resilient queen 🎀

Next: the type-specific context processing; in this case cx_popsub. In part, this looks like:

    cv = cx->blk_sub.cv;
    CvDEPTH(cv) = cx->blk_sub.olddepth;
    cx->blk_sub.cv = NULL;
    SvREFCNT_dec(cv);

which is where it's processing the just-executed CV. Note that before it decrements the CV's reference count, it nulls the blk_sub.cv. That means if it re-enters, the CV won't be freed twice. It also means you can't rely on such type-specific fields having useful values after the return from cx_popfoo. Gone, girl. Bye 💀

Next, cx_popblock restores all the various interpreter vars to their previous values or previous high water marks; it expands to:

    PL_markstack_ptr = PL_markstack + cx->blk_oldmarksp;
    PL_scopestack_ix = cx->blk_oldscopesp;
    PL_curpm         = cx->blk_oldpm;
    PL_curcop        = cx->blk_oldcop;
    PL_tmps_floor    = cx->blk_old_tmpsfloor;

Note that it doesn't restore PL_stack_sp; as mentioned earlier, which value to restore it to depends on the context type (specifically for (list) {}), and what args (if any) it returns — and that will already have been sorted out earlier by leave_adjust_stacks(). She doesn't double-dip. 💖

Finally, the context stack pointer is actually decremented by CX_POP(cx). After this point, it's possible that the current context frame could be overwritten by other contexts being pushed. Although things like ties and DESTROY are supposed to work within a new context stack, it's best not to assume this. Indeed, on debugging builds, CX_POP(cx) deliberately sets cx to null to detect code that is still relying on the field values in that context frame. 👀 Caught in 4K. Note in the pp_leavesub() example above, we grab blk_sub.retop before calling CX_POP.

---

Redoing contexts

And finally, there's cx_topblock(cx), which acts like a super-nextstate as regards to resetting various vars to their base values. It's used in places like pp_next, pp_redo and pp_goto where — rather than exiting a scope — we want to re-initialise the scope. As well as resetting PL_stack_sp like nextstate, it also resets PL_markstack_ptr, PL_scopestack_ix and PL_curpm. Note that it doesn't do a FREETMPS. Glow-up, not a clean-up ✨

---

Reference-counted argument stack

Introduction

As of perl 5.40, there is a build option, PERL_RC_STACK, not enabled by default, which requires that items pushed onto, or popped off the argument stack have their reference counts adjusted. It is intended that eventually this will be the default way (and finally the only way) to configure perl. The future is coming, bestie 🔮

The macros which manipulate the stack such as PUSHs() and POPs() don't adjust the reference count of the SV. Most of the time this is fine, since something else is keeping the SV alive while on the argument stack — such a pointer from the TEMPs stack, or from the pad (e.g. a lexical variable or a PADTMP). Occasionally this can go horribly wrong. 😱 For example, this code:

    my @a = (1,2,3);
    sub f { @a = (); print "(@_)\n" };
    f(@a, 4);

may print undefined or random freed values, since some of the elements of @_, which have been aliased to the elements of @a, have been freed. 💀 PERL_RC_STACK is intended to fix this by making each SV pointer on the argument stack increment the reference count (RC) of the SV by one. Fixed. Slayed. Periodt 💅

In this new environment, unmodified existing PP and XS functions — all of them written back when everybody assumed a non reference-counted stack (we say non-RC for short) — get called through special wrapper functions that adjust the stack before and after they run. 💅

Right now there's literally no API to write an RC XS function, so all XS code keeps being called via a wrapper (which makes it slightly slower, ngl), but here's the slay: it means CPAN distributions containing XS code just keep working without any modification. 🎀

PP functions though? Different story, bestie. Whether they live in perl core, or in XS functions used to implement custom ops or to override the PP functions for built-in ops, they need special handling. For that second group, they can just be wrapped — least work, but there's a performance hit. Longer term (and for core PP functions specifically), they need unwrapping and rewriting using a new API. With that, the old macros such as PUSHs() have been replaced with a new set of (mostly inline) functions sharing a common prefix, like rpp_push_1(). "RPP" stands for "reference-counted push and pop functions". ✨

The new functions modify the reference count on PERL_RC_STACK builds, and leave them unadjusted otherwise. So in core they generally work in both cases, while in XS code they're portable to older perl versions via PPPort (XXX assuming that they get been added to PPPort).

The rest of this section is mainly about how to convert existing PP functions, and how to write new PP functions so they use the new rpp_ API. 💖

A reference-counted perl can be built using the PERL_RC_STACK define. For development and debugging purposes, it's best to enable leaking scalar debugging too, because that displays extra information about scalars that have leaked or been prematurely freed. 🕵️

Configure -DDEBUGGING \
  -Accflags='-DPERL_RC_STACK -DDEBUG_LEAKING_SCALARS'

✨ Reference counted stack states ✨

In the new regime, the current argument stack can be in one of three states, and you can tell which one with the expression shown. 🔍

• not reference-counted — the vibe check is:

!AvREAL(PL_curstack)

In this case, perl will assume when emptying the stack (such as during a croak()) that the items on it don't need freeing. This is the traditional perl behaviour. On PERL_RC_STACK builds, such stacks will be rarely encountered.

• fully reference-counted — the vibe check is:

AvREAL(PL_curstack) && !PL_curstackinfo->si_stack_nonrc_base

All the items on the stack are reference counted, and will be freed by functions like rpp_popfree_1() or if perl croak()s. This is the normal state of the stack in PERL_RC_STACK builds.

• partially reference-counted (split) — the vibe check is:

AvREAL(PL_curstack) && PL_curstackinfo->si_stack_nonrc_base > 0

In this case, items on the stack from the index si_stack_nonrc_base upwards are non-RC; those below are RC. This state occurs when a PP or XS function has been wrapped. In this case, the wrapper function pushes a non-RC copy of the arg pointers above the cut, then calls the real function. When that returns, the wrapper function bumps up the RC of any returned args. See below for more details. 👇

Note that perl uses a stack-of-stacks, and the AvREAL() and si_stack_nonrc_base states are per stack. When perl starts up, the main stack is RC, but by default, new stacks pushed in XS code via PUSHSTACKi() are non-RC, so it's quite possible to get a mixture. The perl core itself uses the new push_stackinfo() function which replaces PUSHSTACKi() and allows you to specify that the new stack should be RC by default. (XXX core mostly hasn't actually been updated yet to use push_stackinfo())

Most places in the core assume a particular RC environment. In particular, it's assumed that within a runops loop, all the PP functions are RC-aware — either because they've been (re)written to be aware, or because they've been wrapped. Whenever a runops loop is entered via CALLRUNOPS(), it will check the current state of the stack, and if it's not fully RC, it'll temporarily update its contents to be fully RC before entering the main runops loop. Then, if necessary, it'll restore the stack to its old state on return. This means functions like call_sv(), which can be called from any environment (e.g. RC core, or wrapped and temporarily non-RC XS code), will always do the Right Thing when invoking the runops loop, no matter what the current stack state is. 👑

Similarly, croaks and the like (which can occur anywhere) have to be able to handle both stack types. So there are a few places in core — call_sv(), eval_sv() etc, Perl_die_unwind() and S_my_exit_jump() — which have been specially crafted to handle both cases; everything else can assume a fixed environment.

🎁 Wrapping 🎁

Normally a core PP function is declared like this:

PP(pp_foo)
{
    ...
}

This expands to something like:

OP* Perl_pp_foo(pTHX)
{
    ...
}

When such a function needs to be wrapped, it is instead declared as:

PP_wrapped(pp_foo, nargs, nlists)
{
    ...
}

which on non-RC builds expands to the same as PP() (the extra args are ignored). On RC builds it expands to something like

OP* Perl_pp_foo(pTHX)
{
    return Perl_pp_wrap(aTHX_ S_Perl_pp_foo_norc, nargs, nlists);
}

static OP* S_Perl_pp_foo_norc(pTHX)
{
    ...
}

Here the externally visible PP function calls pp_wrap(), which adjusts the stack contents, then calls the hidden real body of the PP function, then on return, adjusts the stack back. 🔁

There is an API macro, XSPP_wrapped(), intended for use on PP functions declared in XS code. It is identical to PP_wrapped(), except that it doesn't prepend a Perl_ prefix to the function name.

The nargs and nlists parameters to the macro are numeric constants or simple expressions which specify how many arguments the PP function expects, or how many lists it expects. For example:

PP_wrapped(pp_add, 2, 0);     /* consumes two args off the stack */

PP_wrapped(pp_readline,       /* consumes one or two args */
        ((PL_op->op_flags & OPf_STACKED) ? 2 : 1), 0);

PP_wrapped(pp_push, 0, 1);    /* consumes one list */

PP_wrapped(pp_aassign, 0, 2); /* consumes two lists */

To understand what pp_wrap() does, consider calling Perl_pp_foo() which expects three arguments. On entry the stack may look like:

... A+ B+ C+

(where the + indicates that the pointers to A, B and C are each reference counted). The wrapper function pp_wrap() marks a cut at the current stack position using si_stack_nonrc_base, then, based on the value of nargs, pushes a copy of those three pointers above the cut:

... A+ B+ C+ | A0 B0 C0

(where the 0 indicates that the pointers aren't RC), then calls the real PP function, S_Perl_pp_foo_norc(). That function processes A, B and C, pops them off the stack, and pushes some result SVs. None of this manipulation adjusts any RCs. On return to pp_wrap(), the stack may look something like:

... A+ B+ C+ | X0 Y0

The wrapper function bumps up the RCs of X and Y, decrements A B C, shifts the results down and sets si_stack_nonrc_base to zero, leaving the stack as:

... X+ Y+

In places like pp_entersub(), a similar wrapping (via the functions rpp_invoke_xs() and then xs_wrap()) is done when calling XS subs.

When nlists is positive, a similar action takes place, except that the mark stack is examined (and adjusted) in order to determine the number of args that need copying. 📚

A complex calling environment might have multiple nested stacks with different RC states. Perl starts off with an RC stack. Then for example, pp_entersub() is called, which (via xs_wrap()) splits the stack and executes the XS function in a non-RC environment. That function may call PUSHSTACKi(), which creates a new non-RC stack, then calls call_sv(), which does CALLRUNOPS(), which causes the new stack to temporarily become RC. Then a tied method is called, which pushes a new RC stack, and so on. (XXX currently tied methods actually push a non-RC stack. To be fixed soon).

🌈 (Re)writing a PP function using the rpp_() API 🌈

Wrapping a PP function has a performance overhead, and it's there mainly as a temporary crutch. Eventually, PP functions should be updated to use rpp_() functions, and any new PP functions should be written this way from scratch and thus never need wrapping at all. 💪

A couple examples of core PP functions being converted can be seen in the commits v5.39.1-304-g205fcd8410 and v5.39.1-303-g2fe263a83a, which demonstrate a unary and a binary op being converted (pp_not() and pp_and()).

The traditional PP stack API consisted of a dSP declaration, plus a number of macros to push, pop and extend the stack. A very simplified pp_add() function might look something like:

PP(pp_add)
{
    dSP;
    dTARGET;
    IV right = SvIV(POPs);
    IV left  = SvIV(POPs);
    TARGi(left + right, 1);
    PUSHs(TARG);
    PUTBACK;
    return NORMAL;
}

which expands to something like:

{
    SV **sp = PL_stack_sp;
    SV *targ = PAD_SV(PL_op->op_targ);
    IV right = SvIV(*sp--);
    IV left  = SvIV(*sp--);
    sv_setiv(targ, left + right);
    *++sp = targ;
    PL_stack_sp = sp;
    return PL_op->op_next;
}

The whole dSP thing harks back to the days before decent optimising compilers. It was always error-prone, e.g. if you forgot a PUTBACK or SPAGAIN. 😭 The new API always just accesses PL_stack_sp directly. In fact the first step of upgrading a PP function is always to remove the dSP declaration. This has the happy side effect that any old-style macros left in the pp function which implicitly use sp will become compile errors. The existence of a dSP somewhere in core is a good sign that that function still needs updating. 🚩

An obvious question is: why not just modify the definitions of the PUSHs() etc macros to modify reference counts on RC builds? The basic problem is that an SV may now be kept alive only by a single reference count from the stack (formerly, they tended to be on the TEMPs stack too). So in code like:

SV *sv = POPs;
IV i = SvIV(sv);

including an SvREFCNT_dec() in the POPs macro definition would cause sv to be freed immediately, before its integer value can be read. 💀

So like, the one tiny catch with this whole new regime? perl can croak at basically any moment during execution — bestie, literally any. (Case in point: that SvIV() call up there might go and trigger FETCH() on a tied variable, and then that croaks. betrayal.) Which means at all times the RC of every single SV has to be properly accounted for. No vibes-based refcounting, okay??

In the example above, a naive "let me just avoid a premature free of sv" approach might look like:

    SV *sv = *PL_stack_sp--;
    IV i = SvIV(sv);
    SvREFCNT_dec(sv); // got i, so ok to free sv now

…and that is the trap, because if SvIV() triggers a croak, sv just leaks. gone. never seen again. 🙈

So to dodge that, the new regime has one general outline and it's giving ✨discipline✨: arguments get left on the stack until you're finished with them, and then they get removed and their reference count adjusted right at that point. So with the new API, the pp_add() function looks something like:

    {
        dTARGET;
        IV right = SvIV(PL_stack_sp[ 0]); // NB: arguments left on stack
        IV left  = SvIV(PL_stack_sp[-1]);
        TARGi(left + right, 1);
        rpp_replace_2_1(targ);
        return NORMAL;
    }

The rpp_replace_2_1() function pops two values off the stack and pushes one new value on, adjusting reference counts as appropriate the whole time (depending on whether it was built with PERL_RC_STACK or not). multi-tasking icon.

The rpp_() functions in the new API get a full breakdown below, but here's the TL;DR bestie cheat sheet:

• rpp_extend(n) — the glow-up of EXTEND(SP, n)
• rpp_push_1(sv) — the glow-up of PUSHs(sv)
• rpp_push_2(sv1, sv2)) — the glow-up of PUSHs(sv1); PUSHs(sv2)
• rpp_xpush_1(sv) — the glow-up of XPUSHs(sv)
• rpp_xpush_2(sv1, sv2)) — the glow-up of EXTEND(SP,2); PUSHs(sv1); PUSHs(sv2);
• rpp_push_1_norc(sv) — the glow-up of mPUSHs(sv) — on RC builds it skips the RC++, and on non-RC builds it mortalises instead
• rpp_popfree_1() — the glow-up of (void)POPs;
• rpp_popfree_2() — the glow-up of (void)POPs; (void)POPs;
• rpp_popfree_to(svp) — the glow-up of PL_stack_sp = svp;
• rpp_obliterate_stack_to(ix) — no cute one-liner, see the description below
• sv = rpp_pop_1_norc() — the glow-up of sv = SvREFCNT_inc(POPs)
• rpp_replace_1_1(sv) — the glow-up of (void)POPs; PUSHs(sv);
• rpp_replace_2_1(sv) — the glow-up of (void)POPs; (void)POPs; PUSHs(sv);
• rpp_replace_at(sp, sv) — the glow-up of *sp = sv;
• rpp_replace_at_norc(sp, sv) — the glow-up of *sp = sv_2mortal(sv);
• rpp_context(mark, gimme, extra) — the glow-up of SP -= extra; plus imposing void/scalar/list context on the return args, i.e. SP = (gimme == G_VOID) ? mark : ....
• rpp_try_AMAGIC_1() — the glow-up of tryAMAGICun_MG()
• rpp_try_AMAGIC_2() — the glow-up of tryAMAGICbin_MG()
• rpp_is_lone(sv) — the glow-up of SvTEMP(sv) && SvREFCNT(sv) == 1
• rpp_stack_is_rc() — no equivalent, she's brand new 💅
• rpp_invoke_xs(cv) — the glow-up of CvXSUB(cv)(aTHX_ cv);
• (no replacement) — for dATARGET … just write the macro body out in full, sis

---

There are also some _NN variants, and these ones assume that any items being removed from the stack are non-NULL, which makes them ever so slightly more efficient. efficiency queen 👑

• rpp_popfree_1_NN()
• rpp_popfree_2_NN()
• rpp_popfree_to_NN(svp)
• rpp_replace_1_1_NN(sv)
• rpp_replace_2_1_NN(sv)
• rpp_replace_at_NN(sp, sv)
• rpp_replace_at_norc_NN(sp, sv)

And then there are a few _IMM variants, which are expecting the single pushed-or-replaced value to be an immortal, like &PL_sv_undef — this skips incrementing the ref count of the immortal SV, because honestly? doesn't matter. If the ref count of that SV prematurely hits zero, sv_free2() will just resurrect it. slay. 🦋 Heads up though: not every variant exists — if there's no suitable one for what you need, just use a normal _1 version, totally fine, just a lil slower.

• rpp_push_IMM(&PL_sv_undef)
• rpp_xpush_IMM(&PL_sv_zero)
• rpp_replace_1_IMM_NN(&PL_sv_yes)
• rpp_replace_2_IMM_NN(&PL_sv_no)

---

Other new C and perl functions related to reference-counted stacks, come get y'all juice:

• push_stackinfo(type,rc) — the glow-up of PUSHSTACKi(type)
• pop_stackinfo() — the glow-up of POPSTACK()
• switch_argstack(to) — the glow-up of SWITCHSTACK(from,to)
• (Internals::stack_refcounted() & 1) — tells you perl was built with PERL_RC_STACK

Some of these new functions are, not gonna lie, trivial. But bestie, still use them instead of writing direct code — because they work on both RC and non-RC builds, and they may do extra checks and assertions on DEBUGGING builds. free reliability. 🫶

---

Now, important 😭: rpp_popfree_1() and friends are not direct replacements for POPs. The rpp_() variants don't return a value — they're meant to be called when you're done with the SV. So like:

    SV *sv = POPs;
    ... do stuff with sv ...

becomes

    SV *sv = *PL_stack_sp;
    ... do stuff with sv ...
    rpp_popfree_1(); /* does SvREFCNT_dec(*PL_stack_sp--) */

The rpp_replace_M_N() functions are little shortcuts for popping and freeing M items and then pushing and bumping up the RCs of N items. And yes, they handle the messy edge cases, like when the old and new SV happen to be the same one. iconic behaviour, no drama.

rpp_replace_at(sp, sv) is basically rpp_replace_1_1(), except it replaces an SV at an address in the stack instead of at the top.

rpp_replace_at_norc(sp, sv) is like rpp_replace_at(), except it assumes sv already has a bumped reference count. So a bit like rpp_push_1_norc() (see below 👇), it doesn't bother increasing sv's ref count — or, on non-RC builds, it mortalises it instead.

rpp_popfree_to(svp) was designed to replace code like

    PL_stack_sp = PL_stack_base + cx->blk_oldsp;

…which typically shows up in list ops or scope exits when the arguments are finished with. And, like, if you leave that alone? All the SVs above oldsp would leak. 💀 The new approach is:

    rpp_popfree_to(PL_stack_base + cx->blk_oldsp);

There's a rarely-used variant of this too: rpp_obliterate_stack_to(), which pops the stack back to the specified index regardless of the current RC state of the stack. So for example, if the stack is split, it'll only adjust the RCs of the SVs that are below the split point — whereas rpp_popfree_to() would mindlessly free all the SVs (on RC builds anyway). But for normal PP functions you should only ever use rpp_popfree_to(), because it's faster. fastest girl in the room 🏃‍♀️💨

---

And no, there are no new equivalents for all those convenience macros like POPi() and (shudder) dPOPPOPiirl(). Replace them with the rpp_() functions above, and make the conversions and variable declarations explicit. So dPOPPOPiirl() becomes:

    IV right = SvIV(PL_stack_sp[ 0]);
    IV left  = SvIV(PL_stack_sp[-1]);
    rpp_popfree_2();

---

A couple of the rpp_() functions with norc in the name don't adjust the reference count on RC builds. (But conversely, they do on non-RC builds. chaos, but make it consistent. 🙃)

rpp_push_1_norc(sv) does a simple *++PL_stack_sp = sv on RC builds. It's typically used to "root" a newly-created SV, which already has an RC of 1. On non-RC builds it mortalises the SV instead. So for example, code that used to look like

    mPUSHs(newSViv(i));

and which expanded to the equivalent of:

    PUSHs(sv_2mortal(newSViv(i));

should be rewritten as:

    rpp_push_1_norc(newSViv(i));

This is because newSViv() and similar create a new SV with a reference count one too high (1 rather than 0). That count then gets donated to the stack by pushing it. And conversely, on non-RC builds, the count gets donated to the TEMPs stack instead. generous. 💖

Similarly, on RC builds, sv = rpp_pop_1_norc() does a simple sv = *PL_stack_sv-- without adjusting the reference count, while on non-RC builds it actually increments the SV's reference count. It's meant for cases where you immediately want to bump the reference count back up after popping — like when the SV is about to be immediately embedded somewhere. For example, this code:

    SV *sv = PL_stack_sp[0];
    SvREFCNT_inc(sv);
    av_store(av, i, sv); /* in real life should check return value */
    rpp_popfree_1();

can be written more efficiently as

    av_store(av, i, rpp_pop_1_norc());

And by using this function, the code works correctly on both RC and non-RC builds. range, honey. 🌈

---

A super common operation on list ops is imposing void, scalar or list context on the return arguments — possibly discarding all of them, or all except one. That's the whole job of rpp_context(mark, gimme, extra). First step (for convenience and efficiency) it notionally pops extra args off the stack. Then:

• for list context — leaves everything exactly as it is, no notes
• for void context — the stack pointer gets reset to mark, and everything above it is popped
• for scalar context — the top argument (or &PL_sv_undef) gets moved from the top down to mark+1, and everything above is discarded

---

The macros that show up at the start of many PP functions to check for unary or binary op overloading (among other things) have been replaced with the rpp_try_AMAGIC_1() and _2() inline functions. The difference? Now it's the calling PP function that chooses whether to return immediately — the return is no longer hidden away inside the macro. accountability era. 💅

The rpp_invoke_xs() function calls the XS function associated with the CV, but it may do so via a wrapper function to adjust the stack as necessary.

And in the spirit of hiding away less inside macros: dATARGET didn't get a replacement. Where its effect is needed, it gets written out in full now — go look at pp_add() for an example.

---

Finally, a couple of the rpp_() functions don't manipulate the stack at all, they just give you information. gossip, but productive. ☕

rpp_is_lone(sv) tells you whether sv — assumed to still be on the stack — is being kept alive only by a single reference-counted pointer from the argument and/or temps stacks. Which makes it a candidate for some optimisations (like skipping the copying of return arguments from a subroutine call). lil glow-up opportunity ✨

rpp_stack_is_rc() tells you whether the current stack is currently reference-counted. It's used mainly in a few places like call_sv(), which can be called from anywhere and therefore has to deal with both cases.

So for example, instead of using rpp_xpush_1(), call_sv() has lines like:

        rpp_extend(1);
        *++PL_stack_sp = sv;
    #ifdef PERL_RC_STACK
        if (rpp_stack_is_rc())
            SvREFCNT_inc_simple_void_NN(sv);
    #endif

…which works on both standard builds and RC builds, and works whether call_sv() is called from a standard PP function (where rpp_stack_is_rc() is true) or from a wrapped PP or XS function (where rpp_stack_is_rc() is false).

ngl bestie, you're unlikely to ever need to use this function, because in most places — like PP or XS functions — it's always RC or non-RC respectively. In fact, on debugging builds under PERL_RC_STACK, PUSHs() and similar macros include an assert(!rpp_stack_is_rc()), while rpp_push_1() and similar functions have assert(rpp_stack_is_rc()). so it's got receipts either way 🧾

Honestly? Those macros that push new stackinfos? Gone, bestie. They got replaced with inline functions that don't need dSP to be in scope, and their names are way less ambiguous — they literally make it obvious that a whole new stackinfo is getting pushed, not just some sort of stack. And push_stackinfo() picked up a boolean argument too, telling it whether the new argument stack should be reference-counted or not. Miss the old days? For backwards compatibility, PUSHSTACKi(type) is just defined as push_stackinfo(type, 0). 🫶

Some test scripts do the leak-check thing — they test that the reference count of a particular variable has an expected value. If that number comes out different on a perl built with PERL_RC_STACK, you can call the perl function Internals::stack_refcounted(). It returns an integer, and the lowest bit of it tells you perl was built with PERL_RC_STACK. All the other bits are reserved for future use, so mask them out, okay? 💅

---

✨ Slab-based operator allocation ✨

Note: this section describes a non-public internal API that is subject to change without notice. Don't build your whole personality on it. 💀

Perl's internal error-handling mechanisms implement die (and its internal equivalents) using longjmp. So if one of those fires off during lexing, parsing, or compilation, we must make sure that any ops allocated as part of the compilation process get freed. (Older Perl versions did not handle this situation properly — when a parse failed, they'd leak ops that were sitting in C auto variables and weren't linked anywhere else. rip. 😭)

To handle this, Perl uses op slabs that are attached to the currently-compiling CV. A slab is a chunk of allocated memory. New ops get allocated as regions of the slab. If the slab fills up, a new one is created (and linked from the previous one). Then when an error happens and the CV is freed, any ops still remaining get freed too. ✨

Each op is preceded by two pointers: one points to the next op in the slab, and the other points to the slab that owns it. That next-op pointer is needed so Perl can iterate over a slab and free all of its ops. (Op structures come in different sizes, so a slab's ops can't just be treated as a dense array. 🙈) The slab pointer is needed so we can access a reference count on the slab: when the last op on a slab is freed, the slab itself gets freed.

The slab allocator puts the ops at the end of the slab first. This has the vibe of allocating the leaves of the op tree first, so hopefully the layout ends up cache-friendly. Bonus: it also means there's no need to store the size of the slab (see below on why slabs vary in size), because Perl can just follow pointers to find the last op. 🧚

Now, it might seem possible to eliminate slab reference counts altogether — by having all ops implicitly attached to PL_compcv when they're allocated, and freed when the CV is freed. That would also let op_free skip FreeOp entirely, making op freeing faster. But nope, that doesn't work in those cases where ops need to survive beyond their CVs, like re-evals. Can't have it all.

The CV also has to hold a reference count on the slab. Sometimes the very first op created is immediately freed. If the reference count of the slab hits 0, then it'll be freed while the CV is still pointing at it. 🫠

CVs use the CVf_SLABBED flag to indicate that the CV has a reference count on the slab. When that flag is set, the slab is reachable via CvSTART when CvROOT isn't set, or by subtracting two pointers (2*sizeof(I32 *)) from CvROOT when it is set. The alternative to this whole "sneak the slab into CvSTART during compilation" bit would be to enlarge the xpvcv struct by another pointer. But that would make all CVs bigger, even though slab-based op freeing is typically only a win for programs that make significant use of string eval. So, y'know. Trade-offs. 💅

When the CVf_SLABBED flag is set, the CV takes responsibility for freeing the slab. If CvROOT isn't set when the CV is freed or undeffed, it's assumed a compilation error has happened, so the op slab gets traversed and all the ops get freed. 🎀

Under normal circumstances, the CV forgets about its slab (decrementing the reference count) once the root is attached. That way, the slab reference counting that happens when ops get freed handles freeing the slab. In some cases though, the CV is explicitly told to forget about the slab (cv_forget_slab) — precisely so that the ops can survive after the CV is done away with. 👑

Forgetting the slab when the root is attached isn't strictly necessary, but it avoids potential problems with CvROOT being written over. There's code all over the place, in core and on CPAN, doing things with CvROOT, so forgetting the slab keeps things more robust and dodges those potential problems. 🌈

Since the CV takes ownership of its slab when flagged, that flag is never copied when a CV is cloned — because one CV could free a slab that another CV still points to, since forced freeing of ops ignores the reference count (though it does assert that it looks right). ⭐

To avoid slab fragmentation, freed ops get marked as freed and attached to the slab's freed chain (an idea stolen from DBM::Deep, iconic honestly). Those freed ops get reused when possible. Not reusing freed ops would be simpler, but it would result in significantly higher memory usage for programs with large if (DEBUG) {...} blocks. So we reuse. 🌸

SAVEFREEOP is slightly problematic under this scheme. Sometimes it can cause an op to be freed after its CV. And if the CV has forcibly freed the ops on its slab and the slab itself, then we'd be out here fiddling with a freed slab. Big yikes. Making SAVEFREEOP a no-op doesn't fix it either, because sometimes an op can be savefreed when there's no compilation error, so the op would never get freed. It holds a reference count on the slab, so the whole slab would leak. 💀 So SAVEFREEOP now sets a special flag on the op (->op_savefree). The forced freeing of ops after a compilation error won't free any ops marked that way.

Since so many pieces of code create tiny subroutines consisting of only a few ops, and since a huge slab would be a lot of baggage for those to carry around, the first slab is always very small. And to avoid allocating too many slabs for a single CV, each subsequent slab is twice the size of the previous one. 📈

Smartmatch expects to be able to allocate an op at run time, run it, and then throw it away. For that to work, the op is simply malloced when PL_compcv hasn't been set up. So all slab-allocated ops get marked as such (->op_slabbed), to distinguish them from malloced ops. ✨

---

✨ AUTHORS ✨

Until May 1997, this document was maintained by Jeff Okamoto <okamoto@corp.hp.com>. It is now maintained as part of Perl itself by the Perl 5 Porters <perl5-porters@perl.org>. 🫶

With lots and lots of help and suggestions from Dean Roehrich, Malcolm Beattie, Andreas Koenig, Paul Hudson, Ilya Zakharevich, Paul Marquess, Neil Bowers, Matthew Green, Tim Bunce, Spider Boardman, Ulrich Pfeifer, Stephen McCamant, and Gurusamy Sarathy. Legends, every single one. 👑✨

---

✨ SEE ALSO ✨

• perlapi
• perlintern
• perlxs
• perlembed

---

And that's a wrap on the whole thing bestie!! 🎉💖 This was perl v5.44.0, dated 2026-10-17, and you just read PERLGUTS(1perl) — the whole internals deep-dive, cover to cover, no skimming, no cap. Go forth and be insufferable about op slabs at parties. 🦋💅🌸