INTERNALS(1p)
💖 INTERNALS(1p) — PDL::Internals 💖
From the User Contributed Perl Documentation, bestie: this is INTERNALS(1p), aka PDL::Internals. It's giving deep PDL lore, no cap. 🫶
✨ NAME ✨
PDL::Internals — description of some aspects of the current internals
🌸 SYNOPSIS 🌸
# let PDL tell you what it's doing
use PDL;
PDL::Core::set_debugging(1);
$pa = sequence(6, 3, 2);
$pb = $pa->slice('1:3');
$pc = $pb->matmult($pb);
$pd = $pc->dsumover;
print "pb=$pb\npc=$pc\npd=$pd";💅 DESCRIPTION 💅
🧚 Intro 🧚
Okay so this document explains various aspects of the current implementation of PDL. If you just wanna use PDL for something, you definitely do not need to read this. Even if you wanna interface your C routines to PDL or create new PDL::PP functions, you do not need to read this man page (tho it may be informative). This document is primarily intended for people interested in debugging or changing the internals of PDL. To read this, a good understanding of the C language and programming and data structures in general is required, as well as some Perl understanding. If you read through this document and understand all of it and are able to point what any part of this document refers to in the PDL core sources and additionally struggle to understand PDL::PP, you will be awarded the title "PDL Guru". 👑
⚠️ Warning: If it seems that this document has gotten out of date, please inform the PDL porters email list (pdl-devel@lists.sourceforge.net). This may well happen. 😭
🦋 ndarrays 🦋
The pdl data object is generally an opaque scalar reference into a pdl structure in memory. Alternatively, it may be a hash reference with the "PDL" field containing the scalar reference (this makes overloading ndarrays easy, see PDL::Objects). You can easily find out at the Perl level which type of ndarray you are dealing with. The example code below demonstrates how to do it:
# check if this an ndarray
die "not an ndarray" unless UNIVERSAL::isa($pdl, 'PDL');
# is it a scalar ref or a hash ref?
if (UNIVERSAL::isa($pdl, "HASH")) {
die "not a valid PDL" unless exists $pdl->{PDL} &&
UNIVERSAL::isa($pdl->{PDL},'PDL');
print "This is a hash reference,",
" the PDL field contains the scalar ref\n";
} else {
print "This is a scalar ref that points to address $$pdl in memory\n";
}The scalar reference points to the numeric address of a C structure of type pdl which is defined in pdl.h. The mapping between the object at the Perl level and the C structure containing the actual data and structural that makes up an ndarray is done by the PDL typemap. The functions used in the PDL typemap are defined pretty much at the top of the file pdlcore.h. So what does the structure look like:
struct pdl {
unsigned long magicno; /* Always stores PDL_MAGICNO as a sanity check */
/* This is first so most pointer accesses to wrong type are caught */
int state; /* What's in this pdl */
pdl_trans *trans_parent; /* Opaque pointer to internals of transformation from
parent */
pdl_vaffine *vafftrans;
void* sv; /* (optional) pointer back to original sv.
ALWAYS check for non-null before use.
We cannot inc refcnt on this one or we'd
never get destroyed */
void *datasv; /* Pointer to SV containing data. Refcnt inced */
void *data; /* Null: no data alloced for this one */
PDL_Indx nvals; /* How many values allocated */
int datatype;
PDL_Indx *dims; /* Array of data dimensions */
PDL_Indx *dimincs; /* Array of data default increments */
short ndims; /* Number of data dimensions */
unsigned char *broadcastids; /* Starting index of the broadcast index set n */
unsigned char nbroadcastids;
pdl_trans_children trans_children;
PDL_Indx def_dims[PDL_NDIMS]; /* Preallocated space for efficiency */
PDL_Indx def_dimincs[PDL_NDIMS]; /* Preallocated space for efficiency */
unsigned char def_broadcastids[PDL_NBROADCASTIDS];
struct pdl_magic *magic;
void *hdrsv; /* "header", settable from outside */
PDL_Value value; /* to store at least one value */
};This is quite a structure for just storing some data in — what is going on? 🤔
💾 Data storage 💾
We are going to start with some of the simpler members: first of all, there are the members (as of 2.078)
void *datasv;
PDL_Value value; /* to store at least one value */which are a pointer to a Perl SV structure (SV *), and a union value of all possible single PDL values. If the ndarray's whole data will fit in the value, the datasv will not be used except by get_dataref in PDL::Core for temporary use by Perl code. It will then be destroyed by the call to upd_data in PDL::Core method unless that is passed a true value to keep the datasv around (largely used by the memory-mapping implementation).
Otherwise, the SV is expected to be representing a string, in which the data of the ndarray is stored in a tightly packed form. This pointer counts as a reference to the SV so the reference count has been incremented when the SV * was placed here (this reference count business has to do with Perl's garbage collection mechanism — don't worry if this doesn't mean much to you). This pointer is allowed to have the value NULL which means that there is no actual Perl SV for this data, as alluded above. Note the use of an SV* was purely for convenience, it allows easy transformation of packed data from files into ndarrays. Other implementations are not excluded.
The actual pointer to data is stored in the member
void *data;which contains a pointer to a memory area with space for
PDL_Indx nvals;data items of the data type of this ndarray. PDL_Indx is either 'long' or 'long long' depending on whether your perl is 64bit or not.
The data type of the data is stored in the variable
int datatype;the values for this member are given in the enum pdl_datatypes (see pdl.h). Currently we have byte, short, unsigned short, long, index (either long or long long), long long, float and double (plus complex equivalents) types, see also PDL::Types.
📏 Dimensions 📏
The number of dimensions in the ndarray is given by the member
int ndims;which shows how many entries there are in the arrays
PDL_Indx *dims;
PDL_Indx *dimincs;These arrays are intimately related: dims gives the sizes of the dimensions and dimincs is always calculated by the code
PDL_Indx inc = 1;
for(i=0; i<it->ndims; i++) {
it->dimincs[i] = inc; inc *= it->dims[i];
}in the routine pdl_resize_defaultincs in pdlapi.c. What this means is that the dimincs can be used to calculate the offset by code like
PDL_Indx offs = 0;
for(i=0; i<it->ndims; i++) {
offs += it->dimincs[i] * index[i];
}but this is not always the right thing to do, at least without checking for certain things first. 😬
🗄️ Default storage 🗄️
Since the vast majority of ndarrays don't have more than 6 dimensions, it is more efficient to have default storage for the dimensions and dimincs inside the PDL struct.
PDL_Indx def_dims[PDL_NDIMS];
PDL_Indx def_dimincs[PDL_NDIMS];The dims and dimincs may be set to point to the beginning of these arrays if ndims is smaller than or equal to the compile-time constant PDL_NDIMS. This is important to note when freeing an ndarray struct. The same applies for the broadcastids:
unsigned char def_broadcastids[PDL_NBROADCASTIDS];🔮 Magic 🔮
It is possible to attach magic to ndarrays, much like Perl's own magic mechanism. If the member pointer
struct pdl_magic *magic;is nonzero, the PDL has some magic attached to it. The implementation of magic can be gleaned from the file pdlmagic.c in the distribution. ✨
🚦 State 🚦
One of the first members of the structure is
int state;The possible flags and their meanings are given in pdl.h. These are mainly used to implement the lazy evaluation mechanism and keep track of ndarrays in these operations.
🌀 Transformations and virtual affine transformations 🌀
As you should already know, ndarrays often carry information about where they come from. For example, the code
$y = $x->slice("2:5");
$y .= 1;will alter $x. So $y and $x know that they are connected via a "slice"-transformation. This information is stored in the members
pdl_trans *trans_parent;
pdl_vaffine *vafftrans;Both $x (the parent) and $y (the child) store this information about the transformation in appropriate slots of the pdl structure.
pdl_trans and pdl_vaffine are structures that we will look at in more detail below.
🧬 The Perl SVs 🧬
When ndarrays are referred to through Perl SVs, we store an additional reference to it in the member
void* sv;in order to be able to return a reference to the user when they want to inspect the transformation structure on the Perl side.
Also, we store an opaque
void *hdrsv;which is just for use by the user to hook up arbitrary data with this sv. This one is generally manipulated through sethdr and gethdr calls.
✂️ Smart references and transformations: slicing and dicing ✂️
Smart references and most other fundamental functions operating on ndarrays are implemented via transformations (as mentioned above) which are represented by the type pdl_trans in PDL.
A transformation links input and output ndarrays and contains all the infrastructure that defines how:
- output ndarrays are obtained from input ndarrays;
- changes in smart-linked output ndarrays (e.g. the child of a sliced parent ndarray) are flowed back to the input ndarray in transformations where this is supported (the most often used example being
slicehere); - datatype and size of output ndarrays that need to be created are obtained.
okay so real talk bestie — when you run a PDL function over a group of ndarrays, what you get is a transformation of whatever type you asked for, and that transformation links up all the input and output args (well, at least the ones that are ndarrays, obviously 🫶).
Now here's the tea ☕: in the PDL functions that do the whole data-flow-between-inputs-and-outputs thing (think slice, index), that transformation links the parent (input) and child (output) ndarrays permanently, like, until either somebody explicitly breaks the link on purpose (that's sever at the Perl level, dramatic queen 💅) or every single parent and child has been destroyed. And in those cases the transformation is lazy-evaluated — meaning it only actually runs when you go and touch the ndarray values. Not a second before. main character energy ✨
In the non-flowing functions though — like addition (+) and inner products (inner) — the transformation still gets installed the exact same way as in the flowing ones, but then it's immediately executed and destroyed (link between input and output args: broken 💀) before the function even returns. Gone. So fast. Iconic.
okay but this next bit is important, no cap 🎀 — that super-close link between the input and output args of a flowing function (hi slice 👋) means that ndarray objects linked this way have to be kept alive even after they've wandered out of scope from Perl's point of view. Like, watch:
$x = zeroes(20);
$y = $x->slice('2:4');
undef $x; # last reference to $x is now destroyedSo yeah, $x should be destroyed now per Perl's rules, but ngl the underlying pdl structure can literally only be freed when $y also goes out of scope, because it's still internally referencing some of $x's data. it's giving "you can't get rid of me that easily" 🦋
This example is here to show you that this whole dataflow paradigm between PDL objects demands a special destruction algorithm that actually takes the links between ndarrays into account and couples their lifespans together. That non-trivial algorithm lives in the function pdl_destroy in pdlapi.c. And honestly? Most of the code in pdlapi.c is just about making absolutely sure that ndarrays (pdl *s) get created, updated and freed at exactly the right times, depending on how they're interacting with other ndarrays through PDL transformations (remember, pdl_trans 👑).
🌸 Accessing children and parents of an ndarray 🌸
When ndarrays get dynamically linked via transformations like we just said, the input ndarrays are called parents and the output ndarrays are called children. That's the whole vibe.
A cute little example of processing the children of an ndarray is the badflag method over in PDL::Bad (and fun fact, before 2.079 it only operated on the children — though now it propagates to parents too 💖).
So picture this situation:
pdl> $x = rvals(7,7,{Centre=>[3,4]});
pdl> $y = $x->slice('2:4,3:5');
pdl> ? vars
PDL variables in package main::
Name Type Dimension Flow State Mem
----------------------------------------------------------------
$x Double D [7,7] P 0.38Kb
$y Double D [3,3] -C 0.00KbNow let's say I suddenly decide $x should be flagged as possibly containing bad values, by doing
pdl> $x->badflag(1)Then I want the state of $y — its child — to change too, because it's gonna either share or inherit some of $x's data and therefore be bad as well 😭, so I get a 'B' showing up in the State field:
pdl> ? vars
PDL variables in package main::
Name Type Dimension Flow State Mem
----------------------------------------------------------------
$x Double D [7,7] PB 0.38Kb
$y Double D [3,3] -CB 0.00KbAnd this little bit of magic is performed by the propagate_badflag function, which is in pdlapi.c. Here's how it works: given an ndarray (pdl *it), the routine loops through each pdl_trans structure — and access to that structure is provided by the PDL_CHILDLOOP_THISCHILD macro. The children of the ndarray are stored in the pdls array, after the parents, hence the loop going from i = ...nparents to i = ...npdls - 1. Once we've got the pointer to the child ndarray, we can do whatever we want to it — in this case we change the value of the state variable, but the details are, respectfully, unimportant 🙈. What is important is that we call propagate_badflag on this ndarray, so we loop through its children too. That recursion is what makes sure we reach every single one of the offspring of a particular ndarray. Family reunion fr fr 🌈
Access to parents is basically the same deal, just with the for loop swapped out for:
for( i = 0;
i < trans->vtable->nparents;
i++ ) {
/* do stuff with parent #i: trans->pdls[i] */
}🎀 What's in a transformation (pdl_trans) 🎀
Every transformation is implemented as a structure. All of them. No exceptions:
struct pdl_trans {
int magicno; /* to detect memory overwrites */
short flags; /* state of the trans */
pdl_transvtable *vtable; /* the all important vtable */
int __datatype; /* the type of the transformation */
void *params; /* Opaque pointer to "compiled representation" of transformation */
pdl *pdls[]; /* The pdls involved in the transformation */
};The params member is an opaque pointer — typically to a C struct that holds the "compiled representation" (generated by PDL::PP), and it's the way info like OtherPars and friends gets communicated from the invoking code over to the redodims function. Basically a closure, if you're thinking in Perl/LISP terms 💅. And this is necessary because redodims gets called by a PDL-internal function, so it has to have a fixed parameter list. No wiggle room.
The transformation identifies all the pdls involved in the trans:
pdl *pdls[];This is a C99 "incomplete array type", and the reason it works is that it's sitting at the end of the struct — PDL allocates the right amount of memory based on the npdls member of the vtable. smart, actually ✨
The trans also records the state
short flags;and the datatype
int __datatype;of the trans (and everything has to be converted to that datatype unless it's explicitly typed — PDL functions created with PDL::PP make sure those conversions happen as needed, don't worry bestie 💜). But the most important thing is the pointer to the vtable (virtual table) that holds the actual functionality:
pdl_transvtable *vtable;And the vtable structure itself looks something like this (slightly simplified from pdl.h for clarity, so don't come for me):
typedef struct pdl_transvtable {
int flags;
int nparents; /* number of parent pdls (input) */
int npdls; /* number of child pdls (output) */
char *per_pdl_flags; /* optimization flags */
pdl_error (*redodims)(pdl_trans *tr); /* figure out dims of children */
pdl_error (*readdata)(pdl_trans *tr); /* flow parents to children */
pdl_error (*writebackdata)(pdl_trans *tr); /* flow backwards */
pdl_error (*freetrans)(pdl_trans *tr, char);
int structsize;
char *name; /* the function's name */
} pdl_transvtable;The transformation and vtable code is hardly ever written by hand — it's generated by PDL::PP from concise descriptions. Thank god honestly 😌
okay so certain types of transformations can be optimized very efficiently, to the point where you don't even need explicit readdata and writebackdata methods. Those transformations are called pdl_vaffine. Most of the dimension-manipulating functions (like slice, xchg) are in this club 👑
The basic trick? The parent and the child of such a transformation work on the same (shared) block of data, they just choose to interpret it differently — by using different dims, dimincs and offs on that same data, compare the pdl structure above if you want the receipts. So every operation on an ndarray that shares data with another one this way is automatically flowed from child to parent and back. I mean... they're literally reading and writing the same block of memory, so 💀. ⚠️ Heads up though: this is currently not Perl thread safe — which is no big loss since the whole PDL core isn't reentrant anyway. Moving on 🦋
💖 Callback functions 💖
redodims
Works out the dimensions of ndarrays that need to be created, and it gets called from inside the API function that should be called to make sure the dimensions of an ndarray are accessible (pdlapi.c):
pdl_error pdl_make_physdims(pdl *it)readdata and writebackdata
These two are responsible for the actual computations — of the child data from the parents, and of the parent data from the children, respectively (that's the dataflow aspect, babes ✨). readdata populates the children from the parents, and writebackdata implements updating the parent(s) from the child(ren) if dataflow is part of that transformation. The PDL core makes sure these get called as needed whenever ndarray data is accessed (lazy-evaluation, we love her 💅). The general API function that ensures an ndarray is up-to-date is
pdl_error pdl_make_physvaffine(pdl *it)and you should call that before accessing ndarray data from XS/C (go look at Core.xs for some examples, they're right there 🫶).
freetrans
Frees the dynamically allocated memory associated with the trans, as needed. If redodims has previously been called, it'll free any vaffine-associated memory. And if the destroy parameter is true, it'll also free any bespoke params-connected memory — which will not be the case if it's called before doing another redodims. Again: functions built with PDL::PP make sure that freeing via these callbacks happens at the right times. It's handled. Trust. 😌
🌈 Signatures: broadcasting over elementary operations 🌈
Most of the functionality of PDL broadcasting (that's the automatic iteration of elementary operations over multi-dim ndarrays, iykyk) is implemented in the file pdlbroadcast.c.
The PDL::PP generated functions (and in particular the readdata and writebackdata callbacks) use this infrastructure to make sure that the fundamental operation implemented by the trans gets performed in agreement with PDL's broadcasting semantics. No cap 🎀
⭐ Defining new PDL functions -- Glue code generation ⭐
Please, see PDL::PP and the examples in the PDL distribution. The implementation and syntax are currently far from perfect, but honestly? It does a good job! 👑✨
💜 The Core struct 💜
As we talked about over in PDL::API, PDL uses a pointer to a structure so that PDL modules can access its core routines. The definition of this structure (the Core struct) lives in pdlcore.h (created by pdlcore.h in Basic/Core) and looks something like:
/* Structure to hold pointers core PDL routines so as to be used by
* many modules
*/
struct Core {
I32 Version;
pdl* (*SvPDLV) ( SV* );
void (*SetSV_PDL) ( SV *sv, pdl *it );
pdl* (*pdlnew) ( );
pdl* (*tmp) ( );
pdl* (*create) (int type);
pdl_error (*destroy) (pdl *it);
...
}
typedef struct Core Core;The very first field of the structure (Version) is there to make sure modules stay consistent at run time — and this following code gets placed in the BOOT section of the generated xs code:
if (PDL->Version != PDL_CORE_VERSION)
Perl_croak(aTHX_ "Foo needs to be recompiled against the newly installed PDL");so if you're out here adding a brand new field to the Core struct 🫶 bestie, there's a whole lil checklist, and you will be doing all of it:
- yap about it on the pdl porters email list (
pdl-devel@lists.sourceforge.net) 💌 and use the techniques inPDL::FAQ4.11 — yes, that FAQ section, go read it, it's giving homework but make it worth it. - bump the value of the
PDL_CORE_VERSIONC macro by 1 ⬆️ (that's the macro that populates theVersionfield, over inpdlcore.h). one. single. number. do not skip this one, fr fr. - add documentation for it — like, over in
PDL::API— if it's a "useful" function for external module writers ✨ and also make sure the code itself is as well documented as the rest of PDL ;) (yes the wink is load-bearing, it's from the original, we keep it 💅).
😭 BUGS 😭
ngl, this description is far from perfect. if you need more details or something is still unclear, please ask on the pdl-devel mailing list (pdl-devel@lists.sourceforge.net) 🌸 — the people there are lovely and they want to help, no cap.
👑 AUTHOR 👑
Copyright(C) 1997 Tuomas J. Lukka (lukka@fas.harvard.edu), 2000 Doug Burke (djburke@cpan.org), 2002 Christian Soeller & Doug Burke, 2013 Chris Marshall.
and that's the whole page bestie 💖 this one came to you courtesy of perl v5.38.2, last touched 2024-04-10 — go forth and add your Core struct fields responsibly, mwah 🎀
📦 where this page comes from
- ubuntu noble: pdl 1:2.085-1ubuntu1 · Artistic-1.0-Perl OR GPL-1.0-or-later
- 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms