DATAFLOW(1p)

📦
version 2.085 · 3 versions
, in ubuntu noble

💖 PDL::Dataflow 💖

okay bestie, so this is the tea on how PDL's dataflow works and the whole philosophy behind it, from the PDL::Dataflow page in the User Contributed Perl Documentation. manual name's right there in the title, section 1p, and it's giving "this will change how you think about arrays" energy. let's go 🦋

✨ SYNOPSIS ✨

pdl> $x = zeroes(10);
pdl> $y = $x->slice("2:4:2");
pdl> $y ++;
pdl> print $x;
[0 0 1 0 1 0 0 0 0 0]

like. look at that. $y did a little something and $x felt it. iconic behavior honestly.

🌸 DESCRIPTION 🌸

so as of 2.079, this whole page is a description of the current implementation, plus some design thoughts from the original author, Tuomas Lukka. we love a founder who documents their own vibes.

Two-directional dataflow (the thing that implements ->slice() and friends) is fully functional, as shown up in the SYNOPSIS. no notes, it just works.

One-way flow? it's implemented, but with restrictions. she's on a leash, basically. 💅

🎀 TWO-WAY 🎀

literally just about any function that returns some subset of the values in some ndarray will make a binding. $y has become, effectively, a window to some sub-elements of $x. you can also define your own routines that do different types of subsets, go off.

but listen — if you do not want $y to be a window to $x, you gotta do this:

$y = $x->slice("some parts")->sever;

the sever destroys the slice transform, which turns off all dataflow between the two ndarrays. cutting the cord, no cap. ✂️

🧚 Type conversions 🧚

this works, thanks to a two-way flowing transform that implements type-conversions — particularly for supplied outputs of the "wrong" type for the given transform:

pdl> $a_bad = pdl double, '[1 BAD 3]';
pdl> $b_float = zeroes float, 3;
pdl> $a_bad->assgn($b_float); # could be written as $b_float .= $a_bad
pdl> p $b_float->badflag;
1
pdl> p $b_float;
[1 BAD 3]

the BAD flag survived the whole conversion journey. she's resilient. 🫶

💅 ONE-WAY 💅

you need to explicitly turn on one-way dataflow on an ndarray to activate it for non-flowing operations. so, like:

pdl> $x = pdl 2,3,4;
pdl> $x->doflow;
pdl> $y = $x * 2;
pdl> print $y;
[4 6 8]
pdl> $x->set(0,5);
pdl> print $y;
[10 6 8]

okay so it is not possible to turn on backwards dataflow (the kind used by slice-type operations), because there's no general way for PDL (or maths, honestly, maths itself) to know how to reverse most operations. consider $z = $x * $y, then adding one to $z. like... reverse that, bestie. you can't. 😭

now consider the following code:

$u = sequence(3,3); $u->doflow;
$v = ones(3,3); $v->doflow;
$w = $u + $v; $w->doflow; # must turn on for each
$y = $w + 1; $y->doflow;
$x = $w->diagonal(0,1);
$x += 50;
$z = $w + 2;

what do $y and $z contain now?? 👀

pdl> p $y
[
 [52  3  4]
 [ 5 56  7]
 [ 8  9 60]
]
pdl> p $z
[
 [53  4  5]
 [ 6 57  8]
 [ 9 10 61]
]

and what about when $u is changed and a recalculation is triggered? here's where it gets messy. a problem arises, in that PDL currently (as of 2.079) disallows (see pdlapi.c), for normal transforms, output ndarrays with flow, or output ndarrays with any parent with dataflow. so $u++ throws an exception. 💀

but it is currently possible to use set, which is a sort of micro-transform that calls (in the C API) PDL.set to mutate the data, then PDL.changed to trigger flow updates:

pdl> $u->set(1,1,90)
pdl> p $y
[
 [ 2  3  4]
 [ 5 92  7]
 [ 8  9 10]
]

you'll notice that while the setting of 1,1 (the middle) of $u updated $y, the changes to $y that resulted from adding 50 to the diagonal (via $x, and two-way flow) got lost. that's one-way flow for you. it giveth and it... doesn't giveth back. 🙈

🌈 LAZY EVALUATION 🌈

in a one-way flow context like the above, with:

pdl> $y = $x * 2;

nothing will have been calculated at this point. not even the memory for the contents of $y has been allocated. it's just vibes and promises at this stage.

only the command:

pdl> print $y

will actually cause $y to be calculated. this is important to bear in mind when doing performance measurements and benchmarks, and also when tracking errors. do not come for her before she's done the math. ⭐

there's an explanation for this behaviour: it may save cycles, but more importantly, imagine the following:

pdl> $x = pdl 2,3,4; $x->doflow;
pdl> $y = pdl 5,6,7; $y->doflow;
pdl> $c = $x + $y;
pdl> $x->setdims([4]);
pdl> $y->setdims([4]);
pdl> print $c;

now, if $c were evaluated between the two resizes, an error condition of incompatible sizes would occur. yikes. 🚩

what happens in the current version is that resizing $x raises a flag in $c: PDL_PARENTDIMSCHANGED, and $y just raises the same flag again. when $c is next evaluated, the flags are checked and it's found that a recalculation is needed. she checked the receipts. ✅

of course, lazy evaluation can sometimes make debugging more painful, because errors may occur somewhere where you'd not expect them. it's giving "the bug is not where you think it is". 🕵️♀️

👑 FAMILIES 👑

this is one of the more intricate concepts of dataflow. in order to make dataflow work like you'd expect, a rather strange concept must be introduced: families.

let us make a diagram of the one-way flow example — it uses a hypergraph because the transforms (with +) are connectors between ndarrays (with *):

       u*   *v
         \ /
          +(plus)
          |
   1*     *w
     \   /|\
      \ / | \
 (plus)+  |  +(diagonal)
       |  |  |
      y*  |  *x
          |
          | *1
          |/
          +(plus)
          |
         z*

this is what PDL actually has in memory after the first three lines. when $x is changed, $w changes due to diagonal being a two-way operation.

if you want flow from $w, you opt in using $w->doflow (as shown in this scenario). if you didn't, then don't enable it. simple as.

if you have it but want to stop it, call $ndarray->sever. that will destroy the ndarray's trans_parent (here, a node marked with +), and as you can visually tell, will stop changes flowing thereafter. if you want to leave the flow operating but get a copy of the ndarray at that point, use $ndarray->copy — it will have the same data at that moment, but have no flow relationships. screenshot, not a live stream. 📸

🔥 EVENTS 🔥

there's the start of a mechanism to bind events onto changed data, intended to allow this to work:

pdl> $x = pdl 2,3,4
pdl> $y = $x + 1;
pdl> $c = $y * 2;
pdl> $c->bind( sub { print "A now: $x, C now: $c\n" } )
pdl> PDL::dowhenidle();
A now: [2,3,4], C now: [6 8 10]
pdl> $x->set(0,1);
pdl> $x->set(1,1);
pdl> PDL::dowhenidle();
A now: [1,1,4], C now: [4 4 10]

this hooks into PDL's magic, which resembles Perl's, but does not currently operate. it's giving "concept demo, not shipping feature". 🎀

there would be many kinds of uses for this feature — self-updating charts, for instance. it is not yet fully clear whether it would be most useful to queue up changes (useful for doing asynchronously, e.g. when idle), or to activate things immediately.

in the 2022 era of both GPUs and multiple cores, it is a pity that Perl's dominant model remains single-threaded on CPU, but PDL can use multi-cores for CPU processing (albeit controlled in a single-threaded style) — see PDL::ParallelCPU. it is planned that PDL will gain the ability to use GPUs, and there might be a way to hook that up, albeit probably with an event loop to "subscribe" to GPU events. we're getting there, bestie. 🚀

🦋 TRANSFORMATIONS 🦋

PDL implements nearly everything (except for XS oddities like set) using transforms which connect ndarrays. this includes data transformations like addition, "slicing" to access/operate on subsets, and data-type conversions (which have two-way dataflow, see "Type conversions").

this does not currently include a resizing transformation, and setdims mutates its input. this is intended to change. noted, on the roadmap, manifesting it. ✨

💜 AUTHOR 💜

Copyright(C) 1997 Tuomas J. Lukka (lukka@fas.harvard.edu). Same terms as the rest of PDL.

and that's the whole page, babes 💖 this was DATAFLOW(1p) from perl v5.38.2, dated 2024-04-10 — go forth and let your ndarrays flow, but maybe keep an eye on that lazy evaluation. periodt. 👑🌸

📦 where this page comes from