Data::DPath(3pm)

📦
version 0.60 · 5 versions
, in ubuntu resolute, ubuntu stonking

🌸 NAME 🌸

Data::DPath — bestie, it's DPath. And no, it is not XPath. That's literally the whole tagline: DPath is not XPath! This is the manual page for Data::DPath(3pm) and we're gonna get through it together, okay? 🫶

✨ SYNOPSIS ✨

okay so first thing, you use it and then you go absolutely feral:

 use Data::DPath 'dpath';
 my $data  = {
              AAA  => { BBB => { CCC  => [ qw/ XXX YYY ZZZ / ] },
                        RRR => { CCC  => [ qw/ RR1 RR2 RR3 / ] },
                        DDD => { EEE  => [ qw/ uuu vvv www / ] },
                      },
             };

 # Perl 5.8 style
 my @resultlist = dpath('/AAA/*/CCC')->match($data); # ( ['XXX', 'YYY', 'ZZZ'], [ 'RR1', 'RR2', 'RR3' ] )

 # Perl 5.10 style using overloaded smartmatch operator
 my $resultlist = $data ~~ dpath '/AAA/*/CCC';        # [ ['XXX', 'YYY', 'ZZZ'], [ 'RR1', 'RR2', 'RR3' ] ]
💀 heads up: the match() function hands you back an array, but the overloaded ~~ operator hands you back an array reference. That's just a limitation of overloading, no cap. Don't come for me.

Here's a whole pile of other example paths, straight out of t/data_dpath.t (and no, they don't necessarily all fit the data structure up there, they're just the vibes):

 $data ~~ dpath '/AAA/*/CCC'
 $data ~~ dpath '/AAA/BBB/CCC/../..'    # parents  (..)
 $data ~~ dpath '//AAA'                 # anywhere (//)
 $data ~~ dpath '//AAA/*'               # anywhere + anystep
 $data ~~ dpath '//AAA/*[size == 3]'    # filter by arrays/hash size
 $data ~~ dpath '//AAA/*[size != 3]'    # filter by arrays/hash size
 $data ~~ dpath '/"EE/E"/CCC'           # quote strange keys
 $data ~~ dpath '/AAA/BBB/CCC/*[1]'     # filter by array index
 $data ~~ dpath '/AAA/BBB/CCC/*[ idx == 1 ]' # same, filter by array index
 $data ~~ dpath '//AAA/BBB/*[key eq "CCC"]'  # filter by exact keys
 $data ~~ dpath '//AAA/*[ key =~ /CC/ ]'     # filter by regex matching keys
 $data ~~ dpath '//CCC/*[ value eq "RR2" ]'  # filter by values of hashes

The full details live in t/data_dpath.t, go read it, it's giving documentation energy. 👑

And if you want references pointing into the $data data structure instead of copies, that's dpathr:

 $data ~~ dpathr '//AAA/BBB/*'
 # etc.

And if you want iterators so you can do incremental searches, that's dpathi:

 my $benchmarks_iter = dpathi($data)->isearch("//Benchmark");
 while ($benchmarks_iter->isnt_exhausted)
 {
     my $benchmark = $benchmarks_iter->value;
     my $ancestors_iter = $benchmark->isearch ("/::ancestor");
     while ($ancestors_iter->isnt_exhausted)
     {
         my $ancestor = $ancestors_iter->value;
         print Dumper( $ancestor->deref );
     }
 }

So what this does is find every element anywhere behind a key Benchmark, and for each one it finds, it prints all its ancestors. Respectively. 😌 And yes there's a whole chapter about this below called Iterator style, keep scrolling.

💖 ABOUT 💖

With this module you can point at specific spots in a data structure by describing a "path" to it — using hash keys, array indexes, or some wildcard-ish steps. It's inspired by XPath but it's very much its own thing. Not the same. Different. Periodt.

🦋 Why not XPath? 🦋

XPath is for XML. DPath is for data structures, and it's got a way stronger Perl focus. iykyk.

I know, I know — XML documents are data structures, but they're a special kind, and here's why that matters:

  • Elements in XML always have an order. Hash keys in Perl? Not like that at all.
  • XML element names on the same level can be repeated. Hashes could never.
  • XML element names are way more limited than your average unhinged strange hash key.
  • XML elements can have attributes, and XPath can address those. Perl data structures don't need that. But flip it around: Perl data structures can contain blessed elements, and DPath can address that. Slay.
  • XML has namespaces. Data structures simply do not.
  • Arrays starting at index 1, like XPath does? That would be so confusing to read for data structures. No thank you.
  • DPath lets you use filter expressions that are literally just Perl expressions — not some whole separate sub-language like XPath has. Chef's kiss. 💅

🌈 Comparison with Data::Path 🌈

There's a similar approach on CPAN called Data::Path. Here's the comparison breakdown between Data::Path and Data::DPath:

⚠️ Warning: this is an alpha grade comparison, not yet fully verified, only evaluated by reading the source. The speed comparison is not really benchmarked. Read it with that energy, bestie.
  • real XPath syntax
    • Data::Path: no
    • Data::DPath: no
  • allow strange, non-xml but perl-like hash keys
    • Data::Path: YES — although limited, see next
    • Data::DPath: YES
  • allows special chars of own path syntax in hash keys ("/[]|*.")
    • Data::Path: no
    • Data::DPath: YES — you can quote everything
  • call subs in data structure, like /method()
    • Data::Path: YES
    • Data::DPath: no
  • callbacks on not found keys
    • Data::Path: YES
    • Data::DPath: no
  • element "//" for "ANYWHERE" (//foo/bar)
    • Data::Path: no
    • Data::DPath: YES
  • element "." for "NOSTEP" or "actual position" (/.[filter expr])
    • Data::Path: no
    • Data::DPath: YES
  • element ".." for "PARENT" (//foo/..)
    • Data::Path: no
    • Data::DPath: YES
  • element "::ancestor" for "ANCESTOR" (//foo/::ancestor)
    • Data::Path: no
    • Data::DPath: YES
  • element "::ancestor-or-self"
    • Data::Path: no
    • Data::DPath: YES
  • element "*" for "ANYSTEP" or "all subelements" (/foo/*)
    • Data::Path: no
    • Data::DPath: YES
  • array access like /foo[4]
    • Data::Path: YES — although limited
    • Data::DPath: YES — including negative indexes and whitespace awareness
  • complex filter expressions like /foo[size == 3] or /.[isa("Foo::Bar")]
    • Data::Path: no
    • Data::DPath: YES — full Perl expressions plus sugar functions
  • works with blessed subelements
    • Data::Path: YES
    • Data::DPath: YES
  • arrays start with index 0 (in contrast to 1 as in XPath)
    • Data::Path: YES
    • Data::DPath: YES
  • array semantics is a bit different
    • Data::Path: /foo[2]
    • Data::DPath: /foo/*[2]
  • handling of not matching paths
    • Data::Path: croak — but can be overwritten as callback
    • Data::DPath: RETURN EMPTY
  • usage sugar
    • Data::Path: none
    • Data::DPath: overloaded '~~' operator
  • Speed
    • Data::Path: FAST — raw Perl, considered fast
    • Data::DPath: quite fast — probably comparable speed with expressions that Data::Path handles, but slower on fuzzy paths, eg. with many "//" in it
  • Perl Versions
    • Data::Path: 5.6+
    • Data::DPath: 5.8+
  • Install chance (http://deps.cpantesters.org)
    • Data::Path: 100%
    • Data::DPath: 90%

⭐ Summary

Generally, Data::Path is for simpler use cases but doesn't suffer from surrounding meta problems: it has no dependencies, it's fast, and it works on practically every Perl version.

Whereas Data::DPath gives you more XPath-alike features — but it isn't quite as fast, and it's got more dependencies. Trade-offs, bestie. 💜

🚨 Security warning 🚨

Watch out!! This module "eval"s parts of provided dpaths — in particular, the filter expressions. Do not use it if you don't trust your paths. Seriously. 🙈

Since v0.41 the filter expressions are secured using Safe.pm to only allow basic Perl core ops. That gives you more safety, but it's also significantly slower. 😭

If you want to unrestrict it back to the pre-v0.41 raw "eval" behaviour, you can set $Data::DPath::USE_SAFE to False:

 local $Data::DPath::USE_SAFE;
 # dpath '//CCC//*[ unsecure_perl_expression ]'

Go read Safe.pm to actually understand how secure this is, don't just take my word for it. 🫶

🎀 FUNCTIONS 🎀

🌟 dpath( $path_str )

This is the front end function, meant for everyday use of Data::DPath. It takes a path string and returns a "Data::DPath::Path" object, and on that object you can call the match method with data structures — plus the operator "~~" is overloaded. Iconic.

The function is prototyped to take exactly one argument, so you can omit the parens in many cases. Casual.

See SYNOPSIS up there. ⬆️

🌟 dpathr( $path_str )

Same as "dpath", except it toggles things so the results are references to the matched points in the data structure.

🌟 dpathi( $data )

This one's a whole different approach — the iterator style one.

You hand it the data structure you want to work on, and you get back a current context containing the root element (as if you had searched for the path "/"). And then from there you can do incremental searches with "isearch".

Details are in the chapter Iterator style below. Keep reading bestie, we get there. 💖

🧚 API METHODS 🧚

💫 match( $data, $path )

Returns an array of all values in $data that match the $path.

💫 matchr( $data, $path )

Returns an array ref of all values in $data that match the $path. (Yes, the r is for reference. You're welcome. 🎀)

👑 OPERATOR 👑

💥 ~~

Does a "match" of a dpath against a data structure.

Because of the matching nature of DPath, the operator "~~" should make your code more readable. Main character energy for your source files, honestly. ✨

✨ THE DPATH LANGUAGE ✨

📖 Synopsis

        /AAA/BBB/CCC
        /AAA/*/CCC
        //CCC/*
        //CCC/*[2]
        //CCC/*[size == 3]
        //CCC/*[size != 3]
        /"EE/E"/CCC
        /AAA/BBB/CCC/*[1]
        /AAA/BBB/CCC/*[ idx == 1 ]
        //AAA/BBB/*[key eq "CCC"]
        //AAA/*[ key =~ /CC/ ]
        //CCC/*[value eq "RR2"]
        //.[ size == 4 ]
        /.[ isa("Funky::Stuff") ]/.[ size == 5 ]/.[ reftype eq "ARRAY" ]

🌸 Modeled on XPath 🌸

Okay so the basic idea is literally XPath energy: you define a way through a datastructure and allow some funky ways to describe fuzzy ways. The syntax is roughly looking like XPath but in fact have not much more in common. 💅

💬 Some wording

  • I call the whole path a, well, path.
  • It consists of single (path) steps that are divided by the path separator /.
  • Each step can have a filter appended in brackets [] that narrows down the matching set of results.
  • Additional functions provided inside the filters are called, well, filter functions.
  • Each step has a set of points relative to the set of points before this step, all starting at the root of the data structure.

🦋 Special elements 🦋

  • "//" — Anchors to any hash or array inside the data structure below the currently found points (or the root).

Typically used at the start of a path to anchor the path anywhere instead of only the root node:

//FOO/BAR

but can also happen inside paths to skip middle parts:

/AAA/BBB//FARAWAY

This allows any way between "BBB" and "FARAWAY".

  • "*" — Matches one step of any value relative to the current points (or the root). This step might be any hash key or all values of an array in the step before.
  • ".." — Matches the parent element relative to the current points.
  • "::ancestor" — Matches all ancestors (parent, grandparent, etc.) of the current node.
  • "::ancestor-or-self" — Matches all ancestors (parent, grandparent, etc.) of the current node and the current node itself.
  • "." — A "no step". This keeps passively at the current points, but allows incrementally attaching filters to points or to otherwise hard to reach steps, like the top root element "/". So you can do:
/.[ FILTER ]

or chain filters:

/AAA/BBB/.[ filter1 ]/.[ filter2 ]/.[ filter3 ]

This way you do not need to stuff many filters together into one huge killer expression and can more easily maintain them.

See Filters for more details on filters.

If you need those special elements to be not special but as key names, just quote them:

            /"*"/
            /"*"[ filter ]/
            /"::ancestor"/
            /".."/
            /".."[ filter ]/
            /"."/
            /"."[ filter ]/
            /"//"/
            /"//"[ filter ]/

💅 Difference between "/step[filter]" vs. "/step/.[filter]" vs. "/step/*[filter]" 💅

The filter applies to the matched points of the step to which it is applied, therefore "/part[filter]" is the normal form, but see below how this affects array access.

The "no step" "/." stays on the current step, therefore "/part/.[filter]" should be the same as "/part[filter]".

Lastly, "/part/*[filter]" means: take all the sub elements ("*") below "step" and apply the filter to those. The most common use is to take "all" elements of an array and chose one element via index: "/step/*[4]/". This takes the fifth element of the array inside "step". This is explained in even more depth in the next section.

🍓 Difference between "/affe[2]" vs. "/affe/*[2]" 🍓

Read carefully, bestie. This is different from what you probably expect when you know XPath.

In XPath "/affe[2]" would address an item of all elements named "affe" on this step. This is because in XPath elements with the same name can be repeated, like this:

         <coolanimals>
           <affe>Pavian</affe>
           <affe>Gorilla</affe>
           <affe>Schimpanse</affe>
         </coolanimals>

and "//affe[2]" would get "Schimpanse" (we ignore the fact that in XPath array indexes start with 1, not 0 as in DPath, so we would actually get "Gorilla"; anyway, both are funky fellows).

So what does "/affe[2]" return in DPath? Nothing! It makes no sense, because "affe" is interpreted as a hash key and hash keys can not repeat in Perl data structures.

So what you often want in DPath is to look at the elements below "affe" and takes the third of them, e.g. in such a structure:

        { affe => [
                   'Pavian',
                   'Gorilla',
                   'Schimpanse'
                  ]
        }

the path "/affe/*[2]" would return "Schimpanse".

🔍 Filters 🔍

Filters are conditions in brackets. They apply to all elements that are directly found by the path part to which the filter is appended.

Internally the filter condition is part of a "grep" construct (exception: single integers, they choose array elements). See below.

Examples:

  • "/FOO/*[2]/" — A single integer as filter means choose an element from an array. So the "*" finds all subelements that follow current step "FOO" and the "[2]" reduces them to only the third element (index starts at 0).
  • "/FOO/*[ idx == 2 ]/" — The "*" is a step that matches all elements after "FOO", but with the filter only those elements are chosen that are of index 2. This is actually the same as just "/FOO/*[2]".
  • "/FOO/*[key eq "CCC"]" — In all elements after "FOO" it matches only those elements whose key is "CCC".
  • "/FOO/*[key =~ /CCC/ ]" — In all elements after step "FOO" it matches only those elements whose key matches the regex "/CCC/". It is actually just Perl code inside the filter which works in a grep{}-like context.
  • "//FOO/*[value eq "RR2"]" — Find elements below "FOO" that have the value "RR2".

Combine this with the parent step "..":

  • "//FOO/*[value eq "RR2"]/.." — Find such an element below "FOO" where an element with value "RR2" is contained.
  • "//FOO[size >= 3]" — Find "FOO" elements that are arrays or hashes of size 3 or bigger.

🧚 Filter functions 🧚

The filter condition is internally part of a "grep" over the current subset of values. So you can write any condition like in a grep and also use the variable $_.

Additional filter functions are available that are usually written to use $_ by default. See Data::DPath::Filters for complete list of available filter functions.

Here are some of them:

  • idx — Returns the current index inside array elements.

Please note that the current matching elements might not be in a defined order if resulting from anything else than arrays.

  • size — Returns the size of the current element. If it is an arrayref it returns number of elements, if it's a hashref it returns number of keys, if it's a scalar it returns 1, everything else returns -1.
  • key — Returns the key of the current element if it is a hashref. Else it returns undef.
  • value — Returns the value of the current element. If it is a hashref, return the value. If a scalar, return the scalar. Else return undef.

🔣 Special characters 🔣

There are 4 special characters: the slash "/", paired brackets "[]", the double-quote """ and the backslash "\". They are needed and explained in a logical order.

Path parts are divided by the slash "/".

A path part can be extended by a filter with appending an expression in brackets "[]".

To contain slashes in hash keys, they can be surrounded by double quotes """.

To contain double-quotes in hash keys they can be escaped with backslash "\".

Backslashes in path parts don't need to be escaped, except before escaped quotes (but see below on Backslash handling).

Filters of parts are already sufficiently divided by the brackets "[]". There is no need to handle special characters in them, not even double-quotes. The filter expression just needs to be balanced on the brackets.

So this is the order how to create paths:

  1. backslash double-quotes that are part of the key
  2. put double-quotes around the resulting key
  3. append the filter expression after the key
  4. separate several path parts with slashes

🌈 Backslash handling 🌈

If you know backslash in Perl strings, skip this paragraph, it should be the same.

It is somewhat difficult to create a backslash directly before a quoted double-quote.

Inside the DPath language the typical backslash rules of apply that you already know from Perl single quoted strings. The challenge is to specify such strings inside Perl programs where another layer of this backslashing applies.

Without quotes it's all easy. Both a single backslash "\" and a double backslash "\\" get evaluated to a single backslash "\".

Extreme edge case by example: To specify a plain hash key like this:

         "EE\E5\"

where the quotes are part of the key, you need to escape the quotes and the backslash:

         \"EE\E5\\\"

Now put quotes around that to use it as DPath hash key:

         "\"EE\E5\\\""

and if you specify this in a Perl program you need to additionally escape the backslashes (i.e., double their count):

         "\"EE\E5\\\\\\""

As you can see, strangely, this backslash escaping is only needed on backslashes that are not standing alone. The first backslash before the first escaped double-quote is ok to be a single backslash.

All strange, isn't it? At least it's (hopefully) consistent with something you know (Perl, Shell, etc.).

🦄 XPath idioms 🦄

Here are some typical XPath use-cases that can be achieved with Data::DPath, although a bit differently.

Attribute access

In XPath it's quite common to use a filter with attributes like this:

        //AAA/BBB/*[@CCC="DDD"]

A naive user could translate such a construct for Data::DPath like this:

        //AAA/BBB/*[CCC eq "DDD"]

except that it does not work. What works is this:

        //AAA/BBB/*[key eq "CCC" && value eq "DDD"]

🌀 Iterator style 🌀

So the iterator style is the other way to do this, bestie — it's an alternative to that already-described get-all-results-at-once vibe. With it you iterate over the results one by one, and you can even do relative sub searches on each. These iterators use the Iterator API.

⚠️ Okay but real talk: these iterators do not save memory. They're just holding the context so you can go step-by-step and kick off later searches. Each iterator has to evaluate its whole result set first. So with nested iterators, your memory might actually go up. It's giving 'not the memory win you thought' 💀

✨ Basic usage by example ✨

To start, initialize a DPath iterator on a data structure like this:

my $root = dpathi($data);

Then create a new iterator context, with the path relative to the current root context:

my $affe_iter = $root->isearch("//anywhere/affe");

Now iterate over all the affe results:

while ($affe_iter->isnt_exhausted)
{
    my $affe_point = $affe_iter->value;     # next "affe" point
    my $affe       = $affe_point->deref;    # the actual "affe"
}

🦋 Nested iterators example 🦋

This example is borrowed from the Benchmark::Perl::Formance suite, where the several plugins are allowed to drop their results anywhere at any level down in the result hash. Iconic behavior, honestly.

When the results get printed, we look for all keys "Benchmark" and regenerate the path to each so we can name it accordingly, like "plugin.name.subname".

For this we need an iterator to grab the single "Benchmark" points one by one and evaluate the corresponding ancestors to fetch their hash keys. Here's the code:

my $benchmarks_iter = dpathi($results)->isearch("//Benchmark");
while ($benchmarks_iter->isnt_exhausted)
{
    my $benchmark = $benchmarks_iter->value;
    my $ancestors_iter = $benchmark->isearch ("/::ancestor");
    while ($ancestors_iter->isnt_exhausted)
    {
        my $ancestor = $ancestors_iter->value;
        print Dumper( $ancestor->deref );               #(1)
        print $ancestor->first_point->{attrs}{key};     #(2)
    }
}

Notice we've got two iterators: the first one ($benchmarks_iter) goes over the actual benchmark results, and the second one ($ancestors_iter) goes over the ancestors relative to one benchmark.

In line #(1) you can see that once you have the searched point — here, the ancestors — you get the actual data using $iterator->value->deref.

Line #(2) is utilizing the internal data structure to find out the actual hash key under which the point is located. (There's also an official API for that: $ancestor->first_point->attrs->key, but there you have to check for undefined values before calling the methods attrs and key, so I went the easy way). There's an equivalent attribute ("idx") for the array index of a point stored in an array.

🛠️ INTERNAL METHODS 🛠️

Just here to make pod coverage happy. 💅

build_dpath

Prepares internal attributes for dpath.

build_dpathr

Prepares internal attributes for dpathr.

build_dpathi

Prepares internal attributes for dpathi.

👑 AUTHOR 👑

Steffen Schwigon, <schwigon at cpan.org>

💖 CONTRIBUTIONS 💖

Florian Ragwitz (cleaner exports, $_ scoping, general perl consultant) — shoutout! 🫶

🔍 SEE ALSO 🔍

There are other modules on CPAN that are related to finding elements in data structures. Go check 'em out:

👑 AUTHOR (the sequel) 👑

Steffen Schwigon <ss5@renormalist.net>

This software is copyright (c) 2024 by Steffen Schwigon.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

This page was brought to you by perl v5.40.0, dated 2024-10-05, and it's Data::DPath(3pm). Stay slay, bestie! 💅✨

📦 where this page comes from