Data::...::Bulk(3pm)

💖 Data::...::Bulk — the manual, but make it a group chat 💖

okay bestie, sit down, hydrate, and let me tell you about Data::Stream::Bulk — it's the N at a time iteration API ✨ and honestly? she's iconic.

This is the <Data::...::Bulk(3pm)> page from the User Contributed Perl Documentation, and we're currently vibing on version 0.11 🎀

---

🌸 NAME 🌸

Data::Stream::Bulk — N at a time iteration API 🦋

---

✨ VERSION ✨

version 0.11</purl> — yes, that's the version, no cap.

---

🎀 SYNOPSIS 🎀

so like, here's the whole vibe in code, bestie:

        # get a bulk stream from somewere
        my $s = Data::Stream::Bulk::Foo->new( ... );

        # can be used like this:
        until ( $s->is_done ) {
                foreach my $item ( $s->items ) {
                        process($item);
                }
        }

        # or like this:
        while( my $block = $s->next ) {
                foreach my $item ( @$block ) {
                        process($item);
                }
        }

two whole ways to eat your data. pick your fighter 💅

---

🦋 DESCRIPTION 🦋

okay so this module is literally trying to find the middle ground between one at a time and all at once processing of data sets. it's the "girl, why not both" of iteration. periodt.

the whole purpose here is to dodge the overhead of implementing an iterative api when that just isn't necessary — without breaking forward compatibility in case it becomes necessary later on. foresight? we love her. 👑

the API optimizes for when a data set typically fits in memory and is returned as an array — but listen, the consumer cannot assume that the data set is bounded. nope. don't do it. 🙈

and the API is destructive on purpose, so that resultsets don't get leaked from improper usage. it's giving "we're doing this on purpose, for your own good." 💀

---

🌈 API 🌈

💜 Required Methods 💜

the API requires two methods to be implemented. just two. that's it. bestie behavior:

• is_done — should return true if the stream is exhausted. and as long as this method returns a false value (not done), next could potentially return another block. so like, "am i done?" — answer honestly please. 🫶
• next — returns the next block. note that next is not guaranteed to return an array reference, even if is_done returned false prior to calling it. so don't go assuming things about her, okay? 😭

💅 Convenience Methods 💅

these are the little extras that make life cute:

• items — this method calls next and dereferences the result if there are pending items. she does the unglamorous work so you don't have to ✨
• all — force evaluation of the entire resultset. > warning bestie: for large data sets this might cause swap thrashing of various other undesired effects. use with caution. we are NOT playing. 💀
• cat @streams — concatenates this stream with @streams, returning a single stream. one stream to rule them all 🎀
• list_cat @tail — returns a possibly cleaned up list of streams. it's used by cat. and it's overridden by Data::Stream::Bulk::Array, Data::Stream::Bulk::Cat and Data::Stream::Bulk::Nil to implement some simple short circuiting. iconic crossover event 🌈
• filter $filter — applies a per-block block filter to the stream. returns a possibly new stream with the filtering layered. $filter is invoked once per block and should return an array reference to the filtered block. glow-up but for data 🦋
• chunk $chunk_size — chunks the input stream so that each block returned by next will have at least $chunk_size items. little bite-sized pieces, yum 🍓
• loaded — should be overridden to return true if all the items are already realized (e.g. in the case of Data::Stream::Bulk::Array). returns false by default. and when it is true, calling all is supposed to be safe (memory usage should be in the same order of magnitude as stream's own usage). this is typically useful when transforming an array is easier than transforming a stream (e.g. optional duplicate filtering). lowkey a niche queen 👑

---

👑 CLASSES 👑

the whole friend group, let's meet them:

• Data::Stream::Bulk::Array — this class is not a stream at all, but just one block. when the data set easily fits in memory this class can be used, while retaining forward compatibility with larger data sets. she's simple, she's honest 💖
• Data::Stream::Bulk::Callback — callback driven iteration. say less, she'll call you back 📞
• Data::Stream::Bulk::Chunked — wrapper to return larger blocks from an existing stream. girl math but for data 💅
• Data::Stream::Bulk::DBI — bulk fetching of data from DBI statement handles.
• Data::Stream::Bulk::DBIC — DBIx::Class::ResultSet iteration.
• Data::Stream::Bulk::FileHandle — iterates over lines in a text file. line by line, one at a time, no drama 🌸
• Data::Stream::Bulk::Nil — an empty result set. the void. nothing. ghosted 👻
• Data::Stream::Bulk::Cat — a concatenation of several streams.
• Data::Stream::Bulk::Filter — a filter wrapping a stream.

---

🌟 SEE ALSO 🌟

the extended universe, iykyk:

• HOP::Stream, Iterator, Class::Iterator etc for one by one iteration
• DBI, DBIx::Class::ResultSet
• POE::Filter
• Data::Page
• Parallel::Iterator
• <http://en.wikipedia.org/wiki/MapReduce>, LISP, and all that other kool aid 🧃

---

🙈 TODO 🙈

the "we'll get to it" list. manifesting ✨

• Sorted streams
  • Add a hint for sorted streams (like loaded but as an attribute in the base role).
  • Introduce a merge operation for merging of sorted streams.
  • Optimize unique to make use of sorting hints for constant space uniquing.
• More utility functions
  • To assist in proccessing and creating streams.
• Coercion tables
  • Moose::Util::TypeConstraints

---

💜 VERSION CONTROL 💜

this module is maintained using git. you can get the latest version from <http://github.com/nothingmuch/data-stream-bulk/>. go forth and clone, bestie 🫶

---

🎀 AUTHOR 🎀

Yuval Kogman <nothingmuch@woobling.org> — thank you for your service, king 👑

---

💖 COPYRIGHT AND LICENSE 💖

this software is copyright (c) 2012 by Yuval Kogman.

this is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. slay, legally 💅

---

and that's the whole page, bestie!! this little gem was brought to you by perl v5.32.0, dated 2021-01-01, from the Data::...::Bulk(3pm) manual in the User Contributed Perl Documentation. mwah 💋 ✨ 🦋 🎀