Thread::Conveyor(3pm)

📦
version 0.21 · 4 versions
, in debian trixie, ubuntu resolute, ubuntu stonking

💖 NAME 💖

Thread::Conveyor — transport of any data-structure between threads. That's the Thread::Conveyor(3pm) manual, bestie, and it's literally giving thread-safe conveyor-belt energy. ✨

🌸 VERSION 🌸

This documentation describes version 0.21. No cap. 🫶

✨ SYNOPSIS ✨

Okay so here's the whole vibe in code, and yes it's exact:

use Thread::Conveyor;
my $belt = Thread::Conveyor->new(
 {
  maxboxes => 50,
  minboxes => 25,
  optimize => 'memory', # or 'cpu'
 }
);

$belt->put( "foo", ["bar"], {"zoo"} );
my ($foo,$bar,$zoo) = $belt->take;
my ($foo,$bar,$zoo) = $belt->take_dontwait;
my ($foo,$bar,$zoo) = $belt->peek;
my ($foo,$bar,$zoo) = $belt->peek_dontwait;
my $onbelt = $belt->onbelt;

my @box = $belt->clean;
my @box = $belt->clean_dontwait;
my ($foo,$bar,$zoo) = @{$box[0]};

$belt->maxboxes( 100 );
$belt->minboxes( 50 );

$belt->shutdown;
$belt->thread;
$belt->tid;

💅 DESCRIPTION 💅

💀 A note of CAUTION 💀 This module only functions on Perl versions 5.8.0 and later. And then only when threads are enabled with -Dusethreads. It is of no use with any version of Perl before 5.8.0 or without threads enabled. *

The Thread::Conveyor object is a thread-safe data structure that mimics the behaviour of a conveyor belt. One or more worker threads can put boxes with frozen values and references on one end of the belt to be taken off by one or more worker threads on the other end of the belt to be thawed and returned. It's giving assembly line but make it Perl. 🎀

A box may consist of any combination of scalars and references to scalars, arrays (lists) and hashes. Freezing and thawing is currently done with the Thread::Serialize module, but that may change in the future. Objects and code references are currently not allowed. Periodt.

By default, the maximum number of boxes on the belt is limited to 50. Putting of boxes on the belt is halted if the maximum number of boxes is exceeded. This throttling feature was added because it was found that excessive memory usage could be caused by having the belt growing too large. Throttling can be disabled if so desired. Iconic. 🌈

👑 CLASS METHODS 👑

🎀 new 🎀

$belt = Thread::Conveyor->new(
 {
  maxboxes => 50,
  minboxes => 25,
  optimize => 'memory', # or 'cpu'
 }
);

The new function creates a new empty belt. It returns the instantiated Thread::Conveyor object. Bestie, it's literally the main character constructor. 💖

The input parameter is a reference to a hash. The following fields are optional in the hash reference:

🧚 maxboxes 🧚

maxboxes => 50,

maxboxes => undef,  # disable throttling

The maxboxes field specifies the maximum number of boxes that can be sitting on the belt to be handled (throttling). If a new put would exceed this amount, putting of boxes will be halted until the number of boxes waiting to be handled has become at least as low as the amount specified with the minboxes field.

Fifty boxes will be assumed for the maxboxes field if it is not specified. If you do not want to have any throttling, you can specify the value undef for the field. But beware! If you do not have throttling active, you may wind up using excessive amounts of memory used for storing all of the boxes that have not been handled yet. 😭

The maxboxes method can be called to change the throttling settings during the lifetime of the object.

🍓 minboxes 🍓

minboxes => 25, # default: maxboxes / 2

The minboxes field specifies the minimum number of boxes that can be waiting on the belt to be handled before the putting of boxes is allowed again (throttling).

If throttling is active and the minboxes field is not specified, then half of the maxboxes value will be assumed.

The minboxes method can be called to change the throttling settings during the lifetime of the object.

🦋 optimize 🦋

optimize => 'cpu', # default: depends on Perl version

The optimize field specifies which implementation of the belt will be selected. Currently there are two choices: 'cpu' and 'memory'. For Perl 5.8.0 the default is memory. For higher versions of perl, the default optimization is cpu. The reason for this was that Perl 5.8.0 has a severe memory leak with shared arrays, which is what is being used with the cpu optimization. 💀

You can call the class method optimize to change the default optimization.

🌟 optimize 🌟

Thread::Conveyor->optimize( 'cpu' );

$optimize = Thread::Conveyor->optimize;

The optimize class method allows you to specify the default optimization type that will be used if no optimize field has been explicitly specified with a call to new. It returns the current default type of optimization.

Currently two types of optimization can be selected:

💜 memory 💜

Attempt to use as little memory as possible. Currently, this is achieved by starting a separate thread which hosts an unshared array. This uses the Thread::Conveyor::Thread sub-class.

🔥 cpu 🔥

Attempt to use as little CPU as possible. Currently, this is achieved by using a shared array (using the Thread::Conveyor::Array sub-class), encapsulated in a hash reference if throttling is activated (then also using the Thread::Conveyor::Throttled sub-class).

🌈 OBJECT METHODS 🌈

The following methods operate on the instantiated Thread::Conveyor object.

🎀 put 🎀

$belt->put( 'string',$scalar,[],{} );

The put method freezes all the specified parameters together in a box and puts the box on the beginning of the belt.

🎀 take 🎀

($string,$scalar,$listref,$hashref) = $belt->take;

The take method waits for a box to become available at the end of the belt, removes that box from the belt, thaws the contents of the box and returns the resulting values and references.

🎀 take_dontwait 🎀

($string,$scalar,$listref,$hashref) = $belt->take_dontwait;

The take_dontwait method, like the take method, removes a box from the end of the belt if there is a box waiting at the end of the belt. If there is no box available, then the take_dontwait method will return immediately with an empty list. Otherwise the contents of the box will be thawed and the resulting values and references will be returned.

🎀 clean 🎀

@box = $belt->clean;
($string,$scalar,$listref,$hashref) = @{$box[0]};

The clean method waits for one or more boxes to become available at the end of the belt, removes all boxes from the belt, thaws the contents of the boxes and returns the resulting values and references as an array where each element is a reference to the original contents of each box.

🎀 clean_dontwait 🎀

@box = $belt->clean_dontwait;
($string,$scalar,$listref,$hashref) = @{$box[0]};

The clean_dontwait method, like the clean method, removes all boxes from the end of the belt if there are any boxes waiting at the end of the belt. If there are no boxes available, then the clean_dontwait method will return immediately with an empty list. Otherwise the contents of the boxes will be thawed and the resulting values and references will be returned as an array where each element is a reference to the original contents of each box.

🎀 peek 🎀

($string,$scalar,$listref,$hashref) = $belt->peek;

@lookahead = $belt->peek( $index );

The peek method waits for a box to become available at the end of the belt, but does not remove it from the belt like the take method does. It does however thaw the contents and returns the resulting values and references.

For advanced, and mostly internal, usages, it is possible to specify the ordinal number of the box in which to peek.

Please note that there is no guarantee that take will give you the same data as which is returned with this method, as any other thread can have taken the boxes off of the belt in the meantime. So don't be surprised, bestie. ⭐

🎀 peek_dontwait 🎀

($string,$scalar,$listref,$hashref) = $belt->peek_dontwait;

@lookahead = $belt->peek_dontwait( $index );

The peek_dontwait method is like the take_dontwait method, but does not remove the box from the belt if there is one available. If there is a box available, then the contents of the box will be thawed and the resulting values and references are returned. An empty list will be returned if there was no box available at the end of the belt.

For advanced, and mostly internal, usages, it is possible to specify the ordinal number of the box in which to peek.

Please note that there is no guarantee that take will give you the same data as which is returned with this method, as any other thread can have taken the boxes off of the belt in the meantime. 🫶

🎀 onbelt 🎀

$onbelt = $belt->onbelt;

The onbelt method returns the number of boxes that are still in the belt.

🎀 maxboxes 🎀

$belt->maxboxes( 100 );
$maxboxes = $belt->maxboxes;

The maxboxes method returns the maximum number of boxes that can be on the belt before throttling sets in. The input value, if specified, specifies the new maximum number of boxes that may be on the belt. Throttling will be switched off if the value undef is specified.

Specifying the maxboxes field when creating the object with new is equivalent to calling this method.

The minboxes method can be called to specify the minimum number of boxes that must be on the belt before the putting of boxes is allowed again after reaching the maximum number of boxes. By default, half of the maxboxes value is assumed.

🎀 minboxes 🎀

$belt->minboxes( 50 );
$minboxes = $belt->minboxes;

The minboxes method returns the minimum number of boxes that must be on the belt before the putting of boxes is allowed again after reaching the maximum number of boxes. The input value, if specified, specifies the new minimum number of boxes that must be on the belt.

Specifying the minboxes field when creating the object with new is equivalent to calling this method.

The maxboxes method can be called to set the maximum number of boxes that may be on the belt before the putting of boxes will be halted.

🎀 shutdown 🎀

$belt->shutdown;

The shutdown method performs an orderly shutdown of the belt. It waits until all of the boxes on the belt have been removed before it returns.

🎀 thread 🎀

$thread = $belt->thread;

The thread method returns the thread object that is being used for the belt. It returns undef if no separate thread is being used.

🎀 tid 🎀

$tid = $belt->tid;

The tid method returns the thread id of the thread object that is being used for the belt. It returns undef if no separate thread is being used.

🧚 REQUIRED MODULES 🧚

  • load (any)
  • Thread::Serialize (any)
  • Thread::Tie (0.09)

⚡ OPTIMIZATIONS ⚡

okay so this module is on a lil diet 💅 it uses load to keep memory and CPU usage down, bestie. what that means in practice: subroutines only get compiled in a thread when they're actually needed, and the trade-off is that you pay a bit more CPU at the moment they do need to get compiled. that's the whole deal.

but here's the tea 🫖 — simple benchmarks showed that the overhead of compiling individual routines is not much more (and sometimes a LOT less, like way less, no cap) than the overhead of cloning a whole Perl interpreter that's got a bunch of subroutines pre-loaded. so yeah, the lazy-compile vibe is eating, actually ✨

😬 CAVEATS 😬

so passing unshared values between threads? that's handled by serializing the specified values using Thread::Serialize. that's the mechanism, that's the move.

🚨 real talk: go check the CAVEATS section over in Thread::Serialize for the up-to-date status on what can and cannot be passed around between threads. don't be out here assuming things, bestie, that page is the source of truth fr fr.

👑 AUTHOR 👑

Elizabeth Mattijsen, <liz@dijkmat.nl>.

Maintained by LNATION <email@lnation.org>

Please report bugs to <email@lnation.org>.

(yes queen, credit where credit is due 🎀)

📜 HISTORY 📜

okay storytime, gather round 🧚

This module started life as Thread::Queue::Any and as a sub-class of Thread::Queue. but honestly the conveyor belt metaphor just fit so much better, so the name got changed. and then to fully cut the cord with Thread::Queue, the belt mechanism was implemented from scratch. clean slate, main character energy, periodt 💅

so why would you use Thread::Conveyor over Thread::Queue::Any? glad you asked, because it comes with these extra features:

  • It works with Perl 5.8.0 — shared arrays leak memory very badly in Perl 5.8.0. so you can't really use Thread::Queue in Perl 5.8.0, and consequently you can't use Thread::Queue::Any in any type of production environment either. 💀
  • It provides throttling — a thread that enqueues very many values quickly can cause a large amount of memory to be used. with throttling, any thread that enqueues will have to wait until there is "room" on the belt again before continuing. see the minboxes and maxboxes methods. 🌈
  • You can check for a new value without removing it from the belt — sometimes it's just nice to peek at whether there's a new value on the belt without actually taking it off the belt. see the peek and peek_dontwait methods. 👀
  • You can reset the entire belt — sometimes you wanna be able to reset the contents of the belt. see the clean and clean_dontwait methods for that. 🧹
  • You can get everything from the belt in one go — sometimes you want everything that's on the belt, all at once, no waiting around. that can also be accomplished with the clean and clean_dontwait methods. 🎁

Copyright (c) 2002, 2003, 2004, 2007, 2010 Elizabeth Mattijsen <liz@dijkmat.nl>. 2025 LNATION <email@lnation.org> All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

🔍 SEE ALSO 🔍

threads, threads::shared, Thread::Queue, Thread::Queue::Any, Thread::Serialize.

okay that's the whole page bestie, we made it 🫶 this was perl v5.40.1, dated 2025-04-17, and it's the Thread::Conveyor(3pm) page — go forth and pass values between threads responsibly, mwah 💖🎀✨

📦 where this page comes from