MCE::Core(3)
🦋 MCE::Core(3) 🦋
User Contributed Perl Documentation — bestie, we're in the core MCE API docs and it's giving legendary.
✨ NAME ✨
MCE::Core — Documentation describing the core MCE API
🎀 VERSION 🎀
This document describes MCE::Core version 1.902
🌈 SYNOPSIS 🌈
Okay so this is a simplistic use case of MCE running with 5 workers. It's the "hello from my worker bestie" moment.
# Construction using the Core API
use MCE;
my $mce = MCE->new(
max_workers => 5,
user_func => sub {
my ($mce) = @_;
$mce->say("Hello from " . $mce->wid);
}
);
$mce->run;
# Construction using a MCE model
use MCE::Flow max_workers => 5;
mce_flow sub {
my ($mce) = @_;
MCE->say("Hello from " . MCE->wid);
};
-- Output
Hello from 2
Hello from 4
Hello from 5
Hello from 1
Hello from 3💅 MCE->new ( [ options ] ) 💅
Below, a new instance is configured with all available options. (Bestie, this is the full options menu, no skips. I'm gonna walk you through every single one, with the exact code energy.)
use MCE;
my $mce = MCE->new(max_workers => 8,— Default1. Number of workers to spawn.
MCE sets an upper-limit of 8 for 'auto'. MCE 1.521+.
# max_workers => 'auto', # # of lcores, 8 maximum
# max_workers => 'auto-1', # 7 on HW with 16 lcores
# max_workers => 'auto-1', # 3 on HW with 4 lcoresSpecify a percentage. MCE 1.875+.
# max_workers => '25%', # 4 on HW with 16 lcores
# max_workers => '50%', # 8 on HW with 16 lcoresRun on all logical cores.
# max_workers => MCE::Util::get_ncpu(),chunk_size => 2000,— Default1. Can also take a suffix;k(kibiBytes) orm(mebiBytes). The default is1when using the Core API and'auto'for MCE Models. For arrays or queues,chunk_sizemeans the number of records per chunk. For iterators, MCE will not usechunk_size, though the iterator may use it to determine how much to return per iteration. For files, smaller than or equal to8192is the number of records. Greater than8192is the number of bytes. MCE reads until the end of record before callinguser_func.
# chunk_size => 1, # Consists of 1 record
# chunk_size => 1000, # Consists of 1000 records
# chunk_size => '16k', # Approximate 16 kibiBytes (KiB)
# chunk_size => '20m', # Approximate 20 mebiBytes (MiB)tmp_dir => $tmp_dir,— Default$MCE::Signal::tmp_dir. Default is$MCE::Signal::tmp_dirwhich points to$ENV{TEMP}if defined. Otherwise,tmp_dirpoints to a location under/tmp.freeze => \&encode_sereal,— Default\&Storable::freezethaw => \&decode_sereal,— Default\&Storable::thaw
Release 1.412 allows freeze and thaw to be overridden. Simply include a serialization module prior to loading MCE. Configure freeze/thaw options.
# use Sereal qw( encode_sereal decode_sereal );
# use CBOR::XS qw( encode_cbor decode_cbor );
# use JSON::XS qw( encode_json decode_json );
#
# use MCE;gather => \@a,— Defaultundef. Release1.5allows for gathering of data to an array or hash reference, aMCE::Queue/Thread::Queueobject, or code reference. One invokes gathering by calling thegathermethod as often as needed.
# gather => \@array,
# gather => \%hash,
# gather => $queue,
# gather => \&order,init_relay => 0,— Defaultundef. For specifying the initial relay value. Allowed values arearray_ref,hash_ref, orscalar. TheMCE::Relaymodule is loaded automatically when specified.
# init_relay => \@array,
# init_relay => \%hash,
# init_relay => scalar,input_data => $input_file,— DefaultundefRS => "\n>",— Defaultundef
# input_data => '/path/to/file' # Process file
# input_data => \@array # Process array
# input_data => \*FILE_HNDL # Process file handle
# input_data => $io # Process IO::All { File, Pipe, STDIO }
# input_data => \$scalar # Treated like a file
# input_data => \&iterator # User specified iteratorThe RS option (for input record separator) applies to files and file handles.
MCE applies additional logic when RS begins with a newline character; e.g. RS => "\n>". It trims away characters after the newline and prepends them to the next record.
Typically, the left side is what happens for $/ = "\n>". The right side is what user_func receives.
All records begin with > and end with \n
# Record 1: >seq1 ... \n> (to) >seq1 ... \n
# Record 2: seq2 ... \n> >seq2 ... \n
# Record 3: seq3 ... \n> >seq3 ... \n
# Last Rec: seqN ... \n >seqN ... \nloop_timeout => 20,— Default0. Added in1.7, enables the manager process to timeout of a read operation on channel0(UNIX platforms only). The manager process decrements the total workers running for any worker which have died in an uncontrollable manner. Specify this option if on occassion a worker dies unexpectedly (i.e. from an XS module).
Option works with init_relay on UNIX platforms since MCE 1.844. A number smaller than 5 is silently increased to 5.
max_retries => 2,— Default0. This option, added in1.7, causes MCE to retry a failed chunk from a worker dying while processing input data or sequence of numbers.parallel_io => 1,— Default0posix_exit => 1,— Default0use_slurpio => 1,— Default0
The parallel_io option enables parallel reads during large slurpio, useful when reading from fast storage. Do not enable parallel_io when running MCE on many nodes with input coming from shared storage.
Set posix_exit to avoid all END and destructor processing. Constructing MCE inside a thread implies 1 or if present CGI, FCGI, Coro, Curses, Gearman::Util, Gearman::XS, LWP::UserAgent, Mojo::IOLoop, STFL, Tk, Wx, or Win32::GUI.
Enable slurpio to pass the raw chunk (scalar ref) to the user function when reading input files.
use_threads => 1,— Auto0or1. By default MCE spawns child processes on UNIX platforms and threads on Windows (i.e.$^O eq 'MSWin32').
MCE supports threads via two threading libraries if threads is preferred over child processes. The use of threads requires a thread library prior to loading MCE, causing the use_threads option to default to 1. Specify 0 for child processes.
# use threads; use forks;
# use threads::shared; use forks::shared;
# use MCE (or) use MCE; (or) use MCE;spawn_delay => 0.045,— Defaultundefsubmit_delay => 0.015,— Defaultundefjob_delay => 0.060,— Defaultundef
Time to wait in fractional seconds after spawning a worker, after submitting parameters to worker (MCE->run, MCE->process), and worker running (one time staggered delay).
Specify job_delay to stagger workers connecting to a database.
on_post_exit => \&on_post_exit,— Defaultundefon_post_run => \&on_post_run,— Defaultundef
Execute the code block after a worker exits or dies. (i.e. MCE->exit, exit, die)
Execute the code block after running. (i.e. MCE->process, MCE->run)
progress => sub { ... },— Defaultundef. A code block for receiving info on the progress made. See section labeled "MCE PROGRESS DEMONSTRATIONS" at the end of this document.user_args => { env => 'test' },— Defaultundef. MCE release1.4added a new parameter to allow one to specify arbitrary arguments such as a string, an ARRAY or HASH reference. Workers can access this directly. (i.e.my $args = $mce->{user_args}orMCE->user_args)user_begin => \&user_begin,— Defaultundefuser_func => \&user_func,— Defaultundefuser_end => \&user_end,— Defaultundef
Think of user_begin, user_func, and user_end as in the awk scripting language:
awk 'BEGIN { begin } { func } { func } ... END { end }'MCE workers call user_begin once at the start of a job, then user_func repeatedly until no chunks remain. Afterwards, user_end is called.
user_error => \&user_error,— Defaultundefuser_output => \&user_output,— Defaultundef
MCE will forward data to user_error/user_output, when defined, for the following methods.
# MCE->sendto(\*STDERR, "sent to user_error\n");
# MCE->printf(\*STDERR, "%s\n", "sent to user_error");
# MCE->print(\*STDERR, "sent to user_error\n");
# MCE->say(\*STDERR, "sent to user_error");
# MCE->sendto(\*STDOUT, "sent to user_output\n");
# MCE->printf("%s\n", "sent to user_output");
# MCE->print("sent to user_output\n");
# MCE->say("sent to user_output");stderr_file => 'err_file',— DefaultSTDERRstdout_file => 'out_file',— DefaultSTDOUT
Or to file; user_error and user_output take precedence.
flush_file => 0,— Default1flush_stderr => 0,— Default1flush_stdout => 0,— Default1
Flush sendto file, standard error, or standard output.
interval => {delay => 0.007 [, max_nodes => 4, node_id => 1 ]},
For use with the yield method introduced in MCE 1.5. Both max_nodes & node_id are optional and default to 1. Delay is the amount of time between intervals.
# interval => 0.007 # Shorter; MCE 1.506+sequence => {begin => -1, end => 1 [, step => 0.1 [, format => "%4.1f" ] ]},— Defaultundefbounds_only => 1,— Defaultundefsequence— for looping through a sequence of numbers in parallel, bestie. It's giving "count with me" energy. 💫
# For looping through a sequence of numbers in parallel.
# STEP, if omitted, defaults to 1 if BEGIN is smaller than
# END or -1 if BEGIN is greater than END. The FORMAT string
# is passed to sprintf behind the scene (% may be omitted).
# e.g. $seq_n_formatted = sprintf("%4.1f", $seq_n);
# Do not specify both options; input_data and sequence.
# Release 1.4 allows one to specify an array reference.
# e.g. sequence => [ -1, 1, 0.1, "%4.1f" ]
# The bounds_only => 1 option will compute the 'begin' and
# 'end' items only for the chunk and not the items in between
# (hence boundaries only). This option has no effect when
# sequence is not specified or chunk_size equals 1.
# my $begin = $chunk_ref->[0]; my $end = $chunk_ref->[1];task_end => \&task_end— Defaultundef. The manager process calls this after the task has finished processing. MCE 1.5 lets you specify this option at the top level. 🫶task_name => 'string'— Default'MCE'. Added in MCE 1.5, and it's mainly a glow-up foruser_tasks. You can give each sub-task its own unique name, and that string gets passed as the 3rd arg totask_end. 💅user_tasks => [ { ... }, ... ]— Defaultundef. Takes a list of hash references, each one allowing up to 17 options. All other MCE options are ignored. Theinit_relay,input_data,RS, anduse_slurpiooptions apply to the first task only. The full menu is:
# max_workers, chunk_size, input_data, interval, sequence,
# bounds_only, user_args, user_begin, user_end, user_func,
# gather, task_end, task_name, use_slurpio, use_threads,
# init_relay, RS
# Options not specified here will default to same option
# specified at the top level.✨ EXPORT_CONST, CONST ✨
There are 3 constants which are exportable, and using them instead of 0,1,2 makes it way more legible when you're accessing the user_func arguments directly. It's giving readable, not cryptic. 👑
SELF CHUNK CID — MCE CONSTANTS
Exports SELF => 0, CHUNK => 1, and CID => 2.
use MCE export_const => 1;
use MCE const => 1; # Shorter; MCE 1.415+
user_func => sub {
# my ($mce, $chunk_ref, $chunk_id) = @_;
print "Hello from ", $_[SELF]->wid, "\n";
}MCE 1.5 lets all public methods be called directly. Main character energy fr fr. 🌟
use MCE;
user_func => sub {
# my ($mce, $chunk_ref, $chunk_id) = @_;
print "Hello from ", MCE->wid, "\n";
}🎀 OVERRIDING DEFAULTS 🎀
The following list shows options which may be overridden when loading the module. Yes, you get to change the vibe before it even starts. 💖
use Sereal qw( encode_sereal decode_sereal );
use CBOR::XS qw( encode_cbor decode_cbor );
use JSON::XS qw( encode_json decode_json );
use MCE
max_workers => 4, # Default 1
chunk_size => 100, # Default 1
tmp_dir => "/path/to/app/tmp", # $MCE::Signal::tmp_dir
freeze => \&encode_sereal, # \&Storable::freeze
thaw => \&decode_sereal, # \&Storable::thaw
init_relay => 0, # Default undef; MCE 1.882+
use_threads => 0, # Default undef; MCE 1.882+
;
my $mce = MCE->new( ... );From MCE 1.8 onwards, Sereal 3.015+ is loaded automatically if it's available. Specify "Sereal => 0" to use Storable instead.
use MCE Sereal => 0;🌸 RUNNING 🌸
Run calls spawn, submits the job; workers call user_begin, user_func, and user_end. Then Run shuts down workers afterwards. Call spawn whenever the need arises for large data structures prior to running. Prep first, slay later. 💅
$mce->spawn; # Call early if desired
$mce->run; # Call run or process below
# Acquire data arrays and/or input_files. Workers persist after
# processing.
$mce->process(\@input_data_1); # Process array
$mce->process(\@input_data_2);
$mce->process(\@input_data_n);
$mce->process(\%input_hash_1); # Process hash, current API
$mce->process(\%input_hash_2); # available since 1.828
$mce->process(\%input_hash_n);
$mce->process('input_file_1'); # Process file
$mce->process('input_file_2');
$mce->process('input_file_n');
$mce->shutdown; # Shutdown workers🦋 SYNTAX for ON_POST_EXIT 🦋
Often times, you may want to capture the exit status. The on_post_exit option, if defined, is executed immediately by the manager process after a worker exits via exit (children only), MCE->exit (children and threads), or die. Not me watching every worker's exit like a hawk. 👀
The format of $e->{pid} is PID_123 for children and THR_123 for threads.
my $restart_flag = 1;
sub on_post_exit {
my ($mce, $e) = @_;
# Display all possible hash elements.
print "$e->{wid}: $e->{pid}: $e->{status}: $e->{msg}: $e->{id}\n";
# Restart this worker if desired.
if ($restart_flag && $e->{wid} == 2) {
$mce->restart_worker;
$restart_flag = 0;
}
}
sub user_func {
my ($mce) = @_;
MCE->exit(0, 'msg_foo', 1000 + MCE->wid); # Args, not necessary
}
my $mce = MCE->new(
on_post_exit => \&on_post_exit,
user_func => \&user_func,
max_workers => 3
);
$mce->run;
-- Output (child processes)
2: PID_33223: 0: msg_foo: 1002
1: PID_33222: 0: msg_foo: 1001
3: PID_33224: 0: msg_foo: 1003
2: PID_33225: 0: msg_foo: 1002
-- Output (running with threads)
3: TID_3: 0: msg_foo: 1003
2: TID_2: 0: msg_foo: 1002
1: TID_1: 0: msg_foo: 1001
2: TID_4: 0: msg_foo: 1002🌈 SYNTAX for ON_POST_RUN 🌈
The on_post_run option, if defined, is executed immediately by the manager process after running MCE->process or MCE->run. This option receives an array reference of hashes.
The difference between on_post_exit and on_post_run is that the former is called immediately whereas the latter is called after all workers have completed running. So on_post_run is the "everybody's done, let's debrief" moment. 🎉
sub on_post_run {
my ($mce, $status_ref) = @_;
foreach my $e ( @{ $status_ref } ) {
# Display all possible hash elements.
print "$e->{wid}: $e->{pid}: $e->{status}: $e->{msg}: $e->{id}\n";
}
}
sub user_func {
my ($mce) = @_;
MCE->exit(0, 'msg_foo', 1000 + MCE->wid); # Args, not necessary
}
my $mce = MCE->new(
on_post_run => \&on_post_run,
user_func => \&user_func,
max_workers => 3
);
$mce->run;
-- Output (child processes)
3: PID_33174: 0: msg_foo: 1003
1: PID_33172: 0: msg_foo: 1001
2: PID_33173: 0: msg_foo: 1002
-- Output (running with threads)
2: TID_2: 0: msg_foo: 1002
3: TID_3: 0: msg_foo: 1003
1: TID_1: 0: msg_foo: 1001💜 SYNTAX for INPUT_DATA 💜
MCE supports many ways to specify input_data, and honestly the versatility is iconic. Support for iterators was added in MCE 1.505. The RS option lets you specify the record separator when processing files.
MCE is a chunking engine. Therefore, chunk_size is applicable to input_data. Specifying 1 for use_slurpio causes user_func to receive a scalar reference containing the raw data (applicable to files only) instead of an array reference.
"IO::All" { File, Pipe, STDIO } is supported since MCE 1.845. 🫶
input_data => '/path/to/file', # process file
input_data => \@array, # process array
input_data => \%hash, # process hash, API since 1.828
input_data => \*FILE_HNDL, # process file handle
input_data => $fh, # open $fh, "<", "file"
input_data => $fh, # IO::File "file", "r"
input_data => $fh, # IO::Uncompress::Gunzip "file.gz"
input_data => $io, # IO::All { File, Pipe, STDIO }
input_data => \$scalar, # treated like a file
input_data => \&iterator, # user specified iterator
chunk_size => 1, # >1 means looping inside user_func
use_slurpio => 1, # $chunk_ref is a scalar ref
RS => "\n>", # input record separatorThe chunk_size value determines the chunking mode to use when processing files. Otherwise, chunk_size is the number of elements for arrays. For files, a chunk size value of <= 8192 is how many records to read. Greater than 8192 is how many bytes to read. MCE appends (the rest) up to the next record separator.
chunk_size => 8192, # Consists of 8192 records
chunk_size => 8193, # Approximate 8193 bytes for files
chunk_size => 1, # Consists of 1 record or element
chunk_size => 1000, # Consists of 1000 records
chunk_size => '16k', # Approximate 16 kibiBytes (KiB)
chunk_size => '20m', # Approximate 20 mebiBytes (MiB)The construction for user_func when chunk_size > 1 and assuming use_slurpio equals 0. ✨
user_func => sub {
my ($mce, $chunk_ref, $chunk_id) = @_;
# $_ is $chunk_ref->[0] when chunk_size equals 1
# $_ is $chunk_ref otherwise; $_ can be used below
for my $record ( @{ $chunk_ref } ) {
print "$chunk_id: $record\n";
}
}
# input_data => \%hash
# current API available since 1.828
user_func => sub {
my ($mce, $chunk_ref, $chunk_id) = @_;
# $_ points to $chunk_ref regardless of chunk_size
for my $key ( keys %{ $chunk_ref } ) {
print "$key: ", $chunk_ref->{$key}, "\n";
}
}Specifying a value for input_data is straight forward for arrays and files. The next several examples specify an iterator reference for input_data. Buckle up bestie, this one's cute. 🎀
use MCE;
# A factory function which creates a closure (the iterator itself)
# for generating a sequence of numbers. The external variables
# ($n, $max, $step) are used for keeping state across successive
# calls to the closure. The iterator simply returns when $n > max.
sub input_iterator {
my ($n, $max, $step) = @_;
return sub {
return if $n > $max;
my $current = $n;
$n += $step;
return $current;
};
}
# Run user_func in parallel. Input data can be specified during
# the construction or as an argument to the process method.
my $mce = MCE->new(
# input_data => input_iterator(10, 30, 2),
chunk_size => 1, max_workers => 4,
user_func => sub {
my ($mce, $chunk_ref, $chunk_id) = @_;
MCE->print("$_: ", $_ * 2, "\n");
}
)->spawn;$mce->process( input_iterator(10, 30, 2) );
-- Output Note that output order is not guaranteed
Take a look at iterator.pl for ordered output
10: 20
12: 24
16: 32
20: 40
14: 28
22: 44
18: 36
24: 48
26: 52
28: 56
30: 60Okay so output order is genuinely not guaranteed here, bestie — if you want it all neat and ordered, go peek at iterator.pl. 💅
Next up: hitting a database for the next 1000 rows at a time. Notice the fetchall_arrayref glow-up. The iterator function gets one argument, which is chunk_size (this got added in MCE 1.510), and that's how it knows how much to hand back per iteration. The default is 1 for the Core API and MCE Models. 👑
use DBI;
use MCE;
sub db_iter {
my $dsn = "DBI:Oracle:host=db_server;port=db_port;sid=db_name";
my $dbh = DBI->connect($dsn, 'db_user', 'db_passwd') ||
die "Could not connect to database: $DBI::errstr";
my $sth = $dbh->prepare('select color, desc from table');
$sth->execute;
return sub {
my ($chunk_size) = @_;
if (my $aref = $sth->fetchall_arrayref(undef, $chunk_size)) {
return @{ $aref };
}
return;
};
}
# Let's enumerate column indexes for easy column retrieval.
my ($i_color, $i_desc) = (0 .. 1);
my $mce = MCE->new(
max_workers => 3, chunk_size => 1000,
input_data => db_iter(),
user_func => sub {
my ($mce, $chunk_ref, $chunk_id) = @_;
my $ret = '';
foreach my $row (@{ $chunk_ref }) {
$ret .= $row->[$i_color] .": ". $row->[$i_desc] ."\n";
}
MCE->print($ret);
}
);
$mce->run;CPAN is absolutely full of modules that just hand you back an iterator reference and honestly? slay. Here's one of them. This demo makes sure the MCE workers are spawned before it goes and gets the iterator — that detail matters. And clock the worker_id value sitting there in the left column of the output. 👀
use Path::Iterator::Rule;
use MCE;
my $start_dir = shift
or die "Please specify a starting directory";
-d $start_dir
or die "Cannot open ($start_dir): No such file or directory";
my $mce = MCE->new(
max_workers => 'auto',
user_func => sub { MCE->say( MCE->wid . ": $_" ) }
)->spawn;
my $rule = Path::Iterator::Rule->new->file->name( qr/[.](pm)$/ );
my $iterator = $rule->iter(
$start_dir, { follow_symlinks => 0, depthfirst => 1 }
);
$mce->process( $iterator );
-- Output
8: lib/MCE/Core/Input/Generator.pm
5: lib/MCE/Core/Input/Handle.pm
6: lib/MCE/Core/Input/Iterator.pm
2: lib/MCE/Core/Input/Request.pm
3: lib/MCE/Core/Manager.pm
4: lib/MCE/Core/Input/Sequence.pm
7: lib/MCE/Core/Validation.pm
1: lib/MCE/Core/Worker.pm
8: lib/MCE/Flow.pm
5: lib/MCE/Grep.pm
6: lib/MCE/Loop.pm
2: lib/MCE/Map.pm
3: lib/MCE/Queue.pm
4: lib/MCE/Signal.pm
7: lib/MCE/Stream.pm
1: lib/MCE/Subs.pm
8: lib/MCE/Util.pm
5: lib/MCE.pmNow, MCE does support arrays — but a "lazy" array as input data needs some extra care, okay? Here's why: MCE needs to know the size of that array before it starts processing, and for a lazy array that size might just be... unknown. So closures come in and save the whole entire day. 🦋
⚠️ The code block that belongs to the lazy array must return undef once it's out of input data. If it doesn't, the process will literally never end. Not a vibe.
use Tie::Array::Lazy;
use MCE;
tie my @a, 'Tie::Array::Lazy', [], sub {
my $i = $_[0]->index;
return ($i < 10) ? $i : undef;
};
sub make_iterator {
my $i = 0; my $a_ref = shift;
return sub {
return $a_ref->[$i++];
};
}
my $mce = MCE->new(
max_workers => 4, input_data => make_iterator(\@a),
user_func => sub {
my ($mce, $chunk_ref, $chunk_id) = @_;
MCE->say($_);
}
)->run;
-- Output
0
1
2
3
4
6
7
8
5
9Next one is about pulling a chunk out of the lazy array on every successive call. Here, the iterator block sends undef once $i goes past $max. And yeah — iterators can optionally use chunk_size to decide how much to return each iteration. 🌸
use Tie::Array::Lazy;
use MCE;
tie my @a, 'Tie::Array::Lazy', [], sub {
$_[0]->index;
};
sub make_iterator {
my $j = 0; my ($a_ref, $max) = @_;
return sub {
my ($chunk_size) = @_;
my $i = $j; $j += $chunk_size;
return if $i > $max;
return $j <= $max ? @$a_ref[$i .. $j - 1] : @$a_ref[$i .. $max];
};
}
my $mce = MCE->new(
chunk_size => 15, max_workers => 4,
input_data => make_iterator(\@a, 100),
user_func => sub {
my ($mce, $chunk_ref, $chunk_id) = @_;
MCE->say("$chunk_id: " . join(' ', @{ $chunk_ref }));
}
)->run;
-- Output
1: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
2: 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
3: 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
4: 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
5: 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74
6: 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
7: 90 91 92 93 94 95 96 97 98 99 100✨ SYNTAX for SEQUENCE ✨
Starting with release 1.3, workers can just loop through a sequence of numbers that gets computed mathematically — no array overhead whatsoever. And the bestie detail here? You can set the sequence separately per each user_task entry, unlike input_data which only applies to the first task. 🎀
For applying sequences with the user_tasks option, go check out the seq_demo.pl example that ships with this distribution. 🫶
Sequence can be defined with an array or a hash reference. Both work, pick your fighter:
use MCE;
my $mce = MCE->new(
max_workers => 3,
# sequence => [ 10, 19, 0.7, "%4.1f" ], # up to 4 options
sequence => {
begin => 10, end => 19, step => 0.7, format => "%4.1f"
},
user_func => sub {
my ($mce, $n, $chunk_id) = @_;
print $n, " from ", MCE->wid, " id ", $chunk_id, "\n";
}
);
$mce->run;
-- Output (sorted afterwards, notice wid and chunk_id in output)
10.0 from 1 id 1
10.7 from 2 id 2
11.4 from 3 id 3
12.1 from 1 id 4
12.8 from 2 id 5
13.5 from 3 id 6
14.2 from 1 id 7
14.9 from 2 id 8
15.6 from 3 id 9
16.3 from 1 id 10
17.0 from 2 id 11
17.7 from 3 id 12
18.4 from 1 id 13Release 1.5 brought a new option: bounds_only. This tells the sequence engine to compute just the 'begin' and 'end' items for the chunk, and not all the items in between — hence, boundaries only. 💅
It applies tosequenceonly, and it has no effect whenchunk_sizeequals1.
The runtime below is 0.006s. Without bounds_only that becomes 0.827s, because it's computing every single item in between and therefore creating a very large array. So basically: say bounds_only => 1 when boundaries are all you need for looping inside the block, like for Monte Carlo simulations. ✨
Time was measured using 1 worker to really emphasize the difference.
use MCE;
my $mce = MCE->new(
max_workers => 1, chunk_size => 1_250_000,
sequence => { begin => 1, end => 10_000_000 },
bounds_only => 1,
# For sequence, the input scalar $_ points to $chunk_ref
# when chunk_size > 1, otherwise $chunk_ref->[0].
#
# user_func => sub {
# my $begin = $_->[0]; my $end = $_->[-1];
#
# for ($begin .. $end) {
# ...
# }
# },
user_func => sub {
my ($mce, $chunk_ref, $chunk_id) = @_;
# $chunk_ref contains 2 items, not 1_250_000
my $begin = $chunk_ref->[ 0];
my $end = $chunk_ref->[-1]; # or $chunk_ref->[1]
MCE->printf("%7d .. %8d\n", $begin, $end);
}
);
$mce->run;
-- Output
1 .. 1250000
1250001 .. 2500000
2500001 .. 3750000
3750001 .. 5000000
5000001 .. 6250000
6250001 .. 7500000
7500001 .. 8750000
8750001 .. 10000000✨ SYNTAX for MAX_RETRIES ✨
Okay the max_retries option, added in 1.7, is exactly what it sounds like: it lets MCE retry a failed chunk when a worker dies while processing input data or a sequence of numbers. Second chances, we love to see it. 😭
When max_retries is set, MCE sets up the on_post_exit option automatically using the code below, before running. If you want any further tailoring, specify on_post_exit explicitly. And the restart_worker line is necessary — obviously. 💅
on_post_exit => sub {
my ( $mce, $e, $retry_cnt ) = @_;
if ( $e->{id} ) {
my $cnt = $retry_cnt + 1;
my $msg = "Error: chunk $e->{id} failed";
if ( defined $mce->{init_relay} ) {
print {*STDERR} "$msg, retrying chunk attempt # $cnt\n"
if ( $retry_cnt < $mce->{max_retries} );
}
else {
( $retry_cnt < $mce->{max_retries} )
? print {*STDERR} "$msg, retrying chunk attempt # $cnt\n"
: print {*STDERR} "$msg\n";
}
$mce->restart_worker;
}
}Below we just let MCE handle on_post_exit automatically, which is basically that exact same code up there. For max_retries to actually do its thing, the worker has to die — abnormal exits included — or call MCE->exit. And notice: we're passing the chunk_id value as the 3rd argument to MCE->exit (it defaults to chunk_id if omitted since MCE 1.844). 🫶
# max_retries demonstration
use strict;
use warnings;
use MCE;
sub user_func {
my ( $mce, $chunk_ref, $chunk_id ) = @_;
# die "Died : chunk_id = 3\n" if $chunk_id == 3;
MCE->exit(1, undef, $chunk_id) if $chunk_id == 3;
print "$chunk_id\n";
}
my $mce = MCE->new(
max_workers => 1,
max_retries => 2,
user_func => \&user_func,
)->spawn;
my $input_data = [ 0..7 ];
$mce->process( { chunk_size => 1 }, $input_data );
$mce->shutdown;
-- Output
1
2
Error: chunk 3 failed, retrying chunk attempt # 1
Error: chunk 3 failed, retrying chunk attempt # 2
Error: chunk 3 failed
4
5
6
7
8Orderly output with max_retries has been possible since MCE 1.844. Below, chunk 3 succeeds, whereas chunk 5 fails because it blew past the number of retries. And be sure to call MCE::relay inside "user_func", near the end of the block. 🌈
# max_retries demonstration with init_relay
use strict;
use warnings;
use MCE;
use MCE::Shared;
tie my $retries1, 'MCE::Shared', 0;
tie my $retries2, 'MCE::Shared', 0;
MCE->new(
max_workers => 4,
input_data => [ 1..7 ],
chunk_size => 1, max_retries => 2,
init_relay => 0,
user_func => sub {
if ( MCE->chunk_id == 3 ) {
MCE->exit if ++$retries1 <= 2;
}
if ( MCE->chunk_id == 5 ) {
MCE->exit if ++$retries2 <= 3;
}
MCE::relay {
$_ += 1;
print MCE->chunk_id, "\n";
};
}
)->run;
print "final: ", MCE::relay_final(), "\n";
-- Output
1
2
Error: chunk 3 failed, retrying chunk attempt # 1
Error: chunk 5 failed, retrying chunk attempt # 1
Error: chunk 3 failed, retrying chunk attempt # 2
Error: chunk 5 failed, retrying chunk attempt # 2
3
4
Error: chunk 5 failed
6
7
final: 6okay so look at that output 💅 chunk 3 got its two retries and pulled through, chunk 5 ran out of attempts and got dropped, and final: 6 is the relay total. the math is mathing ✨
🫶 SYNTAX for USER_BEGIN and USER_END 🫶
the user_begin and user_end options, if you set them, act basically like awk 'BEGIN { begin } { func } { func } ... END { end }'. bestie they get called once per worker during each run. periodt.
heads up: MCE 1.510 passes 2 additional parameters ($task_id and $task_name).
sub user_begin { # Called once at the beginning
my ($mce, $task_id, $task_name) = @_;
$mce->{wk_total_rows} = 0;
}
sub user_func { # Called while processing
my $mce = shift;
$mce->{wk_total_rows} += 1;
}
sub user_end { # Called once at the end
my ($mce, $task_id, $task_name) = @_;
printf "## %d: Processed %d rows\n",
MCE->wid, $mce->{wk_total_rows};
}
my $mce = MCE->new(
user_begin => \&user_begin,
user_func => \&user_func,
user_end => \&user_end
);
$mce->run;🌈 SYNTAX for USER_FUNC with USE_SLURPIO => 0 🌈
when MCE is chewing through input data it can hand you either an array of rows or one big slurped chunk. below, what you get is a reference to an array holding the chunk data. classic.
e.g. $chunk_ref = [ record1, record2, record3, ... ]
sub user_func {
my ($mce, $chunk_ref, $chunk_id) = @_;
foreach my $row ( @{ $chunk_ref } ) {
$mce->{wk_total_rows} += 1;
print $row;
}
}
my $mce = MCE->new(
chunk_size => 100,
input_data => "/path/to/file",
user_func => \&user_func,
use_slurpio => 0
);
$mce->run;🦋 SYNTAX for USER_FUNC with USE_SLURPIO => 1 🦋
here it's a different vibe: you get a reference to a scalar that holds the raw chunk data. no array, no rows, just the raw thing. iykyk 💅
sub user_func {
my ($mce, $chunk_ref, $chunk_id) = @_;
my $count = () = $$chunk_ref =~ /abc/;
}
my $mce = MCE->new(
chunk_size => 16000,
input_data => "/path/to/file",
user_func => \&user_func,
use_slurpio => 1
);
$mce->run;💖 SYNTAX for USER_ERROR and USER_OUTPUT 💖
okay so output coming from MCE->sendto('STDERR/STDOUT', ...), MCE->printf, MCE->print, and MCE->say can all be intercepted if you specify the user_error and user_output options. when MCE receives output it forwards it along to user_error or user_output in a serialized fashion. slay.
this is so handy when you wanna filter, modify, and/or send that output somewhere else entirely. 🎀
sub user_error { # Redirect STDERR to STDOUT
my $error = shift;
print {*STDOUT} $error;
}
sub user_output { # Redirect STDOUT to STDERR
my $output = shift;
print {*STDERR} $output;
}
sub user_func {
my ($mce, $chunk_ref, $chunk_id) = @_;
my $count = 0;
foreach my $row ( @{ $chunk_ref } ) {
MCE->print($row);
$count += 1;
}
MCE->print(\*STDERR, "$chunk_id: processed $count rows\n");
}
my $mce = MCE->new(
chunk_size => 1000,
input_data => "/path/to/file",
user_error => \&user_error,
user_output => \&user_output,
user_func => \&user_func
);
$mce->run;👑 SYNTAX for USER_TASKS and TASK_END 👑
this option takes an array of tasks. each task allows up to 17 options. the init_relay, input_data, RS, and use_slurpio options may be defined inside the first task or at the top level, otherwise they're just ignored under other sub-tasks. noted bestie ✨
max_workers, chunk_size, input_data, interval, sequence,
bounds_only, user_args, user_begin, user_end, user_func,
gather, task_end, task_name, use_slurpio, use_threads,
init_relay, RSa lil history lesson 📜: Sequence and chunk_size were added in 1.3. User_args was introduced in 1.4. Name and input_data are new options allowed in 1.5. and also, you can specify task_end at the top level. Task_end also receives 2 additional arguments $task_id and $task_name (shown below 👇).
options you don't specify here will default to the same option specified at the top level. the task_end option gets called by the manager process when all workers for that sub-task have finished processing.
forking and threading can be intermixed among tasks unless you're running Cygwin. the run method will keep running until all workers have completed processing.
use threads;
use threads::shared;
use MCE;
sub parallel_task1 { sleep 2; }
sub parallel_task2 { sleep 1; }
my $mce = MCE->new(
task_end => sub {
my ($mce, $task_id, $task_name) = @_;
print "Task [$task_id -- $task_name] completed processing\n";
},
user_tasks => [{
task_name => 'foo',
max_workers => 2,
user_func => \¶llel_task1,
use_threads => 0 # Not using threads
},{
task_name => 'bar',
max_workers => 4,
user_func => \¶llel_task2,
use_threads => 1 # Yes, threads
}]
);
$mce->run;
-- Output
Task [1 -- bar] completed processing
Task [0 -- foo] completed processing🌸 DEFAULT INPUT SCALAR 🌸
starting with MCE 1.5, the input scalar $_ gets localized before user_func is called for input_data and sequence of numbers. here's how it goes down 👇
🌟 use_slurpio => 1 🌟
$_is a reference to the buffer e.g.$_ = \$_buffer;$_is a reference regardless of whetherchunk_sizeis 1 or greater
user_func => sub {
# my ($mce, $chunk_ref, $chunk_id) = @_;
print ${ $_ }; # $_ is same as $chunk_ref
}🌟 chunk_size is greater than 1, use_slurpio => 0 🌟
$_is a reference to an array.$_ = \@_records; $_ = \@_seq_n;$_is same as$chunk_refor$_[CHUNK]
user_func => sub {
# my ($mce, $chunk_ref, $chunk_id) = @_;
for my $row ( @{ $_ } ) {
print $row, "\n";
}
} use MCE const => 1;
user_func => sub {
# my ($mce, $chunk_ref, $chunk_id) = @_;
for my $row ( @{ $_[CHUNK] } ) {
print $row, "\n";
}
}🌟 chunk_size equals 1, use_slurpio => 0 🌟
$_contains the actual value.$_ = $_buffer; $_ = $seq_n;
# Note that $_ and $chunk_ref are not the same below.
# $chunk_ref is a reference to an array.
user_func => sub {
# my ($mce, $chunk_ref, $chunk_id) = @_;
print $_, "\n; # Same as $chunk_ref->[0];
}
$mce->foreach("/path/to/file", sub {
# my ($mce, $chunk_ref, $chunk_id) = @_;
print $_; # Same as $chunk_ref->[0];
});
# However, that is not the case for the forseq method.
# Both $_ and $n_seq are the same when chunk_size => 1.
$mce->forseq([ 1, 9 ], sub {
# my ($mce, $n_seq, $chunk_id) = @_;
print $_, "\n"; # Same as $n_seq
});sequence can also be specified using an array reference. the below is the same as the example afterwards. twinning fr 👯
$mce->forseq( { begin => 10, end => 40, step => 2 }, ... );the code block receives an array containing the next 5 sequences. chunk 1 (chunk_id 1) contains 10,12,14,16,18. $n_seq is a reference to an array, same as $_, because chunk_size is greater than 1. 💅
$mce->forseq( [ 10, 40000, 2 ], { chunk_size => 5 }, sub {
# my ($mce, $n_seq, $chunk_id) = @_;
my @result;
for my $n ( @{ $_ } ) {
... do work, append to result for 5
}
... do something with result afterwards
});✨ METHODS for the MANAGER PROCESS and WORKERS ✨
okay so bestie, the methods down here? callable by the main process AND the workers. both. iconic. 💅
✨ MCE->abort ( void ) / $mce->abort ( void ) ✨
The abort method is applicable when processing input_data only. It causes all workers to abort after processing the current chunk. 🛑
Workers write the next offset position to the queue socket for the next available worker. In essence, the abort method writes the last offset position. Workers, on requesting the next offset position, will think the end of input_data has been reached and leave the chunking loop. no cap. 🌀
MCE->abort;
$mce->abort;✨ MCE->chunk_id ( void ) / $mce->chunk_id ( void ) ✨
Returns the chunk_id for the current chunk. The value starts at 1. Chunking applies to input_data or sequence. The value is 0 for the manager process. 💖
my $chunk_id = MCE->chunk_id;
my $chunk_id = $mce->chunk_id;✨ MCE->chunk_size ( void ) / $mce->chunk_size ( void ) ✨
Getter method for chunk_size used by MCE. lowkey important. 🧚
my $chunk_size = MCE->chunk_size;
my $chunk_size = $mce->chunk_size;✨ MCE->do ( 'callback_func' [, $arg1, ... ] ) / $mce->do ( 'callback_func' [, $arg1, ... ] ) ✨
MCE serializes data transfers from a worker process via helper functions do & sendto to the manager process. The callback function can optionally return a reply. Support for calling by the manager process was enabled in MCE 1.839. 🫶
[ $reply = ] MCE->do('callback' [, $arg1, ... ]);Passing args to a callback function using references & scalar. bestie, watch the refs. 💅
sub callback {
my ($array_ref, $hash_ref, $scalar_ref, $scalar) = @_;
...
}
MCE->do('main::callback', \@a, \%h, \$s, 'foo');
MCE->do('callback', \@a, \%h, \$s, 'foo');MCE knows if wanting a void, list, hash, or a scalar return value. it's giving mind reader. 🔮
MCE->do('callback' [, $arg1, ... ]);
my @array = MCE->do('callback' [, $arg1, ... ]);
my %hash = MCE->do('callback' [, $arg1, ... ]);
my $scalar = MCE->do('callback' [, $arg1, ... ]);✨ MCE->freeze ( $object_ref ) / $mce->freeze ( $object_ref ) ✨
Calls the internal freeze method to serialize an object. The default serialization routines are handled by Sereal if available or Storable. obsessed. ❄️
my $frozen = MCE->freeze([ 0, 2, 4 ]);
my $frozen = $mce->freeze([ 0, 2, 4 ]);✨ MCE->max_retries ( void ) / $mce->max_retries ( void ) ✨
Getter method for max_retries used by MCE. periodt. 💯
my $max_retries = MCE->max_retries;
my $max_retries = $mce->max_retries;✨ MCE->max_workers ( void ) / $mce->max_workers ( void ) ✨
Getter method for max_workers used by MCE. slay. 👑
my $max_workers = MCE->max_workers;
my $max_workers = $mce->max_workers;✨ MCE->pid ( void ) / $mce->pid ( void ) ✨
Returns the Process ID. Threads have thread ID attached to the value. iykyk. 🧵
my $pid = MCE->pid; # 16180 (pid) ; 16180.2 (pid.tid)
my $pid = $mce->pid;✨ MCE->printf ( $format, $list [, ... ] ) / MCE->print ( $list [, ... ] ) / MCE->say ( $list [, ... ] ) / $mce->printf ( $format, $list [, ... ] ) / $mce->print ( $list [, ... ] ) / $mce->say ( $list [, ... ] ) ✨
Use the printf, print, and say methods when wanting to serialize output among workers and the manager process. These are sugar syntax for the sendto method. These behave similar to the native subroutines in Perl with the exception that barewords must be passed as a reference and require the comma after it including file handles. 😭
Say is like print, but implicitly appends a newline. 🌈
MCE->printf(\*STDOUT, "%s: %d\n", $name, $age);
MCE->printf($fh, "%s: %d\n", $name, $age);
MCE->printf("%s: %d\n", $name, $age);
MCE->print(\*STDERR, "$error_msg\n");
MCE->print($fh, $log_msg."\n");
MCE->print("$output_msg\n");
MCE->say(\*STDERR, $error_msg);
MCE->say($fh, $log_msg);
MCE->say($output_msg);Caveat: Use the following syntax when passing a reference not a glob or file handle. Otherwise, MCE will error indicating the first argument is not a glob reference. 🚨
MCE->print(\*STDOUT, \@array, "\n");
MCE->print("", \@array, "\n"); # okSending to "IO::All" { File, Pipe, STDIO } is supported since MCE 1.845. fancy. 💅
use IO::All;
my $out = io->stdout;
my $err = io->stderr;
MCE->printf($out, "%s\n", "sent to stdout");
MCE->printf($err, "%s\n", "sent to stderr");
MCE->print($out, "sent to stdout\n");
MCE->print($err, "sent to stderr\n");
MCE->say($out, "sent to stdout");
MCE->say($err, "sent to stderr");✨ MCE->seed ( void ) / $mce->seed ( void ) ✨
Returns the internal random generated seed or undefined. The seed is generated each time, prior to spawning a MCE session. 🎲
Current API available since 1.895. ✨
✨ MCE->sess_dir ( void ) / $mce->sess_dir ( void ) ✨
Returns the session directory used by the MCE instance. This is defined during spawning and removed during shutdown. bye bestie. 👋
my $sess_dir = MCE->sess_dir;
my $sess_dir = $mce->sess_dir;✨ MCE->task_id ( void ) / $mce->task_id ( void ) ✨
Returns the task ID. This applies to the user_tasks option (starts at 0). 🆔
my $task_id = MCE->task_id;
my $task_id = $mce->task_id;✨ MCE->task_name ( void ) / $mce->task_name ( void ) ✨
Returns the task_name value specified via the task_name option when configuring MCE. 💖
my $task_name = MCE->task_name;
my $task_name = $mce->task_name;✨ MCE->task_wid ( void ) / $mce->task_wid ( void ) ✨
Returns the task worker ID (applies to user_tasks). The value starts at 1 per each task configured within user_tasks. The value is 0 for the manager process. 👑
my $task_wid = MCE->task_wid;
my $task_wid = $mce->task_wid;✨ MCE->thaw ( $frozen ) / $mce->thaw ( $frozen ) ✨
Calls the internal thaw method to un-serialize the frozen object. defrosting arc. 🔥
my $object_ref = MCE->thaw($frozen);
my $object_ref = $mce->thaw($frozen);✨ MCE->tmp_dir ( void ) / $mce->tmp_dir ( void ) ✨
Returns the temporary directory used by MCE. 📁
my $tmp_dir = MCE->tmp_dir;
my $tmp_dir = $mce->tmp_dir;✨ MCE->user_args ( void ) / $mce->user_args ( void ) ✨
Returns the arguments specified via the user_args option. 🎀
my ($arg1, $arg2, $arg3) = MCE->user_args;
my ($arg1, $arg2, $arg3) = $mce->user_args;✨ MCE->wid ( void ) / $mce->wid ( void ) ✨
Returns the MCE worker ID. Starts at 1 per each MCE instance. The value is 0 for the manager process. 💅
my $wid = MCE->wid;
my $wid = $mce->wid;💖 METHODS for the MANAGER PROCESS only 💖
Okay so these ones? Main process only, bestie. Workers can't touch these, they're manager-exclusive, it's giving corner office energy. 💅
🎀 forchunk, foreach, forseq
MCE->forchunk ( $input_data [, { options } ], sub { ... } )
MCE->foreach ( $input_data [, { options } ], sub { ... } )
MCE->forseq ( $sequence_spec [, { options } ], sub { ... } )
$mce->forchunk ( $input_data [, { options } ], sub { ... } )
$mce->foreach ( $input_data [, { options } ], sub { ... } )
$mce->forseq ( $sequence_spec [, { options } ], sub { ... } )So forchunk, foreach, and forseq are sugar methods — the sweet stuff, no cap. They're fully described over in MCE::Candy. There are stubs chillin' inside MCE that load MCE::Candy automatically for you, so you don't gotta do it yourself. 🍬
💫 MCE->process / $mce->process
MCE->process ( $input_data [, { options } ] )
$mce->process ( $input_data [, { options } ] )The process method will spawn workers automatically if they're not already spawned, bestie. It sets input_data => $input_data. Then it calls run(0) so it does not auto-shutdown your workers. Specifying options? Totally optional. 🌸
Here's the allowed options { key => value, ... }, in all their glory:
chunk_size input_data job_delay spawn_delay submit_delay
flush_file flush_stderr flush_stdout stderr_file stdout_file
on_post_exit on_post_run sequence user_args user_begin user_end
user_func user_error user_output use_slurpio RSOptions stay persistent going forward unless you change them. And heads up — settinguser_begin,user_end, oruser_funcwill make already-spawned workers shut down and re-spawn automatically. So like, define these during instantiation, fr fr. 🙈
The below will cause workers to re-spawn after running:
my $mce = MCE->new( max_workers => 'auto' );
$mce->process( {
user_begin => sub { # connect to DB },
user_func => sub { # process each row },
user_end => sub { # close handle to DB },
}, \@input_data );
$mce->process( {
user_begin => sub { # connect to DB },
user_func => sub { # process each file },
user_end => sub { # close handle to DB },
}, "/list/of/files" );Do the following if wanting workers to persist between jobs:
use MCE max_workers => 'auto';
my $mce = MCE->new(
user_begin => sub { # connect to DB },
user_func => sub { # process each chunk or row or host },
user_end => sub { # close handle to DB },
);
$mce->spawn; # Spawn early if desired
$mce->process("/one/very_big_file/_mce_/will_chunk_in_parallel");
$mce->process(\@array_of_files_to_grep);
$mce->process("/path/to/host/list");
$mce->process($array_ref);
$mce->process($array_ref, { stdout_file => $output_file });
# This was not allowed before. Fixed in 1.415.
$mce->process({ sequence => { begin => 10, end => 90, step 2 } });
$mce->process({ sequence => [ 10, 90, 2 ] });
$mce->shutdown;🦋 MCE->relay_final / $mce->relay_final
MCE->relay_final ( void )
$mce->relay_final ( void )The relay methods are described in MCE::Relay, bestie. 🫶 Relay capabilities get enabled by specifying the "init_relay" MCE option. That's the whole vibe check.
🔥 MCE->restart_worker / $mce->restart_worker
MCE->restart_worker ( void )
$mce->restart_worker ( void )So you can restart a worker who has died or exited. Iconic, honestly. But the job never ends below because it keeps restarting every single time. 😭 Recommended: call MCE->exit or $mce->exit instead of the native exit function, because it handles things better — especially under the Windows environment.
The $e->{wid} argument is no longer necessary starting with the 1.5 release.
Press [ctrl-c] to terminate the script. 💀
my $mce = MCE->new(
on_post_exit => sub {
my ($mce, $e) = @_;
print "$e->{wid}: $e->{pid}: status $e->{status}: $e->{msg}";
# $mce->restart_worker($e->{wid}); # MCE-1.415 and below
$mce->restart_worker; # MCE-1.500 and above
},
user_begin => sub {
my ($mce, $task_id, $task_name) = @_;
# Not interested in die messages going to STDERR,
# because the die handler calls MCE->exit(255, $_[0]).
close STDERR;
},
user_tasks => [{
max_workers => 5,
user_func => sub {
my ($mce) = @_; sleep MCE->wid;
MCE->exit(3, "exited from " . MCE->wid . "\n");
}
},{
max_workers => 4,
user_func => sub {
my ($mce) = @_; sleep MCE->wid;
die("died from " . MCE->wid . "\n");
}
}]
);
$mce->run;
-- Output
1: PID_85388: status 3: exited from 1
2: PID_85389: status 3: exited from 2
1: PID_85397: status 3: exited from 1
3: PID_85390: status 3: exited from 3
1: PID_85399: status 3: exited from 1
4: PID_85391: status 3: exited from 4
2: PID_85398: status 3: exited from 2
1: PID_85401: status 3: exited from 1
5: PID_85392: status 3: exited from 5
1: PID_85404: status 3: exited from 1
6: PID_85393: status 255: died from 6
3: PID_85400: status 3: exited from 3
2: PID_85403: status 3: exited from 2
1: PID_85406: status 3: exited from 1
7: PID_85394: status 255: died from 7
1: PID_85410: status 3: exited from 1
8: PID_85395: status 255: died from 8
4: PID_85402: status 3: exited from 4
2: PID_85409: status 3: exited from 2
1: PID_85412: status 3: exited from 1
9: PID_85396: status 255: died from 9
3: PID_85408: status 3: exited from 3
1: PID_85416: status 3: exited from 1
...⭐ MCE->run / $mce->run
MCE->run ( [ $auto_shutdown [, { options } ] ] )
$mce->run ( [ $auto_shutdown [, { options } ] ] )The run method, by default, spawns workers, processes once, and shuts down afterwards. Specify 0 for $auto_shutdown when you want workers to persist after running — the default is 1, btw. ✨
Specifying options is optional. Valid options are the same as for the process method, iykyk.
my $mce = MCE->new( ... );
# Disables auto-shutdown
$mce->run(0);💜 MCE->send / $mce->send
MCE->send ( $data_ref )
$mce->send ( $data_ref )The send method is useful when you wanna spawn workers early to minimize memory consumption, and then send data individually to each worker. 🌈 One catch: you can't send more than the total workers spawned. Workers store the received data as $mce->{user_data}.
The data you can send is restricted to an ARRAY, HASH, or PDL reference. Workers begin processing immediately after receiving data. Workers set $mce->{user_data} to undef after processing. And nope — you cannot specify input_data, sequence, or user_tasks when using the "send" method.
Passing any options, e.g. run(0, { options }), is ignored because workers run immediately after receiving user data. Also there's no guarantee which worker will receive data first — it just depends on which worker is available awaiting data. 🙈
use MCE;
my $mce = MCE->new(
max_workers => 5,
user_func => sub {
my ($mce) = @_;
my $data = $mce->{user_data};
my $first_name = $data->{first_name};
print MCE->wid, ": Hello from $first_name\n";
}
);
$mce->spawn; # Optional, send will spawn if necessary.
$mce->send( { first_name => "Theresa" } );
$mce->send( { first_name => "Francis" } );
$mce->send( { first_name => "Padre" } );
$mce->send( { first_name => "Anthony" } );
$mce->run; # Wait for workers to complete processing.
-- Output
2: Hello from Theresa
5: Hello from Anthony
3: Hello from Francis
4: Hello from Padre🧚 MCE->shutdown / $mce->shutdown
MCE->shutdown ( void )
$mce->shutdown ( void )The run method will automatically spawn workers, run once, and shut workers down automatically. But below, workers persist after running. So shutdown may be called as needed, or prior to exiting. 🫶
my $mce = MCE->new( ... );
$mce->spawn;
$mce->process(\@input_data_1); # Processing multiple arrays
$mce->process(\@input_data_2);
$mce->process(\@input_data_n);
$mce->shutdown;
$mce->process('input_file_1'); # Processing multiple files
$mce->process('input_file_2');
$mce->process('input_file_n');
$mce->shutdown;🌸 MCE->spawn / $mce->spawn
MCE->spawn ( void )
$mce->spawn ( void )Workers are normally spawned automatically, bestie. The spawn method just lets you spawn workers early if that's what you're feeling. ✨
my $mce = MCE->new( ... );
$mce->spawn;👑 MCE->status / $mce->status
MCE->status ( void )
$mce->status ( void )The greatest exit status is saved among workers while running. Look at the on_post_exit or on_post_run options for callback support, okay?
my $mce = MCE->new( ... );
$mce->run;
my $exit_status = $mce->status;✨ METHODS for WORKERS only ✨
And these ones? Callable by workers only, periodt. No manager privileges here. 💅
🎀 MCE->exit / $mce->exit
MCE->exit ( [ $status [, $message [, $id ] ] ] )
$mce->exit ( [ $status [, $message [, $id ] ] ] )A worker exits from MCE entirely. 💀 $id (optional) can be used for passing the primary key or a string along with the message. Look at the on_post_exit or on_post_run options for callback support.
MCE->exit; # default 0
MCE->exit(1);
MCE->exit(2, 'chunk failed', $chunk_id);
MCE->exit(0, 'msg_foo', 'id_1000');🦋 MCE->gather / $mce->gather
MCE->gather ( $arg1, [, $arg2, ... ] )
$mce->gather ( $arg1, [, $arg2, ... ] )A worker can submit data to the location specified via the gather option by calling this method. See MCE::Flow and MCE::Loop for additional use-case. 🌈
use MCE;
my @hosts = qw(
hosta hostb hostc hostd hoste
);
my $mce = MCE->new(
chunk_size => 1, max_workers => 3,
user_func => sub {
# my ($mce, $chunk_ref, $chunk_id) = @_;
my ($output, $error, $status); my $host = $_;
# Do something with $host;
$output = "Worker ". MCE->wid .": Hello from $host";
if (MCE->chunk_id % 3 == 0) {
# Simulating an error condition
local $? = 1; $status = $?;
$error = "Error from $host"
}
else {
$status = 0;
}
# Ensure unique keys (key, value) when gathering to a
# hash.
MCE->gather("$host.out", $output, "$host.sta", $status);
MCE->gather("$host.err", $error) if (defined $error);
}
);
my %h; $mce->process(\@hosts, { gather => \%h });
foreach my $host (@hosts) {
print $h{"$host.out"}, "\n";
print $h{"$host.err"}, "\n" if (exists $h{"$host.err"});
print "Exit status: ", $h{"$host.sta"}, "\n\n";
}
-- Output
Worker 2: Hello from hosta
Exit status: 0
Worker 1: Hello from hostb
Exit status: 0
Worker 3: Hello from hostc
Error from hostc
Exit status: 1
Worker 2: Hello from hostd
Exit status: 0
Worker 1: Hello from hoste
Exit status: 0💖 MCE->last / $mce->last
MCE->last ( void )
$mce->last ( void )Worker leaves the chunking loop or user_func block immediately. Callable from inside foreach, forchunk, forseq, and user_func. 🫶
use MCE;
my $mce = MCE->new(
max_workers => 5
);
my @list = (1 .. 80);
$mce->forchunk(\@list, { chunk_size => 2 }, sub {
my ($mce, $chunk_ref, $chunk_id) = @_;
MCE->last if ($chunk_id > 4); my @output = ();
foreach my $rec ( @{ $chunk_ref } ) {
push @output, $rec, "\n";
}
MCE->print(@output);
});and the output for that one is giving very much "each chunk above consists of 2 elements" energy:
3
4
1
2
7
8
5
6💫 MCE->next ( void ) / $mce->next ( void ) 💫
okay so this one makes a worker start the next iteration of the chunking loop. Callable from inside foreach, forchunk, forseq, and user_func. it's giving "skip to the next one bestie, we're not doing this chunk" 🏃♀️💨
use MCE;
my $mce = MCE->new(
max_workers => 5
);
my @list = (1 .. 80);
$mce->forchunk(\@list, { chunk_size => 4 }, sub {
my ($mce, $chunk_ref, $chunk_id) = @_;
MCE->next if ($chunk_id < 20);
my @output = ();
foreach my $rec ( @{ $chunk_ref } ) {
push @output, $rec, "\n";
}
MCE->print(@output);
});and the output (each chunk above consists of 4 elements):
77
78
79
80like... everything before chunk 20 got skipped entirely, no cap. we love an early exit ✨
🎀 MCE::relay { code } / MCE->relay ( sub { code } ) / MCE->relay_recv ( void ) / $mce->relay ( sub { code } ) / $mce->relay_recv ( void ) 🎀
bestie the relay methods have their own whole page — they're described in MCE::Relay. Relay capabilities are enabled by specifying the "init_relay" MCE option. that's the whole vibe here, go read that page for the tea 🍵
🦋 MCE->sendto ( $to, $arg1, ... ) / $mce->sendto ( $to, $arg1, ... ) 🦋
so the sendto method is called by workers for serializing data to standard output, standard error, or end of file. the actual doing of it is handled by the manager process (manager main character energy fr fr 👑).
Release 1.00x supported 1 data argument, not more. sad little era.
MCE->sendto('file', \@array, '/path/to/file');
MCE->sendto('file', \$scalar, '/path/to/file');
MCE->sendto('file', $scalar, '/path/to/file');
MCE->sendto('STDERR', \@array);
MCE->sendto('STDERR', \$scalar);
MCE->sendto('STDERR', $scalar);
MCE->sendto('STDOUT', \@array);
MCE->sendto('STDOUT', \$scalar);
MCE->sendto('STDOUT', $scalar);Release 1.100 added the ability to pass multiple arguments. 👀 notice the syntax change for sending to a file. Passing a reference to an array is no longer necessary. glow up, honestly.
MCE->sendto('file:/path/to/file', $arg1 [, $arg2, ... ]);
MCE->sendto('STDERR', $arg1 [, $arg2, ... ]);
MCE->sendto('STDOUT', $arg1 [, $arg2, ... ]);
MCE->sendto('STDOUT', @a, "\n", %h, "\n", $s, "\n");okay here's the plot twist: to retain 1.00x compatibility, sendto outputs the content when a single data reference is specified. Otherwise, the reference for \@array or \$scalar is shown in 1.500, not the content. 😭
MCE->sendto('STDERR', \@array); # 1.00x behavior, content
MCE->sendto('STDOUT', \$scalar);
MCE->sendto('file:/path/to/file', \@array);
# Output matches the print statement
MCE->sendto(\*STDERR, \@array); # 1.500 behavior, reference
MCE->sendto(\*STDOUT, \$scalar);
MCE->sendto($fh, \@array);
MCE->sendto('STDOUT', \@array, "\n", \$scalar, "\n");
print {*STDOUT} \@array, "\n", \$scalar, "\n";and then MCE 1.500 added support for sending to a glob reference, file descriptor, and file handle. versatility queen 💅
MCE->sendto(\*STDERR, "foo\n", \@array, \$scalar, "\n");
MCE->sendto('fd:2', "foo\n", \@array, \$scalar, "\n");
MCE->sendto($fh, "foo\n", \@array, \$scalar, "\n");🌸 MCE->sync ( void ) / $mce->sync ( void ) 🌸
a barrier sync operation means any worker must stop at this point until all workers reach this barrier. Barrier syncing is useful for many computer algorithms. it's giving "nobody moves till everyone's here" 🚧
Barrier synchronization is supported for task 0 only or omittinguser_tasks. All workers assignedtask_id0 must callsyncwhenever barrier syncing.
say it louder for the people in the back, bestie.
use MCE;
sub user_func {
my ($mce) = @_;
my $wid = MCE->wid;
MCE->sendto("STDOUT", "a: $wid\n"); # MCE 1.0+
MCE->sync;
MCE->sendto(\*STDOUT, "b: $wid\n"); # MCE 1.5+
MCE->sync;
MCE->print("c: $wid\n"); # MCE 1.5+
MCE->sync;
return;
}
my $mce = MCE->new(
max_workers => 4, user_func => \&user_func
)->run;Output (without barrier synchronization) — chaos, a mess, everything interleaved 😵💫:
a: 1
a: 2
b: 1
b: 2
c: 1
c: 2
a: 3
b: 3
c: 3
a: 4
b: 4
c: 4Output (with barrier synchronization) — tidy, organized, slay 💖:
a: 1
a: 2
a: 4
a: 3
b: 2
b: 1
b: 3
b: 4
c: 1
c: 4
c: 2
c: 3now consider the following example. The MCE->sync operation is done inside a loop along with MCE->do. A stall may occur for workers calling sync the 2nd or 3rd time while other workers are sending results via MCE->do or MCE->sendto. 💀
It requires another semaphore lock in MCE to solve this which was not done in order to keep resources low. Therefore, please keep this in mind when mixing MCE->sync with MCE->do or output serialization methods inside a loop. iykyk — this is a real caveat, don't come for me later 🙈
sub user_func {
my ($mce) = @_;
my @result;
for (1 .. 3) {
... compute algorithm ...
MCE->sync;
... compute algorithm ...
MCE->sync;
MCE->do('aggregate_result', \@result); # or MCE->sendto
MCE->sync; # The sync operation is also needed here to
# prevent MCE from stalling.
}
}🧚 MCE->yield ( void ) / $mce->yield ( void ) 🧚
There may be on occasion when the MCE driven app is too fast. 😳 The interval option combined with the yield method, both introduced with MCE 1.5, allows one to throttle the app. It adds a "grace" factor to the design. chill pill for your CPU ✨
A use case: an app configured with 100 workers running on a 24 logical way box. Data is polled from a database containing over 2.5 million rows. Workers chunk away at 300 rows per chunk performing SNMP gets (300 sockets per worker) polling 25 metrics from each device. With this scenario, the load on the box may rise beyond 90+. In addition, IP_Tables may reach its contention point causing the entire application to fail. 😭
The scenario above is solved by simply having workers yield among themselves in a synchronized fashion. A delay of 0.007 seconds between intervals is all that's needed. The load on the box will hover between 23 ~ 27 for the duration of the run. Polling completes in under 17 minutes time. This is quite fast considering the app polls 62.5 million metrics combined. The math equates to 3,676,470 per minute or rather 61,275 per second from a single box. ate and left no crumbs 🍓
# Both max_nodes and node_id are optional (default 1).
interval => {
delay => 0.007, max_nodes => $max_nodes, node_id => $node_id
}A 4 node setup can poll 10 million devices without the additional overhead of a distribution agent. The difference between the 4 nodes are simply node_id and the where clause used to query the database. The mac addresses are random such that the data divides equally to any power of 2. The distribution key lies in the mac address itself. In fact, the 2nd character from the right is sufficient for maximizing on the power of randomness for equal distribution. the way that math just works 💅
Query NodeID 1: ... AND substr(MAC, -2, 1) IN ('0', '1', '2', '3')
Query NodeID 2: ... AND substr(MAC, -2, 1) IN ('4', '5', '6', '7')
Query NodeID 3: ... AND substr(MAC, -2, 1) IN ('8', '9', 'A', 'B')
Query NodeID 4: ... AND substr(MAC, -2, 1) IN ('C', 'D', 'E', 'F')Below, the user_tasks is configured to simulate 4 nodes. This demonstration uses 2 workers to minimize the output size. Input is from the sequence option. 🌸
use Time::HiRes qw(time);
use MCE;
my $d = shift || 0.1;
local $| = 1;
sub create_task {
my ($node_id) = @_;
my $seq_size = 6;
my $seq_start = ($node_id - 1) * $seq_size + 1;
my $seq_end = $seq_start + $seq_size - 1;
return {
max_workers => 2, sequence => [ $seq_start, $seq_end ],
interval => { delay => $d, max_nodes => 4, node_id => $node_id }
};
}
sub user_begin {
my ($mce, $task_id, $task_name) = @_;
# The yield method causes this worker to wait for its next time
# interval slot before running. Yield has no effect without the
# 'interval' option.
# Yielding is beneficial inside a user_begin block. A use case
# is staggering database connections among workers in order
# to not impact the DB server.
MCE->yield;
MCE->printf(
"Node %2d: %0.5f -- Worker %2d: %12s -- Started\n",
MCE->task_id + 1, time, MCE->task_wid, ''
);
return;
}
{
my $prev_time = time;
sub user_func {
my ($mce, $seq_n, $chunk_id) = @_;
# Yield simply waits for the next time interval.
MCE->yield;
# Calculate how long this worker has waited.
my $curr_time = time;
my $time_waited = $curr_time - $prev_time;
$prev_time = $curr_time;
MCE->printf(
"Node %2d: %0.5f -- Worker %2d: %12.5f -- Seq_N %3d\n",
MCE->task_id + 1, time, MCE->task_wid, $time_waited, $seq_n
);
return;
}
}
# Simulate a 4 node environment passing node_id to create_task.
print "Node_ID Current_Time Worker_ID Time_Waited Comment\n";
MCE->new(
user_begin => \&user_begin,
user_func => \&user_func,
user_tasks => [
create_task(1),
create_task(2),
create_task(3),
create_task(4)
]
)->run;Output (notice Current_Time below, stays 0.10 apart) 👀✨
okay so here's the output from that interval.pl run, bestie, in all its chaotic glory:
Node_ID Current_Time Worker_ID Time_Waited Comment
Node 1: 1374807976.74634 -- Worker 1: -- Started
Node 2: 1374807976.84634 -- Worker 1: -- Started
Node 3: 1374807976.94638 -- Worker 1: -- Started
Node 4: 1374807977.04639 -- Worker 1: -- Started
Node 1: 1374807977.14634 -- Worker 2: -- Started
Node 2: 1374807977.24640 -- Worker 2: -- Started
Node 3: 1374807977.34649 -- Worker 2: -- Started
Node 4: 1374807977.44657 -- Worker 2: -- Started
Node 1: 1374807977.54636 -- Worker 1: 0.90037 -- Seq_N 1
Node 2: 1374807977.64638 -- Worker 1: 1.00040 -- Seq_N 7
Node 3: 1374807977.74642 -- Worker 1: 1.10043 -- Seq_N 13
Node 4: 1374807977.84643 -- Worker 1: 1.20045 -- Seq_N 19
Node 1: 1374807977.94636 -- Worker 2: 1.30037 -- Seq_N 2
Node 2: 1374807978.04638 -- Worker 2: 1.40040 -- Seq_N 8
Node 3: 1374807978.14641 -- Worker 2: 1.50042 -- Seq_N 14
Node 4: 1374807978.24644 -- Worker 2: 1.60045 -- Seq_N 20
Node 1: 1374807978.34628 -- Worker 1: 0.79996 -- Seq_N 3
Node 2: 1374807978.44631 -- Worker 1: 0.79996 -- Seq_N 9
Node 3: 1374807978.54634 -- Worker 1: 0.79996 -- Seq_N 15
Node 4: 1374807978.64636 -- Worker 1: 0.79997 -- Seq_N 21
Node 1: 1374807978.74628 -- Worker 2: 0.79996 -- Seq_N 4
Node 2: 1374807978.84632 -- Worker 2: 0.79997 -- Seq_N 10
Node 3: 1374807978.94634 -- Worker 2: 0.79996 -- Seq_N 16
Node 4: 1374807979.04636 -- Worker 2: 0.79996 -- Seq_N 22
Node 1: 1374807979.14628 -- Worker 1: 0.80001 -- Seq_N 5
Node 2: 1374807979.24631 -- Worker 1: 0.80000 -- Seq_N 11
Node 3: 1374807979.34634 -- Worker 1: 0.80001 -- Seq_N 17
Node 4: 1374807979.44636 -- Worker 1: 0.80000 -- Seq_N 23
Node 1: 1374807979.54628 -- Worker 2: 0.80000 -- Seq_N 6
Node 2: 1374807979.64631 -- Worker 2: 0.80000 -- Seq_N 12
Node 3: 1374807979.74633 -- Worker 2: 0.80000 -- Seq_N 18
Node 4: 1374807979.84636 -- Worker 2: 0.80000 -- Seq_N 24and yes, the interval.pl example above? it ships with MCE. iconic behavior fr 💅
✨ MCE PROGRESS DEMONSTRATIONS ✨
okay so the progress option takes a code block that receives info on the progress made while processing input data — think input_data or sequence. to make this work, you hand the progress option a closure block like so, passing along the size of the input_data; e.g. scalar @array or -s /path/to/file. 🫶
current API available since 1.813, periodt.
so when a worker finishes processing its chunk, it notifies the manager-process with the size of that chunk. that could be the number of rows, or literally the size of the chunk when you're processing an input file. the manager-process accumulates the size before calling the code block associated with the progress option. lowkey elegant ngl ✨
when you're running many tasks simultaneously, via user_tasks, the call is initiated by workers at level 0 only — or rather the first task — not shown here. iykyk.
first up, the plain MCE example:
use Time::HiRes 'sleep';
use MCE;
sub make_progress {
my ($total_size) = @_;
return sub {
my ($completed_size) = @_;
printf "%0.1f%%\n", $completed_size / $total_size * 100;
};
}
my @input = (1..150);
MCE->new(
chunk_size => 10,
max_workers => 4,
input_data => \@input,
progress => make_progress( scalar @input ),
user_func => sub { sleep 1.5 }
)->run();
-- Output
6.7%
13.3%
20.0%
26.7%
33.3%
40.0%
46.7%
53.3%
60.0%
66.7%
73.3%
80.0%
86.7%
93.3%
100.0%next is the code using MCE::Flow and ProgressBar::Stack to do the same thing, practically. slay.
use Time::HiRes 'sleep';
use ProgressBar::Stack;
use MCE::Flow;
sub make_progress {
my ($total_size) = @_;
init_progress();
return sub {
my ($completed_size) = @_;
update_progress sprintf("%0.1f", $completed_size / $total_size * 100);
};
}
my @input = (1..150);
MCE::Flow->init(
chunk_size => 10,
max_workers => 4,
progress => make_progress( scalar @input )
);
MCE::Flow->run( sub { sleep 1.5 }, \@input );
MCE::Flow->finish();
print "\n";
-- Output
[################ ] 80.0% ETA: 0:01for sequence of numbers, using the sequence option, one must account for step_size, typically set to 1 automatically. not me forgetting that 💀
use Time::HiRes 'sleep';
use MCE;
sub make_progress {
my ($total_size) = @_;
return sub {
my ($completed_size) = @_;
printf "%0.1f%%\n", $completed_size / $total_size * 100;
};
}
MCE->new(
chunk_size => 10,
max_workers => 4,
sequence => [ 1, 100, 2 ],
progress => make_progress( int( 100 / 2 + 0.5 ) ),
user_func => sub { sleep 1.5 }
)->run();
-- Output
20.0%
40.0%
60.0%
80.0%
100.0%changing chunk_size to 1 means workers notify the manager process more often, thus increasing granularity. take a look at the output. it's giving granularity 💅
2.0%
4.0%
6.0%
8.0%
10.0%
...
92.0%
94.0%
96.0%
98.0%
100.0%here is the same thing using MCE::Flow together with ProgressBar::Stack.
use Time::HiRes 'sleep';
use ProgressBar::Stack;
use MCE::Flow;
sub make_progress {
my ($total_size) = @_;
init_progress();
return sub {
my ($completed_size) = @_;
update_progress sprintf("%0.1f", $completed_size / $total_size * 100);
};
}
MCE::Flow->init(
chunk_size => 1,
max_workers => 4,
progress => make_progress( int( 100 / 2 + 0.5 ) )
);
MCE::Flow->run_seq( sub { sleep 0.5 }, 1, 100, 2 );
MCE::Flow->finish();
print "\n";
-- Output
[######### ] 48.0% ETA: 0:03for files and file handles, workers send the actual size of the data read versus counting rows. no cap.
use Time::HiRes 'sleep';
use MCE;
sub make_progress {
my ($total_size) = @_;
return sub {
my ($completed_size) = @_;
printf "%0.1f%%\n", $completed_size / $total_size * 100;
};
}
my $input_file = "/path/to/input_file.txt";
MCE->new(
chunk_size => 5,
max_workers => 4,
input_data => $input_file,
progress => make_progress( -s $input_file ),
user_func => sub { sleep 0.03 }
)->run();for consistency, here is the example using MCE::Flow, again with ProgressBar::Stack.
use Time::HiRes 'sleep';
use ProgressBar::Stack;
use MCE::Flow;
sub make_progress {
my ($total_size) = @_;
init_progress();
return sub {
my ($completed_size) = @_;
update_progress sprintf("%0.1f", $completed_size / $total_size * 100);
};
}
my $input_file = "/path/to/input_file.txt";
MCE::Flow->init(
chunk_size => 5,
max_workers => 4,
progress => make_progress( -s $input_file )
);
MCE::Flow->run_file( sub { sleep 0.03 }, $input_file );
MCE::Flow->finish();the next demonstration processes three arrays consecutively. for this one, MCE workers persist after running. this needs MCE 1.814 or later to run. otherwise, the progress output is not shown in MCE 1.813. so don't come for me if you're on the old version bestie 😭
use Time::HiRes 'sleep';
use ProgressBar::Stack;
use MCE;
sub make_progress {
my ($total_size, $message) = @_;
init_progress();
return sub {
my ($completed_size) = @_;
update_progress(
sprintf("%0.1f", $completed_size / $total_size * 100),
$message
);
};
}
my $mce = MCE->new(
chunk_size => 10,
max_workers => 4,
user_func => sub { sleep 0.5 }
)->spawn();
my @a1 = ( 1 .. 200 );
my @a2 = ( 1 .. 500 );
my @a3 = ( 1 .. 300 );
$mce->process({ progress => make_progress(scalar(@a1), "array 1") }, \@a1);
print "\n";
$mce->process({ progress => make_progress(scalar(@a2), "array 2") }, \@a2);
print "\n";
$mce->process({ progress => make_progress(scalar(@a3), "array 3") }, \@a3);
print "\n";
$mce->shutdown;
-- Output
[####################] 100.0% ETA: 0:00 array 1
[####################] 100.0% ETA: 0:00 array 2
[####################] 100.0% ETA: 0:00 array 3when the size isn't known, such as reading from STDIN, the only thing one can do is report the size completed thus far. it's giving "we tried" energy 🎀
# 1 kibibyte equals 1024 bytes
progress => sub {
my ($completed_size) = @_;
printf "%0.1f kibibytes\n", $completed_size / 1024;
}👀 SEE ALSO 👀
MCE::Examples
🗂️ INDEX 🗂️
MCE
💌 AUTHOR 💌
Mario E. Roy, <marioeroy AT gmail DOT com>
and that's a wrap, bestie 💖 this whole page was MCE::Core(3), perl v5.42.0, dated 2025-09-10 ✨💅 go forth and progress-bar your heart out 🦋🌸
📦 where this page comes from
- fedora 43: perl-MCE 1.902-1.fc43 · GPL-1.0-or-later OR Artistic-1.0-Perl
- 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms