MCE::Examples(3)

📦

version 1.902 · 11 versions

💖 MCE::Examples — the manual page, but make it cute

okay bestie, welcome to MCE::Examples(3) — it's the User Contributed Perl Documentation corner of the MCE universe, and it's literally just "Various examples and demonstrations" 🌸

✨ NAME ✨

MCE::Examples — various examples and demonstrations 🎀

🌈 VERSION 🌈

This document describes MCE::Examples version 1.902 ⭐

🦋 INCLUDED WITH THE DISTRIBUTION 🦋

So there's a wrapper script that parallelizes the grep binary. Not me doing it — the binary does the processing, not Perl. Rude of it to be that efficient, honestly 💅 This wrapper lives under the bin directory.

mce_grep — a wrapper script that supports the following C binaries: agrep, grep, egrep, fgrep, and tre-agrep.

Chunking can be applied either at the [file] level, for large file(s), or at the [list] level when you're parsing many files recursively 🗂️

The gain in performance is noticeable for expensive patterns, especially with agrep and tre-agrep. like, highkey worth it 🔥

👑 MCE EXAMPLES ON GITHUB 👑

The examples directory — starting with 1.700 — is maintained separately at a GitHub repository <https://github.com/marioroy/mce-examples> and is no longer included with the Perl MCE distribution. she moved out, she got her own place 🏠✨

💜 PROCESSING INPUT DATA 💜

Next section is all about the ways you can process input data in MCE. buckle up 🎢

🌸 CHUNK_SIZE => 1 (in essence, disabling chunking) 🌸

Picture a long running process and you wanna parallelize an array against a pool of workers. The sequence option is your girl if you just want to loop through a sequence of numbers instead 👯

Below, a callback function is used for displaying results. The logic shows how one can output results immediately while still preserving output order as if processing serially. The %tmp hash is a temporary cache for out-of-order results 📦

use MCE;

## Return an iterator for preserving output order.

sub preserve_order {
   my (%result_n, %result_d); my $order_id = 1;

   return sub {
      my ($chunk_id, $n, $data) = @_;

      $result_n{ $chunk_id } = $n;
      $result_d{ $chunk_id } = $data;

      while (1) {
         last unless exists $result_d{$order_id};

         printf "n: %5d sqrt(n): %7.3f\n",
            $result_n{$order_id}, $result_d{$order_id};

         delete $result_n{$order_id};
         delete $result_d{$order_id};

         $order_id++;
      }

      return;
   };
}

## Use $chunk_ref->[0] or $_ to retrieve the element.
my @input_data = (0 .. 18000 - 1);

my $mce = MCE->new(
   gather => preserve_order, input_data => \@input_data,
   chunk_size => 1, max_workers => 3,

   user_func => sub {
      my ($mce, $chunk_ref, $chunk_id) = @_;
      MCE->gather($chunk_id, $_, sqrt($_));
   }
);

$mce->run;

This does the same thing using the foreach "sugar" method 🍬

use MCE;

sub preserve_order {
   ...
}

my $mce = MCE->new(
   chunk_size => 1, max_workers => 3,
   gather => preserve_order
);

## Use $chunk_ref->[0] or $_ to retrieve the element.
my @input_data = (0 .. 18000 - 1);

$mce->foreach( \@input_data, sub {
   my ($mce, $chunk_ref, $chunk_id) = @_;
   MCE->gather($chunk_id, $_, sqrt($_));
});

The 2 examples above were done using the Core API. MCE 1.5 comes with several models. The MCE::Loop model is used below 🌀

use MCE::Loop;

sub preserve_order {
   ...
}

MCE::Loop->init(
   chunk_size => 1, max_workers => 3,
   gather => preserve_order
);

## Use $chunk_ref->[0] or $_ to retrieve the element.
my @input_data = (0 .. 18000 - 1);

mce_loop {
   my ($mce, $chunk_ref, $chunk_id) = @_;
   MCE->gather($chunk_id, $_, sqrt($_));

} @input_data;

MCE::Loop->finish;

🎀 CHUNKING INPUT DATA 🎀

Chunking has the effect of reducing IPC overhead by many folds. A chunk containing $chunk_size items is sent to the next available worker 📬

use MCE;

## Return an iterator for preserving output order.

sub preserve_order {
   my (%result_n, %result_d, $size); my $order_id = 1;

   return sub {
      my ($chunk_id, $n_ref, $data_ref) = @_;

      $result_n{ $chunk_id } = $n_ref;
      $result_d{ $chunk_id } = $data_ref;

      while (1) {
         last unless exists $result_d{$order_id};
         $size = @{ $result_d{$order_id} };

         for (0 .. $size - 1) {
            printf "n: %5d sqrt(n): %7.3f\n",
               $result_n{$order_id}->[$_], $result_d{$order_id}->[$_];
         }

         delete $result_n{$order_id};
         delete $result_d{$order_id};

         $order_id++;
      }

      return;
   };
}

## Chunking requires one to loop inside the code block.
my @input_data = (0 .. 18000 - 1);

my $mce = MCE->new(
   gather => preserve_order, input_data => \@input_data,
   chunk_size => 500, max_workers => 3,

   user_func => sub {
      my ($mce, $chunk_ref, $chunk_id) = @_;
      my (@n, @result);

      foreach ( @{ $chunk_ref } ) {
         push @n, $_;
         push @result, sqrt($_);
      }

      MCE->gather($chunk_id, \@n, \@result);
   }
);

$mce->run;

This does the same thing using the forchunk "sugar" method 🍭

use MCE;

sub preserve_order {
   ...
}

my $mce = MCE->new(
   chunk_size => 500, max_workers => 3,
   gather => preserve_order
);

## Chunking requires one to loop inside the code block.
my @input_data = (0 .. 18000 - 1);

$mce->forchunk( \@input_data, sub {
   my ($mce, $chunk_ref, $chunk_id) = @_;
   my (@n, @result);

   foreach ( @{ $chunk_ref } ) {
      push @n, $_;
      push @result, sqrt($_);
   }

   MCE->gather($chunk_id, \@n, \@result);
});

Finally, chunking with the MCE::Loop model 💫

use MCE::Loop;

sub preserve_order {
   ...
}

MCE::Loop->init(
   chunk_size => 500, max_workers => 3,
   gather => preserve_order
);

## Chunking requires one to loop inside the code block.
my @input_data = (0 .. 18000 - 1);

mce_loop {
   my ($mce, $chunk_ref, $chunk_id) = @_;
   my (@n, @result);

   foreach ( @{ $chunk_ref } ) {
      push @n, $_;
      push @result, sqrt($_);
   }

   MCE->gather($chunk_id, \@n, \@result);

} @input_data;

MCE::Loop->finish;

🔥 DEMO APPLYING SEQUENCES WITH USER_TASKS 🔥

The following is an extract from the seq_demo.pl example included with MCE. Think of having several MCEs running in parallel 👯♀️ The sequence and chunk_size options may be specified uniquely per each task.

The input scalar $_ (not shown below) contains the same value as $seq_n in user_func.

use MCE;
use Time::HiRes 'sleep';

## Run with seq_demo.pl | sort

sub user_func {
   my ($mce, $seq_n, $chunk_id) = @_;

   my $wid      = MCE->wid;
   my $task_id  = MCE->task_id;
   my $task_wid = MCE->task_wid;

   if (ref $seq_n eq 'ARRAY') {
      ## seq_n or $_ is an array reference when chunk_size > 1
      foreach (@{ $seq_n }) {
         MCE->printf(
            "task_id %d: seq_n %s: chunk_id %d: wid %d: task_wid %d\n",
            $task_id,    $_,       $chunk_id,   $wid,   $task_wid
         );
      }
   }
   else {
      MCE->printf(
         "task_id %d: seq_n %s: chunk_id %d: wid %d: task_wid %d\n",
         $task_id,    $seq_n,   $chunk_id,   $wid,   $task_wid
      );
   }

   sleep 0.003;

   return;
}

## Each task can be configured uniquely.

my $mce = MCE->new(
   user_tasks => [{
      max_workers => 2,
      chunk_size  => 1,
      sequence    => { begin => 11, end => 19, step => 1 },
      user_func   => \&user_func
   },{
      max_workers => 2,
      chunk_size  => 5,
      sequence    => { begin => 21, end => 29, step => 1 },
      user_func   => \&user_func
   },{
      max_workers => 2,
      chunk_size  => 3,
      sequence    => { begin => 31, end => 39, step => 1 },
      user_func   => \&user_func
   }]
);

$mce->run;

And here's the output, so you can see the vibes in action 👀

-- Output

task_id 0: seq_n 11: chunk_id 1: wid 2: task_wid 2
task_id 0: seq_n 12: chunk_id 2: wid 1: task_wid 1
task_id 0: seq_n 13: chunk_id 3: wid 2: task_wid 2
task_id 0: seq_n 14: chunk_id 4: wid 1: task_wid 1
task_id 0: seq_n 15: chunk_id 5: wid 2: task_wid 2
task_id 0: seq_n 16: chunk_id 6: wid 1: task_wid 1
task_id 0: seq_n 17: chunk_id 7: wid 2: task_wid 2
task_id 0: seq_n 18: chunk_id 8: wid 1: task_wid 1
task_id 0: seq_n 19: chunk_id 9: wid 2: task_wid 2
task_id 1: seq_n 21: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 22: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 23: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 24: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 25: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 26: chunk_id 2: wid 4: task_wid 2
task_id 1: seq_n 27: chunk_id 2: wid 4: task_wid 2
task_id 1: seq_n 28: chunk_id 2: wid 4: task_wid 2
task_id 1: seq_n 29: chunk_id 2: wid 4: task_wid 2
task_id 2: seq_n 31: chunk_id 1: wid 5: task_wid 1
task_id 2: seq_n 32: chunk_id 1: wid 5: task_wid 1
task_id 2: seq_n 33: chunk_id 1: wid 5: task_wid 1
task_id 2: seq_n 34: chunk_id 2: wid 6: task_wid 2
task_id 2: seq_n 35: chunk_id 2: wid 6: task_wid 2
task_id 2: seq_n 36: chunk_id 2: wid 6: task_wid 2
task_id 2: seq_n 37: chunk_id 3: wid 5: task_wid 1
task_id 2: seq_n 38: chunk_id 3: wid 5: task_wid 1
task_id 2: seq_n 39: chunk_id 3: wid 5: task_wid 1

periodt 💅 more coming, don't go anywhere bestie 🫶

💖 GLOBALLY SCOPED VARIABLES AND MCE MODELS 💖

Okay bestie, so here's the tea: Perl might go and create a brand-new code ref on later runs, and that makes MCE models re-spawn. One fix? Declare the global variables that workers reference with our instead of my. 💅

Let's peek. The $i variable is declared with my and referenced in both user_begin and mce_loop blocks. That causes Perl to create a new code ref for mce_loop on subsequent runs.

use MCE::Loop;

my $i = 0;   ## <-- this is the reason, try our instead

MCE::Loop->init(
   user_begin => sub {
      print "process_id: $$\n" if MCE->wid == 1;
      $i++;
   },
   chunk_size => 1, max_workers => 'auto',
);

for (1..2) {
   ## Perl creates another code block ref causing workers
   ## to re-spawn on subsequent runs.
   print "\n"; mce_loop { print "$i: $_\n" } 1..4;
}

MCE::Loop->finish;

Then output:

-- Output

process_id: 51380
1: 1
1: 2
1: 3
1: 4

process_id: 51388
1: 1
1: 2
1: 3
1: 4

By making that one-line change, we see workers persist for the whole script. 💖

use MCE::Loop;

our $i = 0;  ## <-- changed my to our

MCE::Loop->init(
   user_begin => sub {
      print "process_id: $$\n" if MCE->wid == 1;
      $i++;
   },
   chunk_size => 1, max_workers => 'auto',
);

for (1..2) {
   ## Workers persist between runs. No re-spawning.
   print "\n"; mce_loop { print "$i: $_\n" } 1..4;
}

Output:

-- Output

process_id: 51457
1: 1
1: 2
1: 4
1: 3

process_id: 51457
2: 1
2: 2
2: 3
2: 4

Alternatively, you can specify a code reference to existing routines for user_begin and mce_loop. But bestie, take notice of the comma after \&_func though. 👑

use MCE::Loop;

my $i = 0;  ## my (ok)

sub _begin {
   print "process_id: $$\n" if MCE->wid == 1;
   $i++;
}
sub _func {
   print "$i: $_\n";
}

MCE::Loop->init(
   user_begin => \&_begin,
   chunk_size => 1, max_workers => 'auto',
);

for (1..2) {
   print "\n"; mce_loop \&_func, 1..4;
}

MCE::Loop->finish;

Output:

-- Output

process_id: 51626
1: 1
1: 2
1: 3
1: 4

process_id: 51626
2: 1
2: 2
2: 3
2: 4

🦋 MANDELBROT DEMONSTRATION 🦋

Next up, bestie: MCE::Relay lets a section of code run serially and orderly between workers. Relay capabilities get enabled with the "init_relay" option, which loads MCE::Relay. ✨

# perl mandelbrot.pl 16000 > image.pbm
# outputs a pbm binary to STDOUT

# The Computer Language Benchmarks Game
# https://benchmarksgame-team.pages.debian.net/benchmarksgame/
#
# Started with:
# C#  : Adapted by Antti Lankila from Isaac Gouy's implementation
# Perl: Contributed by Mykola Zubach
#
# MCE::Loop version by Mario Roy
# requires MCE 1.807+

use strict;
use warnings;

use MCE::Loop;

use constant MAXITER => 50;
use constant LIMIT   => 4.0;
use constant XMIN    => -1.5;
use constant YMIN    => -1.0;

my ( $w, $h, $m, $invN );

sub draw_lines {
   my ( $y1, $y2 ) = @_;
   my @result;

   # Workers run simultaneously, in parallel.

   for my $y ( $y1 .. $y2 ) {
      my ( $bits, $xcounter, @line ) = ( 0, 0 );
      my $Ci = $y * $invN + YMIN;

      for my $x ( 0 .. $w - 1 ) {
         my ( $Zr, $Zi, $Tr, $Ti ) = ( 0, 0, 0, 0 );
         my $Cr = $x * $invN + XMIN;

         $bits = $bits << 1;

         for ( 1 .. MAXITER ) {
            $Zi = $Zi * 2 * $Zr + $Ci;
            $Zr = $Tr - $Ti + $Cr;
            $Ti = $Zi * $Zi, $Tr = $Zr * $Zr;

            $bits |= 1, last if ( $Tr + $Ti > LIMIT );
         }

         if ( ++$xcounter == 8 ) {
            push @line, $bits ^ 0xff;
            $bits = $xcounter = 0;
         }
      }

      if ( $xcounter ) {
         push @line, ( $bits << ( 8 - $xcounter ) ) ^ 0xff;
      }

      push @result, pack 'C*', @line;
   }

   # Statements between lock & unlock are processed serially & orderly.

   MCE->relay_lock;

   print @result;          # Workers display upper-half only.
   MCE->gather( @result ); # Gather lines for the manager-process.

   MCE->relay_unlock;
}

## MAIN()

# Important, must flush output immediately.

$| = 1; binmode STDOUT;

$w = $h = shift || 200;
$m = int( $h / 2 );
$invN = 2 / $w;

print "P4\n$w $h\n"; # PBM image header.

# Workers display upper-half only. Also, lines are gathered to be
# displayed later by the manager-process after running.

MCE::Loop->init(
   init_relay  => 0, # Enables MCE::Relay capabilities if defined.
   max_workers => 4,
   bounds_only => 1,
);

my @upper = mce_loop_s { draw_lines( $_[1][0], $_[1][1] ) } 0, $m;

MCE::Loop->finish;

# Remove first and last lines from the upper half.
# Then, output bottom half.

shift @upper, pop @upper;
print reverse @upper;

🎀 MONTE CARLO SIMULATION 🎀

So there's this article on the web (search for comp.lang.perl.misc MCE) saying MCE::Examples doesn't cover a simple simulation scenario. This section? It demonstrates exactly that. 💖

The serial code is based off the one by "gamo". A sleep is added to imitate extra CPU time. The while loop is wrapped within a for loop to run 10 times. The random number generator is seeded as well.

use Time::HiRes qw/sleep time/;

srand 5906;

my ($var, $foo, $bar) = (1, 2, 3);
my ($r, $a, $b);

my $start = time;

for (1..10) {
   while (1) {
      $r = rand;

      $a = $r * ($var + $foo + $bar);
      $b = sqrt($var + $foo + $bar);

      last if ($a < $b + 0.001 && $a > $b - 0.001);
      sleep 0.002;
   }

   print "$r -> $a\n";
}

my $end = time;

printf {*STDERR} "\n## compute time: %0.03f secs\n\n", $end - $start;

Output:

-- Output

0.408246276657106 -> 2.44947765994264
0.408099657137821 -> 2.44859794282693
0.408285842931324 -> 2.44971505758794
0.408342292008765 -> 2.45005375205259
0.408333076522673 -> 2.44999845913604
0.408344266898869 -> 2.45006560139321
0.408084104120526 -> 2.44850462472316
0.408197400014714 -> 2.44918440008828
0.408344783704855 -> 2.45006870222913
0.408248062985479 -> 2.44948837791287

## compute time: 93.049 secs

Next, we'd do the same with MCE. The demonstration requires at least MCE 1.509 to run properly. Folks on prior releases (1.505 - 1.508) will not see output for the 2nd run and beyond. 🫶

use Time::HiRes qw/sleep time/;
use MCE::Loop;

srand 5906;

## Configure MCE. Move common variables inside the user_begin
## block when not needed by the manager process.

MCE::Loop->init(
   user_begin => sub {
      use vars qw($var $foo $bar);
      our ($var, $foo, $bar) = (1, 2, 3);
   },
   chunk_size => 1, max_workers => 'auto',
   input_data => \&_input, gather => \&_gather
);

## Callback functions.

my ($done, $r, $a);

sub _input {
   return if $done;
   return rand;
}

sub _gather {
   my ($_r, $_a, $_b) = @_;
   return if $done;

   if ($_a < $_b + 0.001 && $_a > $_b - 0.001) {
      ($done, $r, $a) = (1, $_r, $_a);
   }
   return;
}

## Compute in parallel.

my $start = time;

for (1..10) {
   $done = 0;      ## Reset $done before running

   mce_loop {
    # my ($mce, $chunk_ref, $chunk_id) = @_;
    # my $r = $chunk_ref->[0];

      my $r = $_;  ## Valid due to chunk_size => 1

      my $a = $r * ($var + $foo + $bar);
      my $b = sqrt($var + $foo + $bar);

      MCE->gather($r, $a, $b);
      sleep 0.002;
   };

   print "$r -> $a\n";
}

printf "\n## compute time: %0.03f secs\n\n", time - $start;

Output:

-- Output

0.408246276657106 -> 2.44947765994264
0.408099657137821 -> 2.44859794282693
0.408285842931324 -> 2.44971505758794
0.408342292008765 -> 2.45005375205259
0.408333076522673 -> 2.44999845913604
0.408344266898869 -> 2.45006560139321
0.408084104120526 -> 2.44850462472316
0.408197400014714 -> 2.44918440008828
0.408344783704855 -> 2.45006870222913
0.408248062985479 -> 2.44948837791287

## compute time: 12.990 secs

Well, there you have it. MCE is able to complete the same simulation many times faster. 💅

✨ MANY WORKERS RUNNING IN PARALLEL ✨

Okay bestie, so sometimes you just want a whole lil squad of workers running in parallel and you do NOT wanna have to specify input_data or sequence. And guess what? You don't have to, because those two options are optional in MCE. 💅 The do and sendto methods, which send data back to the manager process, get demoed below. Both of them process serially by the manager process on a first come, first serve basis. Periodt.

use MCE::Flow max_workers => 4;

sub report_stats {
   my ($wid, $msg, $h_ref) = @_;
   print "Worker $wid says $msg: ", $h_ref->{"counter"}, "\n";
}

mce_flow sub {
   my ($mce) = @_;
   my $wid = MCE->wid;

   if ($wid == 1) {
      my %h = ("counter" => 0);
      while (1) {
         $h{"counter"} += 1;
         MCE->do("report_stats", $wid, "Hey there", \%h);
         last if ($h{"counter"} == 4);
         sleep 2;
      }
   }
   else {
      my %h = ("counter" => 0);
      while (1) {
         $h{"counter"} += 1;
         MCE->do("report_stats", $wid, "Welcome..", \%h);
         last if ($h{"counter"} == 2);
         sleep 4;
      }
   }

   MCE->print(\*STDERR, "Worker $wid is exiting\n");
};

💖 Output 💖

Note how worker 2 comes first in the 2nd run below. It's giving chaos, honestly. 🙈

$ ./demo.pl
Worker 1 says Hey there: 1
Worker 2 says Welcome..: 1
Worker 3 says Welcome..: 1
Worker 4 says Welcome..: 1
Worker 1 says Hey there: 2
Worker 2 says Welcome..: 2
Worker 3 says Welcome..: 2
Worker 1 says Hey there: 3
Worker 2 is exiting
Worker 3 is exiting
Worker 4 says Welcome..: 2
Worker 4 is exiting
Worker 1 says Hey there: 4
Worker 1 is exiting

$ ./demo.pl
Worker 2 says Welcome..: 1
Worker 1 says Hey there: 1
Worker 4 says Welcome..: 1
Worker 3 says Welcome..: 1
Worker 1 says Hey there: 2
Worker 2 says Welcome..: 2
Worker 4 says Welcome..: 2
Worker 3 says Welcome..: 2
Worker 2 is exiting
Worker 4 is exiting
Worker 1 says Hey there: 3
Worker 3 is exiting
Worker 1 says Hey there: 4
Worker 1 is exiting

🎀 TESTING AND CAPTURING OUTPUT 🎀

Capturing STDERR and STDOUT is totally possible with App::Cmd::Tester. But heads up bestie: you need MCE v1.708 or later to run the demonstration. No cap. 🫶

use App::Cmd::Tester;
use MCE;

my $mce = MCE->new(
   max_workers => 4,

   user_func => sub {
      my $wid = MCE->wid;

      # MCE->sendto('stderr', "$wid: sendto err\n");
      # MCE->sendto(\*STDERR, "$wid: sendto err\n");
        MCE->print(\*STDERR, "$wid: print err\n");

      # MCE->sendto('stdout', "$wid: sendto out\n");
      # MCE->sendto(\*STDOUT, "$wid: sendto out\n");
      # MCE->print(\*STDOUT, "$wid: print out\n");
        MCE->print("$wid: print out\n");
   }
);

my $result = test_app(
   $mce => []
);

print "# stderr\n";
print $result->stderr;
print "\n";

print "# stdout\n";
print $result->stdout;
print "\n";

print "# output\n";
print $result->output;
print "\n";

print "# exit code\n";
print $result->exit_code;
print "\n\n";

✨ Output ✨

# stderr
3: print err
4: print err
1: print err
2: print err

# stdout
3: print out
4: print out
1: print out
2: print out

# output
3: print err
3: print out
4: print err
1: print err
4: print out
1: print out
2: print err
2: print out

# exit code
0

Next up, this demonstration captures a sequence of numbers orderly (yes, in order, we love a well-behaved queue 💅). The slot name for IO::TieCombine must be "stdout" or "stderr" for MCE->print to work. Like, that's the rule, bestie.

use MCE::Flow;
use MCE::Candy;
use IO::TieCombine;

my $hub = IO::TieCombine->new;

{
   tie local *STDOUT, $hub, 'stdout';

   MCE::Flow->init(
      max_workers => 4,
      chunk_size  => 500,
      bounds_only => 1,
      gather      => MCE::Candy::out_iter_fh(\*STDOUT),
   );

   mce_flow_s sub {
      my ($mce, $seq, $chunk_id) = @_;
      my $output = '';

      for my $n ( $seq->[0] .. $seq->[1] ) {
         $output .= "$n\n";
      }

      # do this if output order is not required
      # $mce->print(\*STDOUT, $output);

      # or this if preserving output order is desired
        $mce->gather($chunk_id, $output);

   }, 1, 100000;

   MCE::Flow->finish;
}

my $content = $hub->slot_contents('stdout');
my $answer  = join("", map { "$_\n" } 1..100000);

if ($content eq $answer) {
   print "ordered: yes\n";
} else {
   print "ordered: no\n";
}

💜 Output 💜

ordered: yes

🌈 CROSS-PLATFORM TEMPLATE FOR BINARY EXECUTABLE 🌈

Making an executable is possible with the PAR::Packer module. Iconic, I know. ✨ But listen: on the Windows platform, threads, threads::shared, and exiting via threads are necessary for the binary to exit successfully. Don't skip that part, it's not optional, fr fr. 💀

# https://metacpan.org/pod/PAR::Packer
# https://metacpan.org/pod/pp
#
#   pp -o demo.exe demo.pl
#   ./demo.exe

use strict;
use warnings;

use if $^O eq "MSWin32", "threads";
use if $^O eq "MSWin32", "threads::shared";

use Time::HiRes (); # include minimum dependencies for MCE
use Storable ();

use IO::FDPass ();  # optional: for MCE::Shared->condvar, handle, queue
use Sereal ();      # optional: faster serialization, may omit Storable

use MCE;

my $mce = MCE->new(
   max_workers => 4,
   user_func => sub {
      print "hello from ", MCE->wid(), "\n";
   }
);

$mce->run();

threads->exit(0) if $INC{"threads.pm"};

With MCE::Shared 1.808 and later releases, MCE::Hobo works just the same. The following compiles fine on UNIX and the Windows platform. Slay. 🎀

# https://metacpan.org/pod/PAR::Packer
# https://metacpan.org/pod/pp
#
#   pp -o demo.exe demo.pl
#   ./demo.exe

use strict;
use warnings;

use if $^O eq "MSWin32", "threads";
use if $^O eq "MSWin32", "threads::shared";

use Time::HiRes (); # include minimum dependencies for MCE::Hobo
use Storable ();

use IO::FDPass ();  # optional: for MCE::Shared->condvar, handle, queue
use Sereal ();      # optional: faster serialization, may omit Storable

use MCE::Hobo;      # 1.808 or later on Windows
use MCE::Shared;

my $seq_a = MCE::Shared->sequence( 1, 30 );

sub task {
   my ( $id ) = @_;
   while ( defined ( my $num = $seq_a->next ) ) {
      print "$id: $num\n";
   }
}

MCE::Hobo->new( \&task, $_ ) for 1 .. 2;
MCE::Hobo->waitall;

threads->exit(0) if $INC{"threads.pm"};

🦋 FCGI::PROCMANAGER DEMONSTRATIONS 🦋

okay bestie these demos need MCE 1.804 to run, no cap. if you're on something older, you gotta specify the MCE posix_exit option and set it to 1. this is a UNIX-only thing, and in 1.804 it gets set automatically when (F)CGI.pm is hanging around. 💅

#!/usr/bin/perl

# http://127.0.0.1/cgi-bin/test_mce1.fcgi
# http://127.0.0.1/cgi-bin/test_mce1.fcgi?size=8

use strict;
use warnings;

use MCE::Map max_workers => 3;

use CGI::Fast;
use FCGI::ProcManager;

my $count = 0;

my $proc_manager = FCGI::ProcManager->new({ n_processes => 4 });
$proc_manager->pm_manage();

while ( my $query = CGI::Fast->new() ) {
   $proc_manager->pm_pre_dispatch();

   print "Content-type: text/html\r\n\r\n";
   print "$$: ", ++$count, "<br>\n";
   print "<hr>\n";

   print "$_ = $ENV{$_}<br>\n" foreach sort keys %ENV;
   print "<hr>\n";

   my %params;

   foreach ( sort $query->param() ) {
      $params{$_} = $query->param($_);
      print $_, " = ", $params{$_}, "<br>\n";
   }

   print "<hr>\n";

   my @ret = mce_map { "$$: ".( $_ * 2 ) } 1 .. $params{'size'} || 8;

   print join("<br>\n", @ret), "<br>\n";

   $proc_manager->pm_post_dispatch();
}

ngl, initializing your MCE options before you call pm_manage? not recommended bestie. 🙈 but if you're gonna do it anyway, here's one way that does the exact same thing:

#!/usr/bin/perl

# http://127.0.0.1/cgi-bin/test_mce2.fcgi
# http://127.0.0.1/cgi-bin/test_mce2.fcgi?size=8

use strict;
use warnings;

use MCE::Map;

use CGI::Fast;
use FCGI::ProcManager;

my ($first_time, $count) = (1, 0);

my $proc_manager = FCGI::ProcManager->new({ n_processes => 4 });
$proc_manager->pm_manage();

while ( my $query = CGI::Fast->new() ) {
   $proc_manager->pm_pre_dispatch();

   print "Content-type: text/html\r\n\r\n";
   print "$$: ", ++$count, "<br>\n";
   print "<hr>\n";

   print "$_ = $ENV{$_}<br>\n" foreach sort keys %ENV;
   print "<hr>\n";

   my %params;

   foreach ( sort $query->param() ) {
      $params{$_} = $query->param($_);
      print $_, " = ", $params{$_}, "<br>\n";
   }

   print "<hr>\n";

   if ( $first_time ) {
      MCE::Map->init( max_workers => 3 );
   }

   my @ret = mce_map { "$$: ".( $_ * 2 ) } 1 .. $params{'size'} || 8;

   print join("<br>\n", @ret), "<br>\n";

   $proc_manager->pm_post_dispatch();
}

so like, sharing data is totally possible via MCE::Shared between FCGI and MCE workers ✨ the demo below uses a shared counter variable that increments by one no matter which FCGI worker is serving the request. iconic behaviour honestly.

#!/usr/bin/perl

# http://127.0.0.1/cgi-bin/test_mce3.fcgi
# http://127.0.0.1/cgi-bin/test_mce3.fcgi?size=8

use strict;
use warnings;

use MCE::Map;
use MCE::Shared;

use CGI::Fast;
use FCGI::ProcManager;

# Shared variables must be defined before FCGI::ProcManager.
my $count = MCE::Shared->scalar( 0 );
my $first_time = 1;

my $proc_manager = FCGI::ProcManager->new({ n_processes => 4 });
$proc_manager->pm_manage();

# Optional, the following statement must come after $pm->pm_manage.
MCE::Shared->init(); # enables shared parallel-IPC capabilities

while ( my $query = CGI::Fast->new() ) {
   $proc_manager->pm_pre_dispatch();

   print "Content-type: text/html\r\n\r\n";
   print "$$: ", $count->incr(), "<br>\n";
   print "<hr>\n";

   print "$_ = $ENV{$_}<br>\n" foreach sort keys %ENV;
   print "<hr>\n";

   my %params;

   foreach ( sort $query->param() ) {
      $params{$_} = $query->param($_);
      print $_, " = ", $params{$_}, "<br>\n";
   }

   print "<hr>\n";

   if ( $first_time ) {
      MCE::Map->init( max_workers => 3 );
      $first_time = 0;
   }

   my @ret = mce_map { "$$: ".( $_ * 2 ) } 1 .. $params{'size'} || 8;

   print join("<br>\n", @ret), "<br>\n";

   $proc_manager->pm_post_dispatch();
}

lowkey resetting the environment is so helpful during development 🫶 the shared-manager process stops immediately upon receiving the TERM signal:

killall -TERM perl-fcgi perl-fcgi-pm ; service httpd restart

🌸 TK DEMONSTRATIONS 🌸

these demos need MCE 1.805 to run. if you're not there yet, the MCE posix_exit option must be specified and set to 1. UNIX only, bestie — and in 1.805 it's set automatically when Tk.pm is present. 💖

#!/usr/bin/perl

use strict;
use warnings;

use MCE;
use Tk;

my $mw = MainWindow->new( -title => 'MCE/Tk Test' );

$mw->geometry( '300x300' );
$mw->Button( -text => "Test MCE", -command => \&test_mce )->pack();

my $frame = $mw->Frame->pack( -fill => 'x' );

my $mce = MCE->new(
   max_workers => 4,
   user_func => sub {
      my @args = @{ MCE->user_args() };
      print MCE->pid(), ": $_\n";
   },
)->spawn;

MainLoop;

# Do not call $mce->shutdown on Windows ($^O eq 'MSWin32').
# Workers terminate with the application.
#
# $mce->shutdown();

print "Exiting...\n";

sub test_mce {
   $mce->process({
      user_args  => [ 'arg1', 'arg2', 'argN' ],
      input_data => [ 1 .. 10 ],
      chunk_size => 1,
   });
}

⚠️ bestie listen — do not call $mce->shutdown on Windows ($^O eq 'MSWin32'). workers terminate with the application, periodt.

and the next one does the same thing but via MCE::Flow ✨

#!/usr/bin/perl

use strict;
use warnings;

use MCE::Flow max_workers => 4;
use Tk;

my $mw = MainWindow->new( -title => 'MCE/Tk Test' );

$mw->geometry( '300x300' );
$mw->Button( -text => "Test MCE", -command => \&test_mce )->pack();

my $frame = $mw->Frame->pack( -fill => 'x' );

sub task {
   my @args = @{ MCE->user_args() };
   print MCE->pid(), ": $_\n";
}

MainLoop;

print "Exiting...\n";

sub test_mce {
   MCE::Flow->init(
      user_args  => [ 'arg1', 'arg2', 'argN' ],
      chunk_size => 1
   );
   MCE::Flow->run( \&task, [ 1 .. 10 ] );
}

MCE::Hobo 1.804 or later is another possibility if you're running on a UNIX platform 💜

#!/usr/bin/perl

use strict;
use warnings;

use MCE::Hobo;
use Tk;

my $mw = MainWindow->new( -title => 'MCE/Tk Test' );

$mw->geometry( '300x300' );
$mw->Button( -text => "Test MCE", -command => \&test_mce )->pack();

my $frame = $mw->Frame->pack( -fill => 'x' );

sub task {
   my @args = @_;
   print MCE::Hobo->pid(), ": $_\n";
}

MainLoop;

print "Exiting...\n";

sub test_mce {
   MCE::Hobo->create(\&task, 'arg1', 'arg2', 'argN') for ( 1 .. 4 );
   MCE::Hobo->waitall();
}

👑 INDEX 👑

MCE, MCE::Core

💌 AUTHOR 💌

Mario E. Roy, <marioeroy AT gmail DOT com>

and that's a wrap on this whole page bestie!! 🎀✨ this was MCE::Examples(3) from perl v5.42.0, dated 2025-09-10 — go forth and parallelize, you absolute icon 👑💅

📦 where this page comes from

fedora 43: perl-MCE 1.902-1.fc43 · GPL-1.0-or-later OR Artistic-1.0-Perl
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-10-14: a language model's rewrite of the original, under its terms