Proc::Queue(3pm)

📦
version 1.23 · 4 versions
, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu resolute

💖 Proc::Queue 💖

okay so bestie, welcome to Proc::Queue(3pm) — straight outta the User Contributed Perl Documentation ✨ yes it's a Perl module, yes it's about forks, and yes it's gonna save your machine from absolutely melting 🫶

🌸 NAME 🌸

Proc::Queue — limit the number of child processes running 💅

like… it caps how many kids you're allowed to spawn at once. that's the whole vibe. periodt.

✨ SYNOPSIS ✨

  use Proc::Queue size => 4, debug => 1;

  package other;
  use POSIX ":sys_wait_h"; # imports WNOHANG

  # this loop creates new children, but Proc::Queue makes it wait every
  # time the limit (4) is reached until enough children exit
  foreach (1..100) {
    my $f=fork;
    if(defined ($f) and $f==0) {
      print "-- I'm a forked process $$\n";
      sleep rand 5;
      print "-- I'm tired, going away $$\n";
      exit(0)
    }
    1 while waitpid(-1, WNOHANG)>0; # reaps children
  }

  Proc::Queue::size(10); # changing limit to 10 concurrent processes
  Proc::Queue::trace(1); # trace mode on
  Proc::Queue::debug(0); # debug is off
  Proc::Queue::delay(0.2); # set 200 miliseconds as minimum
                           # delay between fork calls

  package other; # just to test it works on any package

  print "going again!\n";

  # another loop with different settings for Proc::Queue
  foreach (1..20) {
    my $f=fork;
    if(defined ($f) and $f==0) {
      print "-- I'm a forked process $$\n";
      sleep rand 5;
      print "-- I'm tired, going away $$\n";
      exit(0)
    }
  }

  1 while wait != -1;

no cap, that's the whole demo. fork like you always do, and the module quietly handles the queueing for you 😌

🦋 DESCRIPTION 🦋

So here's the tea: this module lets you parallelise a perl program using the fork, exit, wait and waitpid calls exactly like you normally would, but without you having to babysit how many processes you're creating and overloading the machine 💀

It redefines perl's fork, exit, wait and waitpid core functions. Old programs do not need to be reprogrammed — you literally just add the use Proc::Queue ... sentence to them and you're done. iconic behaviour, honestly 👑

On top of that, the module has two debugging modes (debug and trace) that seem too be very useful when developing parallel aplications:

  • debug mode: when active, dumps lots of information about processes being created, exiting, being caught by parent, etc.
  • trace mode: prints a line every time one of the fork, exit, wait or waitpid functions are called.

It is also possible to set a minimun delay time between fork calls to stop too many processes for starting in a short time interval. (yes, throttling, we love a rate limit ✋)

Child processes continue to use the modified functions, but their queues are reset and the maximun process number for them is set to 1 (anyway, children can change their queue size themselves).

🚨 REAL TALK: Proc::Queue doesn't work if the CHLD signal handler is set to IGNORE. ✋

Internally, Proc::Queue automatically catches zombies 🧟 and stores their exit status in a private hash. To avoid leaking memory in long running programs you have to call wait or waitpid to delete entries from that hash — or alternatively active the ignore_children mode:

  Proc::Queue::ignore_children(1)

or

  use Proc::Queue ignore_children=>1, ...

🎀 EXPORT 🎀

This module redefines the fork, wait, waitpid and exit calls. free real estate, they just show up 💖

🌈 EXPORT_OK 🌈

Functions fork_now, waitpids, run_back, run_back_now, all_exit_ok, running_now, system_back and system_back_now can be imported. Tag :all is defined to import all of them. (gotta ask for these ones, bestie — they're not automatic ✨)

💅 FUNCTIONS 💅

There are several not exported functions that can be used to configure the module:

  • size(), size($number) — If an argument is given the maximun number of concurrent processes is set to it and the number of maximun processes that were allowed before is returned. If no argument is given, the number of processes allowed is returned. (it's giving… getter-slash-setter, iykyk)
  • delay(), delay($time) — lets you set a minimun time in seconds to elapse between consecutive calls to fork. It is useful to avoid creating too many processes in a short time (that could degrade performance). If Time::HiRes module is available delays shorted that 1 second are allowed. If no arg is given, the current delay is returned. To clear it use Proc::Queue::delay(0). ⏳ gentle pacing, bestie
  • weight(), weight($weight) — by default any process forked count as 1 through the max number of processes allowed to run simultaneously (the queue size). weight allows to change this, i.e.:
  Proc::Queue::weight(3);
  run_back { ... heavy process here ... };
  Proc::Queue::weight(1);

causes the "heavy process" to count as three normal processes. Valid weight values are integers greater than zero.

⚠️ Remember to reset the weight back to 1 (or whatever) after the heavier processes have been forked!. not me forgetting to reset it and wondering why my queue is stuck 💀
  • allow_excess(), allow_excess($allow_excess) — by default the next queued process will be started as soon as the number of running processes is smaller than the queue size — this is regardless of the weight of the next queued process, so the queue could become overloaded. Setting allow_excess to false forces the next queued process to wait until there is room for it in the queue, that is, the size of the queue less the weighted number of currently running processes must be no smaller than the weight of the next queued process in order for the next process to start. Setting allow_excess to any value greater than zero (default is 1) resets the default behavior.
  • ignore_children($on) — calling
  Proc::Queue::ignore_children(1);

is the equivalent to

  $SIG{CHLD}='IGNORE'

when using Proc::Queue. same energy, different spelling ✨

  • debug(), debug($boolean), trace(), trace($boolean) — Change or return the status for the debug and trace modes. toggle toggle toggle 🎛️

Other utility subroutines that can be imported from Proc::Queue are:

  • fork_now() — Sometimes you would need to fork a new child without waiting for other children to exit if the queue is full, fork_now does that. It is exportable so you can do…
  use Proc::Queue size => 5, qw(fork_now), debug =>1;

  $f=fork_now;
  if(defined $f and $f == 0) {
      print "I'm the child\n"; exit;
  }
  • waitpids(@pid) — waits for all the processes in @pid to exit. It returns an array with pairs of pid and exit values (pid1, exit1, pid2, exit2, pid3, exit3,...) as returned by individual waitpid calls. pairs on pairs on pairs 🫶
  • run_back(\&code), run_back { code } — Runs the argument subrutine in a forked child process and returns the pid number for the new process. fire and forget, but make it structured 💖
  • run_back_now(\&code), run_back_now { code } — A mix between run_back and fork_now. the collab we didn't know we needed ✨
  • system_back(@command) — Similar to the system call but runs the command in the background and waits for other children to exit first if there are already too many running. Returns the pid of the forked process or undef if the program was not found.
  • system_back_now(@command) — As system_back but without checking if the maximun number of children allowed has been reached. yolo mode, respectfully 😭
  • all_exit_ok(@pid) — Do a waitpids call and test that all the processes exit with code 0. vibe check: passed ✅
  • running_now() — Returns the number of child processes currently running. a little census, as a treat 🔢
  • import(pkg,opt,val,opt,val,...,fnt_name,fnt_name,...) — The import function is not usually explicitally called but by the use Proc::Queue statement. Options allowed are size, debug, weight and trace, i.e:
  use Proc::Queue size=>10, debug=>1;

Anything that is not size, debug, weight or trace is expected to be a function name to be imported.

  use Proc::Queue size=>10, ':all';

🐛 BUGS 🐛

Proc::Queue is a very stable module, no bugs have been reported for a long time. ngl that's a flex 💅

Support for Win32 OSs is still experimental. ⚠️ proceed accordingly, bestie.

👀 SEE ALSO 👀

perlfunc(1), perlipc(1), POSIX, perlfork(1), Time::HiRes, Parallel::ForkManager. The example.pl script contained in the module distribution.

go read the whole friend group, they're all linked 💖

Copyright 2001-2003, 2005-2008 by Salvador Fandiño <sfandino@yahoo.com>

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

and that's the whole page, bestie 🫶 this was perl v5.32.0, dated 2021-01-07 — stay forky, stay queued, and never let your machine get overloaded 💅✨🎀

📦 where this page comes from