Directory::Queue(3)

💖 Directory::Queue(3) — bestie edition 💖

okay so this is the Directory::Queue(3) page, manual name Directory::Queue, filed under User Contributed Perl Documentation. It's literally an object oriented interface to a directory based queue, and I'm gonna walk you through it like we're in the group chat. facts preserved, vibes upgraded 💅

---

✨ NAME ✨

Directory::Queue — object oriented interface to a directory based queue. that's the whole thing, bestie: queue energy, but the storage is your filesystem. 🎀

🌸 SYNOPSIS 🌸

okay so here's the sample producer and sample consumer, exactly as written. no edits, no cap:

use Directory::Queue;

#
# sample producer
#

$dirq = Directory::Queue->new(path => "/tmp/test");
foreach $count (1 .. 100) {
    $name = $dirq->add(... some data ...);
    printf("# added element %d as %s\n", $count, $name);
}

#
# sample consumer (one pass only)
#

$dirq = Directory::Queue->new(path => "/tmp/test");
for ($name = $dirq->first(); $name; $name = $dirq->next()) {
    next unless $dirq->lock($name);
    printf("# reading element %s\n", $name);
    $data = $dirq->get($name);
    # one could use $dirq->unlock($name) to only browse the queue...
    $dirq->remove($name);
}

💜 DESCRIPTION 💜

the goal of this module is to offer a queue system using the underlying filesystem for storage, security, and to prevent race conditions via atomic operations. it focuses on simplicity, robustness, and scalability. periodt.

this module allows multiple concurrent readers and writers to interact with the same queue. a Python implementation of the same algorithm is available at <https://github.com/cern-mig/python-dirq>, a Java implementation at <https://github.com/cern-mig/java-dirq>, and a C implementation at <https://github.com/cern-mig/c-dirq>, so readers and writers can be written in different programming languages. multilingual queue slay 🌈

there is no knowledge of priority within a queue. if multiple priorities are needed, multiple queues should be used. that's the rule, bestie.

🦋 TERMINOLOGY 🦋

an element is something that contains one or more pieces of data. with Directory::Queue::Simple queues, an element can only contain one binary string. with Directory::Queue::Normal queues, more complex data schemas can be used. fancy, but in a structured way ✨

a queue is a "best effort" FIFO (First In - First Out) collection of elements.

it is very hard to guarantee pure FIFO behavior with multiple writers using the same queue. consider for instance:

• Writer1: calls the add() method
• Writer2: calls the add() method
• Writer2: the add() method returns
• Writer1: the add() method returns

who should be first in the queue, Writer1 or Writer2? 😭

for simplicity, this implementation provides only "best effort" FIFO, i.e. there is a very high probability that elements are processed in FIFO order but this is not guaranteed. this is achieved by using a high-resolution timer and having elements sorted by the time their final directory gets created. so it's giving order, but not a legally binding promise 💀

🌈 QUEUE TYPES 🌈

different queue types are supported. more detailed information can be found in the modules implementing these types:

• Directory::Queue::Normal
• Directory::Queue::Simple
• Directory::Queue::Null

compared to Directory::Queue::Normal, Directory::Queue::Simple:

• is simpler
• is faster
• uses less space on disk
• can be given existing files to store
• does not support schemas
• can only store and retrieve binary strings
• is not compatible (at filesystem level) with Directory::Queue::Normal

Directory::Queue::Null is special: it is a kind of black hole with the same API as the other directory queues. iconic, honestly 🕳️👑

🔒 LOCKING 🔒

adding an element is not a problem because the add() method is atomic.

in order to support multiple reader processes interacting with the same queue, advisory locking is used. processes should first lock an element before working with it. in fact, the get() and remove() methods report a fatal error if they are called on unlocked elements. so like, lock first, then touch. no skipping the vibe check 🚨

if the process that created the lock dies without unlocking the element, we end up with a staled lock. the purge() method can be used to remove these staled locks. ghost locks got purged, periodt.

an element can basically be in only one of two states: locked or unlocked.

a newly created element is unlocked as a writer usually does not need to do anything more with it.

iterators return all the elements, regardless of their states.

there is no method to get an element state as this information is usually useless since it may change at any time. instead, programs should directly try to lock elements to make sure they are indeed locked. don't ask for the state, just try the lock, bestie 💅

🧚 CONSTRUCTOR 🧚

the new() method of this module can be used to create a Directory::Queue object that will later be used to interact with the queue. it can have a "type" attribute specifying the queue type to use. if not specified, the type defaults to "Simple".

this method is however only a wrapper around the constructor of the underlying module implementing the functionality. so:

$dirq = Directory::Queue->new(type => Foo, ... options ...);

is identical to:

$dirq = Directory::Queue::Foo->new(... options ...);

same energy, different constructor door. 🚪✨

👑 INHERITANCE 👑

regardless of how the directory queue object is created, it inherits from the "Directory::Queue" class. you can therefore test if an object is a directory queue (of any kind) by using:

if ($object->isa("Directory::Queue")) ...

that's the inheritance check, bestie. main character class energy 👑

💎 BASE METHODS 💎

here are the methods available in the base class and inherited by all directory queue implementations:

• new(PATH) — return a new object (class method)
• copy() — return a copy of the object
• path() — return the queue toplevel path
• id() — return a unique identifier for the queue
• first() — return the first element in the queue, resetting the iterator; return an empty string if the queue is empty
• next() — return the next element in the queue, incrementing the iterator; return an empty string if there is no next element

every single one is there, no skipping 💖

🔐 SECURITY 🔐

there are no specific security mechanisms in this module.

the elements are stored as plain files and directories. the filesystem security features (owner, group, permissions, ACLs...) should be used to adequately protect the data.

by default, the process' umask is respected. see the class constructor documentation if you want an other behavior.

if multiple readers and writers with different uids are expected, the easiest solution is to have all the files and directories inside the toplevel directory world-writable (i.e. umask=0). then, the permissions of the toplevel directory itself (e.g. group-writable) are enough to control who can access the queue. filesystem perms are the security vibe here, no cap 🔐

🎀 SEE ALSO 🎀

Directory::Queue::Normal, Directory::Queue::Null, Directory::Queue::Set, Directory::Queue::Simple.

🌟 AUTHOR 🌟

Lionel Cons <http://cern.ch/lionel.cons>

Copyright (C) CERN 2010-2022

---

💖 cute sign-off: this page says perl v5.32.1, dated 2022-07-20, and it's the Directory::Queue(3) manual — bye bestie, go queue something iconic 🫶