AnyEvent::IO(3)

🌸 NAME 🌸

okay bestie, this is AnyEvent::IO(3) from User Contributed Perl Documentation — aka AnyEvent::IO, and it is literally the DBI of asynchronous I/O implementations. no cap. 💅✨

✨ SYNOPSIS ✨

First vibe check: use the module, then load /etc/passwd and let the callback get the file data when it's done.

use AnyEvent::IO;

# load /etc/passwd, call callback with the file data when done.
aio_load "/etc/passwd", sub {
   my ($data) = @_
      or return AE::log error => "/etc/passwd: $!";

   warn "/etc/passwd contains ", ($data =~ y/://) , " colons.\n";
};

# the rest of the SYNOPSIS does the same, but with individual I/O calls

# also import O_XXX flags
use AnyEvent::IO qw(:DEFAULT :flags);

my $filedata = AE::cv;

# first open the file
aio_open "/etc/passwd", O_RDONLY, 0, sub {
   my ($fh) = @_
      or return AE::log error => "/etc/passwd: $!";

   # now stat the file to get the size
   aio_stat $fh, sub {
      @_
         or return AE::log error => "/etc/passwd: $!";

      my $size = -s _;

      # now read all the file data
      aio_read $fh, $size, sub {
         my ($data) = @_
            or return AE::log error => "/etc/passwd: $!";

         $size == length $data
            or return AE::log error => "/etc/passwd: short read, file changed?";

         # mostly the same as aio_load, above - $data contains
         # the file contents now.
         $filedata->($data);
      };
   };
};

my $passwd = $filedata->recv;
warn length $passwd, " octets.\n";

💖 DESCRIPTION 💖

This module gives you functions that do I/O in an asynchronous way. It's to I/O what AnyEvent is to event libraries: it only talks to other implementations, or to a portable pure-perl implementation — and heads up, that pure-perl one does not do asynchronous I/O. like, at all.

The only other implementation that's supported (or even known to the author) is IO::AIO. It gets used automatically when it can be loaded, through AnyEvent::AIO — and yes, that also needs to be installed. If it isn't available, AnyEvent::IO falls back to its synchronous pure-perl implementation. 🫶

Unlike AnyEvent, which model gets used is currently decided at module load time, not at first use. Future releases might change that. 👀

🧠 RATIONALE

Disk I/O often feels "instant" compared to, say, socket I/O, but there are so many situations where your program can block for extended time periods while doing disk I/O. Example: you're hitting a disk on an NFS server and it's gone — it can take ages to respond again, if it ever does. 😭 Or your system is extremely busy creating or restoring a backup — reading data from disk can then take seconds. Or you're on Linux, which for so many years has had a close-to-broken VM/IO subsystem that can often induce minutes or more of delay for disk I/O, even under what the author would consider light I/O loads. 💀

Whatever the situation, some programs just can't afford to block for long times (say, half a second or more), because they need to respond as fast as possible.

For those cases, you need asynchronous I/O. periodt.

The problem is, AnyEvent itself sometimes reads disk files (for example, when looking at /etc/hosts), and under the above situations, this can bring your program to a complete halt even if your program otherwise takes care to only use asynchronous I/O for everything (e.g. by using IO::AIO). 🫠

On the other hand, requiring IO::AIO for AnyEvent is clearly impossible, because AnyEvent promises to stay pure-perl, and the overhead of IO::AIO for small programs would be immense, especially when asynchronous I/O isn't even needed.

Clearly, this calls for an abstraction layer, and that is what you are looking at right now. :)

🦋 ASYNCHRONOUS VS. NON-BLOCKING 🦋

Many people are continuously confused on what the difference is between asynchronous I/O and non-blocking I/O. In fact, those two terms are not well defined, which often makes it hard to even talk about the difference. Here's a short guideline that should leave you less confused. It only talks about read operations, but the reasoning works with other I/O operations as well. 💖

Non-blocking I/O means that data is delivered by some external means, automatically — something pushes data towards your file handle, without you having to do anything. Non-blocking means that if your operating system currently has no data (or EOF, or some error) available for you, it will not wait ("block") as it would normally do, but immediately return with an error (e.g. EWOULDBLOCK — "I would have blocked, but you forbid it"). 😤

Your program can then wait for data to arrive by other means, for example an I/O watcher which tells you when to re-attempt the read, after which it can try to read again, and so on.

Often, you'd expect this to work for disk files as well — if the data isn't already in memory, one might want to wait for it and then re-attempt the read, for example. While that's sound reasoning, the POSIX API does not support this, because disk drives and file systems do not send data "on their own", and more so, the OS already knows that data is there. It doesn't need to "wait" until it arrives from some external entity; it only needs to transfer the data from disk to your memory buffer. the more you know ✨

So basically, while the concept is sound, the existing OS APIs do not support this. Therefore, it makes no sense to switch a disk file handle into non-blocking mode — it will behave exactly the same as in blocking mode, namely it will block until the data has been read from the disk. do not do this and expect magic, bestie.

The alternative to non-blocking I/O that actually works with disk files is usually called asynchronous I/O. Asynchronous, because the actual I/O is done while your program does something else: there's no need to call the read function to see if data is there, you only order the read once, and it will notify you when the read has finished and the data is in your buffer — all the work is done in the background. 🧚

This works with disk files, and even with sockets and other sources. It is, however, not very efficient when used with sources that could be driven in a non-blocking way, because it usually has higher overhead in the OS than non-blocking I/O, because it ties memory buffers for a potentially unlimited time and often only a limited number of operations can be done in parallel. lowkey important.

That's why asynchronous I/O makes most sense when confronted with disk files, and non-blocking I/O only makes sense with sockets, pipes and similar streaming sources. 🎀

🎀 IMPORT TAGS 🎀

By default, this module exports all aio_* functions. In addition, the following import tags can be used:

• :aio — all aio_* functions, same as :DEFAULT
• :flags — the fcntl open flags (O_CREAT, O_RDONLY, ...)

🦋 API NOTES 🦋

The functions in this module are not meant to be the most versatile or the highest-performers (they are not very slow either, of course). They are primarily meant to give users of your code the option to do the I/O asynchronously (by installing IO::AIO and AnyEvent::AIO), without adding a dependency on those modules. 🫶

🏷️ NAMING

All the functions in this module implement an I/O operation, usually with the same or similar name as the Perl built-in that they mimic, but with an aio_ prefix. If you like, you can think of the aio_* functions as "AnyEvent I/O" or "Asynchronous I/O" variants of Perl built-ins. 💅

📞 CALLING CONVENTIONS AND ERROR REPORTING

Each function expects a callback as their last argument. The callback is usually called with the result data or result code. An error is usually signalled by passing no arguments to the callback, which is then free to look at $! for the error code.

This makes all of the following forms of error checking valid:

aio_open ...., sub {
   my $fh = shift   # scalar assignment - will assign undef on error
      or return AE::log error => "...";

   my ($fh) = @_    # list assignment - will be 0 elements on error
      or return AE::log error => "...";

   @_               # check the number of elements directly
      or return AE::log error => "...";

🚨 CAVEAT: RELATIVE PATHS 🚨

When a path is specified, this path must be an absolute path, unless you make certain that nothing in your process calls chdir or an equivalent function while the request executes.

bestie, seriously — relative paths here are giving undefined chaos unless you've locked down every chdir in the whole process. you have been warned 💀

🚨 CAVEAT: OTHER SHARED STATE 🚨

Changing the umask while any requests execute that create files (or otherwise rely on the current umask) results in undefined behaviour — likewise changing anything else that would change the outcome, such as your effective user or group ID. undefined behaviour is not the vibe.

⚡ CALLBACKS MIGHT BE CALLED BEFORE FUNCTION RETURNS TO CALLER ⚡

Unlike other functions in the AnyEvent module family, these functions may call your callback instantly, before returning. This should not be a real problem, as these functions never return anything useful. 😌

🌙 BEHAVIOUR AT PROGRAM EXIT 🌙

Both AnyEvent::IO::Perl and AnyEvent::IO::IOAIO implementations make sure that operations that have started will be finished on a clean program exit. That makes programs work that start some I/O operations and then exit. For example, this complete program:

use AnyEvent::IO;

aio_stat "path1", sub {
   aio_stat "path2", sub {
      warn "both stats done\n";
   };
};

Starts a stat operation and then exits by "falling off the end" of the program. Nevertheless, both stat operations will be executed, as AnyEvent::IO waits for all outstanding requests to finish and you can start new requests from request callbacks. ✨

In fact, since AnyEvent::IO::Perl is currently synchronous, the program will do both stats before falling off the end, but with AnyEvent::IO::IOAIO, the program first falls of the end, then the stats are executed.

While not guaranteed, this behaviour will be present in future versions, if reasonably possible (which is extreemly likely :). 🫶

👑 GLOBAL VARIABLES AND FUNCTIONS 👑

🧬 $AnyEvent::IO::MODEL

Contains the package name of the backend I/O model in use — at the moment, this is usually AnyEvent::IO::Perl or AnyEvent::IO::IOAIO.

📥 aio_load $path, $cb->($data)

Tries to open $path and read its contents into memory (obviously, should only be used on files that are "small enough"), then passes them to the callback as a string. 🍓

Example: load /etc/hosts.

aio_load "/etc/hosts", sub {
   my ($hosts) = @_
      or return AE::log error => "/etc/hosts: $!";

   AE::log info => "/etc/hosts contains ", ($hosts =~ y/\n/), " lines\n";
};

📂 aio_open $path, $flags, $mode, $cb->($fh)

Tries to open the file specified by $path with the O_XXX-flags $flags (from the Fcntl module, or see below) and the mode $mode (a good value is 0666 for O_CREAT, and 0 otherwise).

...and then that (totally normal, standard, perl) file handle tied to the file you just opened gets handed straight to your callback, bestie 🎀

This thing works very much like Perl's sysopen function — same energy, just async.

⚠️ ok so real talk: changing the umask while this request is executing = undefined behaviour. Same goes for changing anything else that would change the outcome, like your effective user ID or group ID. Don't do it, bestie 💀

So you don't have to drag Fcntl into your life, this module ships its own constants for O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_EXCL, O_TRUNC and O_APPEND — grab them directly (AnyEvent::IO::O_RDONLY) or import them with the :flags import tag (check SYNOPSIS, iykyk ✨).

Example: securely open a file in /var/tmp, and fail if it already exists or if it's a symlink. Very demure, very secure 🔐

use AnyEvent::IO qw(:flags);

aio_open "/var/tmp/mytmp$$", O_CREAT | O_EXCL | O_RDWR, 0600, sub {
   my ($fh) = @_
      or return AE::log error => "$! - denial of service attack?";

   # now we have $fh
};

---

aio_close $fh, $cb->($success)

Closes the file handle — and yes, close can block your process indefinitely, no cap 😭 — and passes a true value to the callback on success 🫶

Due to perl's little idiosyncrasies, instead of literally calling close, the file handle might get closed by dup2'ing another file descriptor over it. Translation: $fh might still be open, but it can be closed safely afterwards and must not be used for anything after that, periodt.

Example: close a file handle, and because we're dirty like that, don't even bother checking for errors 💅

aio_close $fh, sub { };

aio_read $fh, $length, $cb->($data)

Tries to read $length octets from the current position of $fh and passes those bytes to $cb. Other than that, the semantics are very much like Perl's sysread 📖

If fewer than $length octets were read, $data holds only the bytes actually read. At EOF, $data is a zero-length string. If an error happens, then nothing at all is passed to the callback.

Obviously, running multiple aio_reads or aio_writes at the same time on file handles that share the same underlying open file description = undefined behaviour, because they share the current file offset. And less obviously so, because OS X is not thread safe and straight up corrupts your data when you try 💀

Example: read 128 octets from a file.

aio_read $fh, 128, sub {
   my ($data) = @_
      or return AE::log error "read from fh: $!";

   if (length $data) {
      print "read ", length $data, " octets.\n";
   } else {
      print "EOF\n";
   }
};

aio_seek $fh, $offset, $whence, $callback->($offs)

Seeks the filehandle to the new $offset, similarly to Perl's sysseek. The $whence values are the classic ones: 0 counts from the start, 1 counts from the current position, and 2 counts from the end 🎯

The resulting absolute offset gets passed to the callback on success.

Example: measure the size of the file the old-fashioned way, using seek.

aio_seek $fh, 0, 2, sub {
   my ($size) = @_
      or return AE::log error => "seek to end failed: $!";

   # maybe we need to seek to the beginning again?
   aio_seek $fh, 0, 0, sub {
      # now we are hopefully at the beginning
   };
};

aio_write $fh, $data, $cb->($length)

Tries to write the octets in $data to the current position of $fh and passes the actual number of bytes written to $cb. Other than that, semantics are very much like Perl's syswrite ✍️

If fewer than length $data octets got written, $length reflects that. If an error happens, nothing gets passed to the callback.

Obviously — saying it again because it matters — multiple aio_reads or aio_writes at the same time on file handles sharing the underlying open file description = undefined behaviour, due to the shared current file offset (and less obviously so, because OS X is not thread safe and corrupts data when you try) 😭

aio_truncate $fh_or_path, $new_length, $cb->($success)

Calls truncate on the path or perl file handle and passes a true value to the callback on success ✂️

Example: truncate /etc/passwd to zero length — this only works on systems that support truncate, should not be tried out for obvious reasons, and debian will probably open yet another security bug about this example 🙈

aio_truncate "/etc/passwd", sub {
   @_
      or return AE::log error => "/etc/passwd: $! - are you root enough?";
};

aio_utime $fh_or_path, $atime, $mtime, $cb->($success)

Calls utime on the path or perl file handle and passes a true value to the callback on success ⏰

Special case, bestie: if both $atime and $mtime are undef, it sets the times to the current time — on systems that support that, of course.

Example: try to touch a file.

aio_utime "file", undef, undef, sub { };

aio_chown $fh_or_path, $uid, $gid, $cb->($success)

Calls chown on the path or perl file handle and passes a true value to the callback on success 👑

$uid or $gid can be specified as undef, in which case the uid or gid of the file is not changed. Lowkey this differs from Perl's chown built-in, which wants -1 for that.

Example: update the group of file to 0 (root), but leave the owner alone.

aio_chown "file", undef, 0, sub {
   @_
      or return AE::log error => "chown 'file': $!";
};

aio_chmod $fh_or_path, $perms, $cb->($success)

Calls chmod on the path or perl file handle and passes a true value to the callback on success 🔓

Example: change file to be user/group/world-readable, but leave the other flags alone.

aio_stat "file", sub {
   @_
      or return AE::log error => "file: $!";

   aio_chmod "file", (stat _)[2] & 07777 | 00444, sub { };
};

aio_stat $fh_or_path, $cb->($success)

aio_lstat $path, $cb->($success)

Calls stat or lstat on the path or perl file handle and passes a true value to the callback on success 📊

The stat data becomes available by stat'ing the _ file handle (so like -x _, stat _ and so on).

Example: see if we can find the number of subdirectories of /etc.

aio_stat "/etc", sub {
   @_
      or return AE::log error => "/etc: $!";

   (stat _)[3] >= 2
      or return AE::log warn => "/etc has low link count - non-POSIX filesystem?";

   print "/etc has ", (stat _)[3] - 2, " subdirectories.\n";
};

aio_link $oldpath, $newpath, $cb->($success)

Calls link on the paths and passes a true value to the callback on success 🔗

Example: link file to file.bak, then rename file.new over file, to atomically replace it. Main character energy, honestly.

aio_link "file", "file.bak", sub {
   @_
      or return AE::log error => "file: $!";

   aio_rename "file.new", "file", sub {
      @_
         or return AE::log error => "file.new: $!";

      print "file atomically replaced by file.new, backup file.bak\n";
   };
};

aio_symlink $oldpath, $newpath, $cb->($success)

Calls symlink on the paths and passes a true value to the callback on success 🪄

Example: create a symlink slink containing random data.

aio_symlink "random data", "slink", sub {
   @_
      or return AE::log error => "slink: $!";
};

aio_readlink $path, $cb->($target)

Calls readlink on the paths and passes the link target string to the callback 🔍

Example: read the symlink called slink and verify that it contains random data.

aio_readlink "slink", sub {
   my ($target) = @_
      or return AE::log error => "slink: $!";

   $target eq "random data"
      or AE::log critical => "omg, the world will end!";
};

aio_rename $oldpath, $newpath, $cb->($success)

Calls rename on the paths and passes a true value to the callback on success ✨

See aio_link for an example (it's right up there, bestie).

aio_unlink $path, $cb->($success)

Tries to unlink the object at $path and passes a true value to the callback on success 🗑️

Example: try to delete the file tmpfile.dat~.

aio_unlink "tmpfile.dat~", sub { };

aio_mkdir $path, $perms, $cb->($success)

Calls mkdir on the path with the given permissions $perms (when in doubt, 0777 is a good value) and passes a true value to the callback on success 📁

Example: try to create the directory subdir and leave it to whoever comes after us to check whether it worked.

aio_mkdir "subdir", 0777, sub { };

aio_rmdir $path, $cb->($success)

Tries to remove the directory at $path and passes a true value to the callback on success 🧹

Example: try to remove the directory subdir and don't give a damn if that fails.

aio_rmdir "subdir", sub { };

aio_readdir $path, $cb->(\@names)

Reads all the filenames from the directory specified by $path and passes them to the callback as an array reference of names (no path prefix). The . and .. names get filtered out first 🎀

The ordering of the file names is undefined — backends that can manage it (like IO::AIO) will return the ordering that most likely is fastest to stat through, and on top of that they put entries that are probably directories first in the array.

If you need the best possible performance for recursive directory traversal, or when you're looking at really big directories, you're advised to use IO::AIO directly — specifically the aio_readdirx and aio_scandir functions, which have more options for tuning performance 🔥

Example: recursively scan a directory hierarchy, silently skip directories we couldn't read, and print all the others.

sub scan($); # visibility-in-next statement is not so useful these days
sub scan($) {
   my ($path) = @_;

   aio_readdir $path, sub {
      my ($names) = @_
         or return;

      print "$path\n";

      for my $name (@$names) {
         aio_lstat "$path/$name", sub {
            scan "$path/$name"
               if -d _;
         };
      }
   };
}

scan "/etc";

---

🌈 ENVIRONMENT VARIABLES 🌈

See the description of PERL_ANYEVENT_IO_MODEL in the AnyEvent manpage, bestie 💖

🫶 AUTHOR 🫶

Okay bestie, this whole glorious thing was written by Marc Lehmann — literally the main character of this manpage — and you can reach him at schmorp@schmorp.de if you ever wanna send fan mail (or, like, a bug report, which is also a love letter tbh).

His website is http://anyevent.schmorp.de if you wanna go lurk and see what else he's been cooking up. 👑

---

And that's a wrap, bestie!! 🎀 This page is from AnyEvent::IO(3), shipping with perl v5.42.0, dated 2019-09-18 — iconic, immaculate, no notes. Now go read some files asynchronously and slay the day away 💅✨🌸