disk_log(3)

📦
version 29.0.6 · 3 versions
, in ubuntu stonking

✨ disk_log(3) — the Erlang Module Definition that's about to be your logging bestie. ✨

okay bestie, welcome to disk_log. This is part 1 of 5, so hydrate, stay with me, and let's get into it. 💖

🌸 NAME 🌸

disk_log — a disk-based term logger that enables efficient logging of items on files. It's giving "your terms, but on disk, and fast." No cap. 🦋

💖 DESCRIPTION 💖

Three types of logs are supported. Let's get into it, no gatekeeping:

  • halt logs — appends items to a single file. The size of that file can be limited by the disk_log module. One file, one vibe. 💅
  • wrap logs — uses a sequence of wrap log files of limited size. As a wrap log file is filled up, further items are logged on to the next file in the sequence, starting all over with the first file when the last file is filled up. It's a little circular bestie. 🌀
  • rotate logs — uses a sequence of rotate log files of limited size. As a log file is filled up, it is rotated and then compressed. There is one active log file and up to the configured number of compressed log files. Only externally formatted logs are supported. It follows the same naming convention as the handler logger_std_h for Logger. For more details about the naming convention, check the file parameter for open/1.

It follows the same naming convention as that for the compressed files for Linux's logrotate and BSD's newsyslog. ✨

For efficiency reasons, items are always written to files as binaries. (yes, binaries. iconic.)

Two formats of the log files are supported:

  • internal format — supports automatic repair of log files that are not properly closed, and enables efficient reading of logged items in chunks using a set of functions defined in this module. This is the only way to read internally formatted logs. An item logged to an internally formatted log must not occupy more than 4 GB of disk space (the size must fit in 4 bytes). 👑
  • external format — leaves it up to the user to read and interpret the logged data. The disk_log module cannot repair externally formatted logs. you're on your own, bestie.

For each open disk log, one process handles requests made to the disk log. This process is created when open/1 is called, provided there exists no process handling the disk log. A process that opens a disk log can be an owner or an anonymous user of the disk log. Each owner is linked to the disk log process, and an owner can close the disk log either explicitly (by calling close/1) or by terminating. 💀

Owners can subscribe to notifications, messages of the form {disk_log, Node, Log, Info}, which are sent from the disk log process when certain events occur. See the functions and in particular the open/1 option notify. A log can have many owners, but a process cannot own a log more than once. However, the same process can open the log as a user more than once. not me learning ownership rules at 2am.

For a disk log process to close its file properly and terminate, it must be closed by its owners and once by some non-owner process for each time the log was used anonymously. The users are counted and there must not be any users left when the disk log process terminates. 🎀

Items can be logged synchronously by using functions log/2, blog/2, log_terms/2, and blog_terms/2. For each of these functions, the caller is put on hold until the items are logged (but not necessarily written, use sync/1 to ensure that). By adding an a to each of the mentioned function names, we get functions that log items asynchronously. Asynchronous functions do not wait for the disk log process to write the items to the file, but return the control to the caller more or less immediately. async said "I'll handle it" and dipped. 💅

When using the internal format for logs, use functions log/2, log_terms/2, alog/2, and alog_terms/2. These functions log one or more Erlang terms. By prefixing each of the functions with a b (for "binary"), we get the corresponding blog() functions for the external format. These functions log one or more chunks of bytes. For example, to log the string "hello" in ASCII format, you can use disk_log:blog(Log, "hello"), or disk_log:blog(Log, list_to_binary("hello")). The two alternatives are equally efficient. equal effort, equal slay.

The blog() functions can also be used for internally formatted logs, but in this case they must be called with binaries constructed with calls to term_to_binary/1. There is no check to ensure this, it is entirely the responsibility of the caller. If these functions are called with binaries that do not correspond to Erlang terms, the chunk/2,3 and automatic repair functions fail. The corresponding terms (not the binaries) are returned when chunk/2,3 is called. so don't just send random bytes and act surprised. 😭

An open disk log is only accessible from the node where the disk log process runs. All processes on the node where the disk log process runs can log items or otherwise change, inspect, or close the log. 🌐

Errors are reported differently for asynchronous log attempts and other uses of the disk_log module. When used synchronously, this module replies with an error message, but when called asynchronously, this module does not know where to send the error message. Instead, owners subscribing to notifications receive an error_status message. async errors go straight to the group chat, basically.

The disk_log module does not report errors to the error_logger module. It is up to the caller to decide whether to employ the error logger. Function format_error/1 can be used to produce readable messages from error replies. However, information events are sent to the error logger in two situations, namely when a log is repaired, or when a file is missing while reading chunks. 👀

Error message no_such_log means that the specified disk log is not open. Nothing is said about whether the disk log files exist or not. it's giving "not open, and that's all I'm saying."

NOTE: If an attempt to reopen or truncate a log fails (see reopen/2,3 and truncate/1,2), the disk log process terminates immediately. Before the process terminates, links to owners and blocking processes (see block/1,2) are removed. The effect is that the links work in one direction only. Any process using a disk log must check for error message no_such_log if some other process truncates or reopens the log simultaneously. 🚨 this is the part where you check no_such_log, fr fr.

👀 See Also 👀

  • file
  • wrap_log_reader

✨ DATA TYPES ✨

okay bestie, this is the ✨type lore✨ section. it's giving very "here's every shape of everything you could possibly get back" and honestly? we love a thorough queen. no skimming, we respect the craft here 💅

bchunk_ret() — what bchunk/2,3 hands back to you:

{Continuation2 :: continuation(), Binaries :: [binary()]} |
{Continuation2 :: continuation(),
 Binaries :: [binary()],
 Badbytes :: non_neg_integer()} |
eof |
{error, Reason :: chunk_error_rsn()}.

block_error_rsn() — the reasons block/2 might be like "nope":

no_such_log | nonode | {blocked_log, log()}.

chunk_error_rsn() — the reasons a chunk call can fail:

no_such_log |
{format_external, log()} |
{blocked_log, log()} |
{badarg, continuation} |
{not_internal_wrap, log()} |
{corrupt_log_file, FileName :: file:filename()} |
{file_error, file:filename(), file_error()}.

chunk_ret() — the return shape of chunk/2,3:

{Continuation2 :: continuation(), Terms :: [term()]} |
{Continuation2 :: continuation(),
 Terms :: [term()],
 Badbytes :: non_neg_integer()} |
eof |
{error, Reason :: chunk_error_rsn()}.

close_error_rsn() — when closing gets messy:

no_such_log | nonode |
{file_error, file:filename(), file_error()}.

-opaque continuation() — bestie this one is opaque, meaning hands off 🙈. it's the chunk continuation you get back from chunk/2,3, bchunk/2,3, or chunk_step/3. don't poke at its insides, just pass it right back where it came from.

dlog_format() :: external | internal. — two formats, two vibes.

dlog_head_opt() :: none | term() | iodata(). — head options, keep it cute.

dlog_info() — the full tea that info/1 spills about a log:

{name, Log :: log()} |
{file, File :: file:filename()} |
{type, Type :: dlog_type()} |
{format, Format :: dlog_format()} |
{size, Size :: dlog_size()} |
{mode, Mode :: dlog_mode()} |
{owners, [{pid(), Notify :: boolean()}]} |
{users, Users :: non_neg_integer()} |
{status,
 Status :: ok | {blocked, QueueLogRecords :: boolean()}} |
{node, Node :: node()} |
{head,
 Head ::
     none |
     {head, binary()} |
     (MFA :: {atom(), atom(), list()})} |
{no_written_items, NoWrittenItems :: non_neg_integer()} |
{full, Full :: boolean} |
{no_current_bytes, non_neg_integer()} |
{no_current_items, non_neg_integer()} |
{no_items, non_neg_integer()} |
{current_file, pos_integer()} |
{no_overflows,
 {SinceLogWasOpened :: non_neg_integer(),
  SinceLastInfo :: non_neg_integer()}}.

dlog_mode() :: read_only | read_write. — pick your lane.

dlog_optattr() — the valid option attribute names, i.e. the little keys:

name | file | linkto | repair | type | format | size |
notify | head | head_func | mode.

dlog_option() — every option you can throw at a log when opening it:

{name, Log :: log()} |
{file, FileName :: file:filename()} |
{linkto, LinkTo :: none | pid()} |
{repair, Repair :: true | false | truncate} |
{type, Type :: dlog_type()} |
{format, Format :: dlog_format()} |
{size, Size :: dlog_size()} |
{notify, boolean()} |
{head, Head :: dlog_head_opt()} |
{head_func, MFA :: {atom(), atom(), list()}} |
{quiet, boolean()} |
{mode, Mode :: dlog_mode()}.

dlog_options() :: [dlog_option()]. — and a whole list of those, periodt.

dlog_size() — plus infinity if you're feeling limitless:

infinity |
pos_integer() |
{MaxNoBytes :: pos_integer(), MaxNoFiles :: pos_integer()}.

dlog_type() :: halt | wrap | rotate. — the three flavors of log energy.

file_error() :: term(). — it's a term, that's all we're saying.

inc_wrap_error_rsn() :: next_file_error_rsn(). — literally just an alias, iconic behavior.

invalid_header() :: term(). — also a term. mysterious queen.

log() :: term(). — yes bestie, the log name itself is a term.

log_error_rsn() — everything that can go wrong when you log:

no_such_log | nonode |
{read_only_mode, log()} |
{format_external, log()} |
{blocked_log, log()} |
{full, log()} |
{invalid_header, invalid_header()} |
{file_error, file:filename(), file_error()}.

next_file_error_rsn() — reasons moving to the next file can fail:

no_such_log | nonode |
{read_only_mode, log()} |
{blocked_log, log()} |
{halt_log, log()} |
{rotate_log, log()} |
{invalid_header, invalid_header()} |
{file_error, file:filename(), file_error()}.

notify_ret() :: ok | {error, no_such_log}. — short and sweet.

open_error_rsn() — the whole saga of opening failures, and there are many:

no_such_log |
{badarg, term()} |
{size_mismatch,
 CurrentSize :: dlog_size(),
 NewSize :: dlog_size()} |
{arg_mismatch,
 OptionName :: dlog_optattr(),
 CurrentValue :: term(),
 Value :: term()} |
{name_already_open, Log :: log()} |
{open_read_write, Log :: log()} |
{open_read_only, Log :: log()} |
{need_repair, Log :: log()} |
{not_a_log_file, FileName :: file:filename()} |
{invalid_index_file, FileName :: file:filename()} |
{invalid_header, invalid_header()} |
{file_error, file:filename(), file_error()} |
{node_already_open, Log :: log()}.

open_ret() — what you actually get back from opening, including the glow-up repair case:

{ok, Log :: log()} |
{repaired,
 Log :: log(),
 {recovered, Rec :: non_neg_integer()},
 {badbytes, Bad :: non_neg_integer()}} |
{error, open_error_rsn()}.

reopen_error_rsn() — reopening can also be dramatic:

no_such_log | nonode |
{read_only_mode, log()} |
{blocked_log, log()} |
{same_file_name, log()} |
{invalid_index_file, file:filename()} |
{invalid_header, invalid_header()} |
{file_error, file:filename(), file_error()}.

sync_error_rsn() — syncing said no:

no_such_log | nonode |
{read_only_mode, log()} |
{blocked_log, log()} |
{file_error, file:filename(), file_error()}.

trunc_error_rsn() — truncating said no:

no_such_log | nonode |
{read_only_mode, log()} |
{blocked_log, log()} |
{invalid_header, invalid_header()} |
{file_error, file:filename(), file_error()}.

unblock_error_rsn() — unblocking said no:

no_such_log | nonode |
{not_blocked, log()} |
{not_blocked_by_pid, log()}.

💖 FUNCTIONS 💖

this is the main character section fr. every function, every arg, no crumbs left behind 🍓

all() — say hi to every log on this node

all() -> [Log] when Log :: log().

Returns the names of the disk logs accessible on the current node. that's it, that's the whole function, and honestly? so clean ✨

alog(Log, Term) — the async version of log/2

alog(Log, Term) -> notify_ret() when Log :: log(), Term :: term().

Asynchronously version of log/2.

Owners subscribing to notifications receive message read_only, blocked_log, or format_external if the item cannot be written on the log, and possibly one of the messages wrap, full, or error_status if an item is written on the log. Message error_status is sent if something is wrong with the header function or if a file error occurs. bestie that is a LOT of possible messages, so watch your inbox 💌

alog_terms(Log, TermList) — async log_terms/2

alog_terms(Log, TermList) -> notify_ret()
                    when Log :: log(), TermList :: [term()].

Asynchronously version of log_terms/2.

Owners subscribing to notifications receive message read_only, blocked_log, or format_external if the items cannot be written on the log, and possibly one or more of the messages wrap, full, and error_status if items are written on the log. Message error_status is sent if something is wrong with the header function or if a file error occurs.

balog(Log, Bytes) — async blog/2, the binary edition

balog(Log, Bytes) -> notify_ret()
               when Log :: log(), Bytes :: iodata().

Asynchronously version of blog/2.

Owners subscribing to notifications receive message read_only, blocked_log, or format_external if the item cannot be written on the log, and possibly one of the messages wrap, full, or error_status if an item is written on the log. Message error_status is sent if something is wrong with the header function or if a file error occurs.

balog_terms(Log, ByteList) — async blog_terms/2

balog_terms(Log, ByteList) -> notify_ret()
                     when Log :: log(), ByteList :: [iodata()].

Asynchronously version of blog_terms/2.

Owners subscribing to notifications receive message read_only, blocked_log, or format_external if the items cannot be written on the log, and possibly one or more of the messages wrap, full, and error_status if items are written on the log. Message error_status is sent if something is wrong with the header function or if a file error occurs.

bchunk(Log, Continuation, N) — chunk but raw binaries

bchunk(Log, Continuation, N) -> bchunk_ret()
                when
                    Log :: log(),
                    Continuation :: start | continuation(),
                    N :: pos_integer() | infinity.

Equivalent to chunk(Log, Continuation, N) except that it returns the binaries read from the file, that is it does not call binary_to_term/1. so if you like your data un-decoded and feral, this is your girl 🔥

block(Log, QueueLogRecords) — freeze that log

block(Log, QueueLogRecords) -> ok | {error, block_error_rsn()}
               when Log :: log(), QueueLogRecords :: boolean().

With a call to block/2 a process can block a log. (yes, a whole process can just put a log in time-out, the audacity 👑)

If the blocking process is not an owner of the log, a temporary link is created between the disk log process and the blocking process. The link ensures that the disk log is unblocked if the blocking process terminates without first closing or unblocking the log. so like, if the blocker ghosts, the log gets freed. mutual protection, we love to see it 🫶

Any process can probe a blocked log with info/1 or close it with close/1. The blocking process can also use functions chunk/2,3, bchunk/2,3, chunk_step/3, and unblock/1 without being affected by the block. Any other attempt than those mentioned so far to update or read a blocked log suspends the calling process until the log is unblocked or returns error message {blocked_log, Log}, depending on whether the value of QueueLogRecords is true or false. (so true = you wait in line like a polite queen, false = instant rejection, no cap 💀)

blog(Log, Bytes) — log/2 for externally formatted logs

blog(Log, Bytes) -> ok | {error, Reason :: log_error_rsn()}
              when Log :: log(), Bytes :: iodata().

Equivalent to log/2 except that it is used for externally formatted logs.

blog/2 can also be used for internally formatted logs if the binaries are constructed with calls to term_to_binary/1. versatile icon ⭐

blog_terms(Log, BytesList) — log_terms/2 for externally formatted logs

blog_terms(Log, BytesList) ->
                    ok | {error, Reason :: log_error_rsn()}
                    when Log :: log(), BytesList :: [iodata()].

Equivalent to log_terms/2 except that it is used for externally formatted logs.

blog_terms/2 can also be used for internally formatted logs if the binaries are constructed with calls to term_to_binary/1.

breopen(Log, File, BHead) — reopen, external glow

breopen(Log, File, BHead) -> ok | {error, reopen_error_rsn()}
                 when
                     Log :: log(),
                     File :: file:filename(),
                     BHead :: iodata().

Equivalent to reopen except that it is used for externally formatted logs.

btruncate/2

btruncate(Log, BHead) -> ok | {error, trunc_error_rsn()}
                          when Log :: log(), BHead :: iodata().

okay so this one is literally just truncate/2, but for externally formatted logs. that's the whole thing. same energy, different format. slay 💅

change_header/2

change_header(Log, Header) -> ok | {error, Reason}
                        when
                            Log :: log(),
                            Header ::
                                {head, dlog_head_opt()} |
                                {head_func,
                                 MFA :: {atom(), atom(), list()}},
                            Reason ::
                                no_such_log | nonode |
                                {read_only_mode, Log} |
                                {blocked_log, Log} |
                                {badarg, head}.

changes the value of the head option or the head_func option — for an owner of a disk log. that's the qualifications bestie, you gotta be an owner. ✨

change_notify/3

change_notify(Log, Owner, Notify) -> ok | {error, Reason}
                        when
                            Log :: log(),
                            Owner :: pid(),
                            Notify :: boolean(),
                            Reason ::
                                no_such_log | nonode |
                                {blocked_log, Log} |
                                {badarg, notify} |
                                {not_owner, Owner}.

same deal but for the notify option. flips that boolean for an owner of a disk log. and if you're not the owner?? {not_owner, Owner}. caught. 🙈

change_size/2

change_size(Log, Size) -> ok | {error, Reason}
                      when
                          Log :: log(),
                          Size :: dlog_size(),
                          Reason ::
                              no_such_log | nonode |
                              {read_only_mode, Log} |
                              {blocked_log, Log} |
                              {new_size_too_small, Log,
                               CurrentSize :: pos_integer()} |
                              {badarg, size} |
                              {file_error, file:filename(),
 file_error()}.

changes the size of an open log. and the rules are different depending on your log type, so pay attention:

  • halt log — you can always go bigger. you can go smaller, but not below the current file size. that's the floor, periodt.
  • wrap log or rotate log — you can always increase both the size and the number of files, as long as the number of files doesn't go over 65000. 📈
  • wrap logs specifically — if you decrease the max number of files, the change doesn't actually take effect until the current file fills up and the log wraps to the next one. the redundant files get yeeted the next time the log wraps around, i.e. when it starts logging to file number 1.
  • rotate logs specifically — if you decrease the max number of files, the redundant ones get deleted instantly. no waiting, gone. 💀

and here's the little example they give us, okay, so it's like:

say the old max number of files is 10 and the new max is 6. if the current file number isn't greater than the new max, then files 7-10 are removed when file 6 is full and the log starts writing to file number 1 again. otherwise, the files greater than the current file get removed when the current file is full — for example, if the current file is 8, then files 9 and 10 are removed. the files sitting between the new max and the current file (so, files 7 and 8) get removed the next time file 6 is full. 🎀

also, on shrinking the size of the files:

  • the change hits the current log immediately, but it does not change log files that are already full until the next time they're used.
  • if you're shrinking the log size to, like, save space? use next_file/1 to force the log to wrap. queen behavior 👑

chunk/3

chunk(Log, Continuation, N) -> chunk_ret()
                when
                    Log :: log(),
                    Continuation :: start | continuation(),
                    N :: pos_integer() | infinity.

efficiently reads the terms appended to an internally formatted log. it keeps disk I/O low by reading 64 kilobyte chunks from the file. 🦋

first time you call chunk(), you gotta hand it an initial continuation — the atom start. no start, no chunking.

when you call chunk/3, N sets the max number of terms read from the log in each chunk. infinity means all the terms in that 64 kilobyte chunk get read. and heads up: if fewer than N terms come back, that does not necessarily mean you hit the end of the file. don't assume, bestie. 😭

chunk/3 returns a tuple {Continuation2, Terms}, where Terms is the list of terms it found. Continuation2 is yet another continuation, and you pass it along to the following chunk() calls. do that in a series and you can extract every term from a log. 🫶

if the log is opened in read-only mode and the read chunk is corrupt, you get {Continuation2, Terms, Badbytes} instead. Badbytes is the number of bytes in the file that turned out not to be Erlang terms in that chunk. note: the log is not repaired. we don't fix, we just report. 💅 and if you're reading chunks from a log opened in read-write mode and the chunk is corrupt, you get {corrupt_log_file, FileName}.

chunk/3 returns eof when you reach the end of the log, or {error, Reason} if something goes wrong. if a wrap log file is missing, a message gets output on the error log.

⚠️ when chunk/2,3 is used with wrap logs, the continuation you get back might not be valid on the next chunk/3 call — because the log can wrap and delete the file that continuation points into. to stop that from happening, block the log while you search.

chunk_info/1

chunk_info(Continuation) -> InfoList | {error, Reason}
                     when
                         Continuation :: continuation(),
                         InfoList :: [{node, Node :: node()}, ...],
                         Reason :: {no_continuation, Continuation}.

returns the pair {node, Node} that describes the chunk continuation you got from chunk/2,3, bchunk/2,3, or chunk_step/3.

terms get read from the disk log running on Node. that's the whole point. 🌈

chunk_step/3

chunk_step(Log, Continuation, Step) ->
                     {ok, any()} | {error, Reason}
                     when
                         Log :: log(),
                         Continuation :: start | continuation(),
                         Step :: integer(),
                         Reason ::
                             no_such_log | end_of_log |
                             {format_external, Log} |
                             {blocked_log, Log} |
                             {badarg, continuation} |
                             {file_error, file:filename(), file_error()}.

works together with chunk/2,3 and bchunk/2,3 to search through an internally formatted wrap log. 🔍

you hand it a continuation (as returned by chunk/2,3, bchunk/2,3, or chunk_step/3), and it steps forward — or backward, cause Step is an integer — Step files in the wrap log. the continuation it hands back points to the first log item in the new current file.

if you pass the atom start as the continuation, the first file of the wrap log becomes the new current file. fresh start energy ✨

and if the wrap log isn't full yet (like, not all the files are used), then trying to step outside the log gets you {error, end_of_log}. stay in your lane 🚧

close/1

close(Log) -> ok | {error, close_error_rsn()} when Log :: log().

closes a disk log properly. and "properly" is doing a lot of work in that sentence, so listen:

🚨 an internally formatted log must be closed before the Erlang system is stopped. if you don't, the log is regarded as unclosed, and the automatic repair procedure kicks in the next time you open it. do not make the repair procedure do unpaid labor.

the disk log process doesn't terminate as long as there are owners or users on the log. all owners have to close the log — maybe by terminating, that counts too. also, any other process (not just the ones that opened the log anonymously) can decrement the users counter by closing the log. but attempts to close a log by a process that isn't an owner are ignored if there are no users. ignored, bestie. not even mad, just ignored. 💀

and if the closing process is the one that blocked the log? the log gets unblocked too. two birds. 🐦

format_error/1

format_error(Error) -> io_lib:chars() when Error :: term().

give it the error some function in this module returned, and it gives you back a descriptive string of that error, in English. that's it, that's the whole function, and it's lowkey a lifesaver. 🙏

for file errors, it calls function format_error/1 in module file instead. delegation, queen behavior. 👑

inc_wrap_file/1

inc_wrap_file(Log) -> ok | {error, inc_wrap_error_rsn()}
                        when Log :: log().
⚠️ Deprecated: disk_log:inc_wrap_file/1 is deprecated; use disk_log:next_file/1 instead.

forces the internally formatted disk log to start logging to the next log file. you can use it, for example, with change_size/2 to cut down the amount of disk space the disk log is hogging. 📉

owners who subscribe to notifications normally receive a wrap message. but if an error happens with a reason tag of invalid_header or file_error, then an error_status message gets sent instead.

info/1

info(Log) -> InfoList | {error, no_such_log}
               when Log :: log(), InfoList :: [dlog_info()].

returns a list of {Tag, Value} pairs describing a log running on the node. like a little ✨ vibe check ✨ for your log. here's everything you get for all logs:

  • {name, Log} — Log is the log name, exactly as you specified it in the open/1 option name.
  • {file, File} — for halt logs, File is the filename. for wrap logs, File is the base name.
  • {type, Type} — Type is the log type, as specified by the open/1 option type.
  • {format, Format} — Format is the log format, as specified by the open/1 option format.
  • {size, Size} — Size is the log size, as specified by the open/1 option size, or the size set by change_size/2. anything you set with change_size/2 shows up here immediately. no delay, no lag. ⚡
  • {mode, Mode} — Mode is the log mode, as specified by the open/1 option mode.
  • {owners, [{pid(), Notify}]} — Notify is the value set by the open/1 option notify, or by function change_notify/3, for the owners of the log.
  • {users, Users} — Users is the number of anonymous users of the log. see the open/1 option linkto. 👀
  • {status, Status} — Status is ok, or {blocked, QueueLogRecords}, as set by functions block/1,2 and unblock/1.
  • {node, Node} — the info coming back from this particular info/1 call was gathered straight from the disk log process vibing on Node. 🛰️

The following pairs come back for all logs opened in read_write mode:

  • {head, Head} — depends on the head and head_func options of open/1, or on whatever change_header/2 set. So Head is none (the default), or {head, H} if you're on the head option, or {M,F,A} if you're on head_func. ✨
  • {no_written_items, NoWrittenItems} — NoWrittenItems is the number of items written to the log since the disk log process was created. 🎀

The following pair comes back for halt logs opened in read_write mode:

  • {full, Full} — Full is true or false, depending on whether the halt log is full or not. Lowkey stressful but okay. 💀

The following pairs come back for wrap logs opened in read_write mode:

  • {no_current_bytes, integer() >= 0} — how many bytes got written to the current wrap log file.
  • {no_current_items, integer() >= 0} — how many items got written to the current wrap log file, header inclusive.
  • {no_items, integer() >= 0} — the total number of items across all the wrap log files.
  • {current_file, integer()} — the ordinal of the current wrap log file, in the range 1..MaxNoFiles, where MaxNoFiles is set by the open/1 option size or by change_size/2.
  • {no_overflows, {SinceLogWasOpened, SinceLastInfo}} — SinceLogWasOpened (SinceLastInfo) is how many times a wrap log file has filled up and a new one got opened, or how many times inc_wrap_file/1 has been called, since the disk log was last opened (aka since info/1 was last called). The very first time info/2 is called after a log was (re)opened or truncated, those two values are equal. 👯
Real talk: functions chunk/2,3, bchunk/2,3, and chunk_step/3 do not affect any value returned by info/1. Not even a little. 🚫

🌸 log/2 🌸

log(Log, Term) -> ok | {error, Reason :: log_error_rsn()}
                    when Log :: log(), Term :: term().

Synchronously appends a term to an internally formatted disk log, bestie. Returns ok or {error, Reason} when the term is written to disk. 💅

Terms get written by the ordinary write() function of the operating system, so nope, it is not guaranteed that the term is written to disk — it can just chill in the operating system kernel for a while. If you need it on disk for real, you gotta call sync/1. 😤

Owners subscribing to notifications get notified of an error with an error_status message if the error reason tag is invalid_header or file_error.

🦋 log_terms/2 🦋

log_terms(Log, TermList) ->
                    ok | {error, Reason :: log_error_rsn()}
                    when Log :: log(), TermList :: [term()].

Synchronously appends a list of items to an internally formatted log. ✨

Using this is more efficient than log/2. The list you hand over gets split into as large sublists as possible (limited by the size of wrap log files), and each sublist is logged as one single item, which cuts down the overhead. It's giving optimization. 👑

Owners subscribing to notifications get notified of an error with an error_status message if the error reason tag is invalid_header or file_error.

🌈 next_file/1 🌈

next_file(Log) -> ok | {error, next_file_error_rsn()}
                    when Log :: log().

For wrap logs, this forces the disk log to start logging to the next log file. You could use it, say, with change_size/2 to reduce the amount of disk space the disk log is hogging. 🧹

Owners subscribing to notifications normally receive a wrap message, but if an error occurs with a reason tag of invalid_header or file_error, an error_status message is sent instead.

For rotate logs, it forces rotation of the currently active log file, compresses it, and opens a new active file for logging. Spin that cycle. 🔄

💖 open/1 💖

open(ArgL) -> open_ret() when ArgL :: dlog_options().

Opens a new disk log file for reading or writing. Say hi to your new log. 🫶

Parameter ArgL is a list of the following options:

  • {name, Log} — specifies the log name. You must pass this name as a parameter in all subsequent logging operations. A name must always be supplied. No name, no log. Periodt. 🎀
  • {file, FileName} — specifies the name of the file to be used for logged terms. If you omit this value and the log name is an atom or a string, the filename defaults to lists:concat([Log, ".LOG"]) for halt logs.

For wrap logs, this is the base name of the files. Each file in a wrap log is called <FileName>.N, where N is an integer. Each wrap log also has two files called <FileName>.idx and <FileName>.siz. 🗂️

For rotate logs, this is the name of the active log file. The compressed files are named <FileName>.N.gz, where N is an integer and <FileName>.0.gz is the latest compressed log file. All the compressed files get renamed at each rotation so that the latest files have the smallest index. The maximum value for N is the value of MaxNoFiles minus 1.

  • {linkto, LinkTo} — if LinkTo is a pid, it becomes an owner of the log. If LinkTo is none, the log records that it's used anonymously by some process by incrementing the users counter. By default, the process that calls open/1 owns the log. 💫
  • {repair, Repair} — if Repair is true, the current log file is repaired, if needed. As the restoration kicks off, a message is output on the error log. If false is specified, no automatic repair is attempted; instead the tuple {error, {need_repair, Log}} is returned if you try to open a corrupt log file. If truncate is specified, the log file becomes truncated, creating an empty log, regardless of previous content. Defaults to true, which has no effect on logs opened in read-only mode. 😭
  • {type, Type} — the log type. Defaults to halt.
  • {format, Format} — disk log format. Defaults to internal.
  • {size, Size} — log size. 📏

When a halt log has reached its maximum size, all attempts to log more items are rejected. Defaults to infinity, which for halt implies that there is no maximum size.

For wrap and rotate logs, parameter Size can be a pair {MaxNoBytes, MaxNoFiles}. For wrap logs it can also be infinity. In that latter case, if the files of an existing wrap log with the same name can be found, the size is read from the existing wrap log, otherwise an error is returned.

Wrap logs write at most MaxNoBytes bytes on each file and use MaxNoFiles files before starting all over with the first wrap log file. Regardless of MaxNoBytes, at least the header (if there is one) and one item are written on each wrap log file before wrapping to the next file. 🌀

The first time an existing wrap log is opened, that is, when the disk log process is created, the value of the option size is allowed to differ from the current log size, and the size of the disk log is changed as per change_size/2.

When opening an existing wrap log, it is not necessary to supply a value for option size, but if the log is already open, that is, the disk log process exists, the supplied value must equal the current log size, otherwise the tuple {error, {size_mismatch, CurrentSize, NewSize}} is returned.

NOTE: Before Erlang/OTP 24.0, the supplied value of option size was to be equal to the current log size when opening an existing wrap log for the first time, that is, when creating the disk log process. Old behavior, bestie, we've moved on. 🙈

Rotate logs write at most MaxNoBytes bytes on the active log file and keep the latest MaxNoFiles compressed files. Regardless of MaxNoBytes, at least the header (if there is one) and one item are written on each rotate log file before rotation.

When opening an already open halt log, option size is ignored.

  • {notify, boolean()} — if true, the log owners are notified when certain log events occur. Defaults to false. Owners get sent one of the following messages when an event happens: 🔔
    • {disk_log, Node, Log, {wrap, NoLostItems}} — sent when a wrap log has filled up one of its files and a new file is opened. NoLostItems is the number of previously logged items that were lost when truncating existing files.
    • {disk_log, Node, Log, {truncated, NoLostItems}} — sent when a log is truncated or reopened. For halt logs, NoLostItems is the number of items written on the log since the disk log process was created. For wrap logs, NoLostItems is the number of items on all wrap log files.
    • {disk_log, Node, Log, {read_only, Items}} — sent when an asynchronous log attempt is made to a log file opened in read-only mode. Items is the items from the log attempt.
    • {disk_log, Node, Log, {blocked_log, Items}} — sent when an asynchronous log attempt is made to a blocked log that does not queue log attempts. Items is the items from the log attempt.
    • {disk_log, Node, Log, {format_external, Items}} — sent when function alog/2 or alog_terms/2 is used for internally formatted logs. Items is the items from the log attempt.
    • {disk_log, Node, Log, full} — sent when an attempt to log items to a wrap log would write more bytes than the limit set by option size. 💀
    • {disk_log, Node, Log, {error_status, Status}} — sent when the error status changes. The error status is defined by the outcome of the last attempt to log items to the log, or to truncate the log, or the last use of function sync/1, inc_wrap_file/1, or change_size/2. Status is either ok or {error, Error}, the former being the initial value. 🌟

…and here's the rest of the open/1 option squad, bestie, we're almost done 💅

  • {head, Head} — drops a header at the very top of the log file. If your log is a wrap or rotate log, then Head gets written first in every new file. Head needs to be a term if the format is internal, otherwise it's an iodata/0. Defaults to none, which literally means no header gets written on the file at all. 🎀
  • {head_func, {M,F,A}} — names a function to run every single time a new log file gets opened. The call M:F(A) is assumed to return {ok, Head}, and that Head is written first in each file. Again: Head is a term if the format is internal, otherwise an iodata/0. ✨
  • {mode, Mode} — says whether the log opens read-only or read-write. Defaults to read_write.
  • {quiet, Boolean} — says whether messages get sent to error_logger when there are recoverable errors with the log files. Defaults to false.

open/1 returns {ok, Log} when the log file is successfully opened. But if the file got successfully repaired, you get the tuple {repaired, Log, {recovered, Rec}, {badbytes, Bad}} instead, where Rec is the number of whole Erlang terms found in the file and Bad is the number of bytes in the file that are non-Erlang terms. 😭 (repair arc, we love to see it)

When a disk log is opened in read-write mode, any existing log file gets checked for. No file? Brand new empty log, periodt. File exists? It's opened at the position after the last logged item, and logging of items starts from there. And if the format is internal and the existing file is not recognized as an internally formatted log, you get a tuple {error, {not_a_log_file, FileName}} back.

open/1 cannot be used for changing the values of options of an open log — absolutely not, no. When there are prior owners or users of a log, all option values except name, linkto, and notify are only checked against the values supplied before as option values to function open/1, change_header/2, change_notify/3, or change_size/2. So: none of the options except name is mandatory. And if some specified value differs from the current value, you get a tuple {error, {arg_mismatch, OptionName, CurrentValue, Value}}.

NOTE: If an owner attempts to open a log as owner once again, it is acknowledged with the return value {ok, Log}, but the state of the disk log is not affected. (it's giving "yes bestie, nothing changed" 💀)

A log file can be opened more than once by giving different values to option name, or by using the same file when opening a log on different nodes. But it is up to the user of module disk_log to ensure that not more than one disk log process has write access to any file, otherwise the file can be corrupted. 🙈

If an attempt to open a log file for the first time fails, the disk log process terminates with the EXIT message {{failed,Reason},[{disk_log,open,1}]}. The function returns {error, Reason} for all other errors.

pid2name(Pid) -> {ok, Log} | undefined

pid2name(Pid) -> {ok, Log} | undefined
                  when Pid :: pid(), Log :: log().

Returns the log name given the pid of a disk log process on the current node, or undefined if the specified pid is not a disk log process. 🌸

This function is meant to be used for debugging only. iykyk 🔥

reopen(Log, File) -> ok | {error, reopen_error_rsn()}

reopen(Log, File) -> ok | {error, reopen_error_rsn()}
                  when Log :: log(), File :: file:filename().

Equivalent to reopen(Log, File, Head), where Head is the Head specified in open/1.

reopen(Log, File, Head) -> ok | {error, reopen_error_rsn()}

reopen(Log, File, Head) -> ok | {error, reopen_error_rsn()}
                  when
                      Log :: log(),
                      File :: file:filename(),
                      Head :: term().

Renames an internally formatted log file to File and then recreates a new log file. If a wrap/rotate log exists, File is used as the base name of the renamed files. 🦋

Writes the value of Head first in the newly opened log file. The header argument is used only once — next time a wrap/rotate log file is opened, the header given to open/1 is used instead.

Owners subscribing to notifications receive a truncate message. 💌

Upon failure to reopen the log, the disk log process terminates with the EXIT message {{failed,Error},[{disk_log,Fun,Arity}]}. Other processes having requests queued receive the message {disk_log, Node, {error, disk_log_stopped}}. 😭

sync(Log) -> ok | {error, sync_error_rsn()}

sync(Log) -> ok | {error, sync_error_rsn()} when Log :: log().

Ensures that the contents of the log are written to the disk. This is usually a rather expensive operation, so like… use it with intention, bestie. 💸

truncate(Log) -> ok | {error, trunc_error_rsn()}

truncate(Log) -> ok | {error, trunc_error_rsn()} when Log :: log().

Equivalent to truncate(Log, Head), where Head is the Head specified in open/1.

This function can be used for both internally and externally formatted logs. ✨

truncate(Log, Head) -> ok | {error, trunc_error_rsn()}

truncate(Log, Head) -> ok | {error, trunc_error_rsn()}
                  when Log :: log(), Head :: term().

Removes all items from an internally formatted disk log. The argument Head is written first in the newly truncated log. 🧹

The header argument is used only once. Next time a wrap/rotate log file is opened, the header given to open/1 is used.

Owners subscribing to notifications receive a truncate message. 🎀

If the attempt to truncate the log fails, the disk log process terminates with the EXIT message {{failed,Reason},[{disk_log,Fun,Arity}]}. Other processes having requests queued receive the message {disk_log, Node, {error, disk_log_stopped}}.

unblock(Log) -> ok | {error, unblock_error_rsn()}

unblock(Log) -> ok | {error, unblock_error_rsn()}
                  when Log :: log().

Unblocks a log. And here's the rule: a log can only be unblocked by the blocking process. That's it, that's the whole vibe. 🔒

and that's the whole page, bestie!! you ate it, every last option, every last error tuple 💖 you now officially understand disk_log better than most people and that's iconic 🫶

This one came from Ericsson AB, it's part of kernel 11.0.3, and it lives in section 3 of the manual as disk_log(3) — go off, you gorgeous nerd 👑

📦 where this page comes from