ms_transform(3)

✨ ms_transform ✨

okay bestie, welcome to the ms_transform manual page — it's an Erlang Module Definition, and it's literally the glow-up your match specifications have been waiting for 🫶

---

🌸 NAME 🌸

ms_transform — a parse transformation that translates fun syntax into match specifications. periodt. 💅

---

🦋 DESCRIPTION 🦋

So here's the tea, bestie: this module provides the parse transformation that makes calls to ets and dbg:fun2ms/1 translate into literal match specifications. It also provides the back end for the same functions when called from the Erlang shell. ✨

The translation from funs to match specifications is accessed through the two "pseudo functions" ets:fun2ms/1 and dbg:fun2ms/1.

ngl, everyone trying to use ets:select/2 or dbg seems to end up reading this manual page, so like… this description is an introduction to the concept of match specifications. It's giving "required reading" energy 💀

Read the whole manual page if it is the first time you are using the transformations. no skimming, bestie, I see you 👀

Match specifications are used more or less as filters. They resemble usual Erlang matching in a list comprehension or in a fun used with lists:foldl/3, and so on. However — and this is the part that's lowkey tragic — the syntax of pure match specifications is awkward, because they are made up purely by Erlang terms, and the language has no syntax to make the match specifications more readable. 😭

As the execution and structure of the match specifications are like that of a fun, it is more straightforward to write it using the familiar fun syntax and to have that translated into a match specification automatically. A real fun is clearly more powerful than the match specifications allow, but bearing the match specifications in mind, and what they can do, it is still more convenient to write it all as a fun. This module contains the code that translates the fun syntax into match specification terms. slay. 💖

---

🎀 Example 1 🎀

Using ets:select/2 and a match specification, one can filter out rows of a table and construct a list of tuples containing relevant parts of the data in these rows. You could use ets:foldl/3 instead, but the ets:select/2 call is far more efficient. Without the translation provided by ms_transform, you must struggle with writing match specifications terms to accommodate this. (struggle. yes. that's the word. 🙈)

Consider a simple table of employees:

-record(emp, {empno,     %Employee number as a string, the key
              surname,   %Surname of the employee
              givenname, %Given name of employee
              dept,      %Department, one of {dev,sales,prod,adm}
              empyear}). %Year the employee was employed

We create the table using:

ets:new(emp_tab, [{keypos,#emp.empno},named_table,ordered_set]).

We fill the table with randomly chosen data:

[{emp,"011103","Black","Alfred",sales,2000},
 {emp,"041231","Doe","John",prod,2001},
 {emp,"052341","Smith","John",dev,1997},
 {emp,"076324","Smith","Ella",sales,1995},
 {emp,"122334","Weston","Anna",prod,2002},
 {emp,"535216","Chalker","Samuel",adm,1998},
 {emp,"789789","Harrysson","Joe",adm,1996},
 {emp,"963721","Scott","Juliana",dev,2003},
 {emp,"989891","Brown","Gabriel",prod,1999}]

Assuming that we want the employee numbers of everyone in the sales department, there are several ways. Let's vibe-check them all. 💅

ets:match/2 can be used:

1> ets:match(emp_tab, {'_', '$1', '_', '_', sales, '_'}).
[["011103"],["076324"]]

ets:match/2 uses a simpler type of match specification, but it is still unreadable, and one has little control over the returned result. It is always a list of lists. not the serve we wanted 😭

ets:foldl/3 or ets:foldr/3 can be used to avoid the nested lists:

ets:foldr(fun(#emp{empno = E, dept = sales},Acc) -> [E | Acc];
             (_,Acc) -> Acc
          end,
          [],
          emp_tab).

The result is ["011103","076324"]. The fun is straightforward, so the only problem is that all the data from the table must be transferred from the table to the calling process for filtering. That is inefficient compared to the ets:match/2 call where the filtering can be done "inside" the emulator and only the result is transferred to the process. 👑

Consider a "pure" ets:select/2 call that does what ets:foldr does:

ets:select(emp_tab, [{#emp{empno = '$1', dept = sales, _='_'},[],['$1']}]).

Although the record syntax is used, it is still hard to read and even harder to write. The first element of the tuple, #emp{empno = '$1', dept = sales, _='_'}, tells what to match. Elements not matching this are not returned, as in the ets:match/2 example. The second element, the empty list, is a list of guard expressions, which we do not need. The third element is the list of expressions constructing the return value (in ETS this is almost always a list containing one single term). In our case '$1' is bound to the employee number in the head (first element of the tuple), and hence the employee number is returned. The result is ["011103","076324"], as in the ets:foldr/3 example, but the result is retrieved much more efficiently in terms of execution speed and memory consumption. 🔥

Using ets:fun2ms/1, we can combine the ease of use of the ets:foldr/3 and the efficiency of the pure ets:select/2 example:

-include_lib("stdlib/include/ms_transform.hrl").

ets:select(emp_tab, ets:fun2ms(
                      fun(#emp{empno = E, dept = sales}) ->
                              E
                      end)).

This example requires no special knowledge of match specifications to understand. The head of the fun matches what you want to filter out and the body returns what you want returned. As long as the fun can be kept within the limits of the match specifications, there is no need to transfer all table data to the process for filtering as in the ets:foldr/3 example. It is easier to read than the ets:foldr/3 example, as the select call in itself discards anything that does not match, while the fun of the ets:foldr/3 call needs to handle both the elements matching and the ones not matching. it's giving effortless. ✨

In the ets:fun2ms/1 example above, it is needed to include ms_transform.hrl in the source code, as this is what triggers the parse transformation of the ets:fun2ms/1 call to a valid match specification. This also implies that the transformation is done at compile time (except when called from the shell) and therefore takes no resources in runtime. That is, although you use the more intuitive fun syntax, it gets as efficient in runtime as writing match specifications by hand. 🌈

---

🧚 Example 2 🧚

Assume that we want to get all the employee numbers of employees hired before year 2000. Using ets:match/2 is not an alternative here, as relational operators cannot be expressed there. Once again, ets:foldr/3 can do it (slowly, but correct):

ets:foldr(fun(#emp{empno = E, empyear = Y},Acc) when Y < 2000 -> [E | Acc];
             (_,Acc) -> Acc
          end,
          [],
          emp_tab).

The result is ["052341","076324","535216","789789","989891"], as expected. The equivalent expression using a handwritten match specification would look like this:

ets:select(emp_tab, [{#emp{empno = '$1', empyear = '$2', _='_'},
                     [{'<', '$2', 2000}],
                     ['$1']}]).

This gives the same result. [{'<', '$2', 2000}] is in the guard part and therefore discards anything that does not have an empyear (bound to '$2' in the head) less than 2000, as the guard in the foldr/3 example. 🫶

We write it using ets:fun2ms/1:

-include_lib("stdlib/include/ms_transform.hrl").

ets:select(emp_tab, ets:fun2ms(
                      fun(#emp{empno = E, empyear = Y}) when Y < 2000 ->
                           E
                      end)).

---

🍓 Example 3 🍓

Assume that we want the whole object matching instead of only one element. One alternative is to assign a variable to every part of the record and build it up once again in the body of the fun, but the following is easier:

ets:select(emp_tab, ets:fun2ms(
                      fun(Obj = #emp{empno = E, empyear = Y})
                         when Y < 2000 ->
                              Obj
                      end)).

As in ordinary Erlang matching, you can bind a variable to the whole matched object using a "match inside the match", that is, a =. Unfortunately in funs translated to match specifications, it is allowed only at the "top-level", that is, matching the whole object arriving to be matched into a separate variable. If you are used to writing match specifications by hand, we mention that variable A is simply translated into '$_'. Alternatively, pseudo function object/0 also returns the whole matched object, see section Warnings and Restrictions. 💜

---

💀 Example 4 💀

This example concerns the body of the fun. Assume that all employee numbers beginning with zero (0) must be changed to begin with one (1) instead, and that we want to create the list [{<Old empno>,<New empno>}]:

ets:select(emp_tab, ets:fun2ms(
                      fun(#emp{empno = [$0 | Rest] }) ->
                              {[$0|Rest],[$1|Rest]}
                      end)).

This query hits the feature of partially bound keys in table type ordered_set, so that not the whole table needs to be searched, only the part containing keys beginning with 0 is looked into. efficiency?? we love to see it 🔥

---

👑 Example 5 👑

The fun can have many clauses. Assume that we want to do the following:

• If an employee started before 1997, return the tuple {inventory, <employee number>}.
• If an employee started 1997 or later, but before 2001, return {rookie, <employee number>}.
• For all other employees, return {newbie, <employee number>}, except for those named Smith as they would be affronted by anything other than the tag guru and that is also what is returned for their numbers: {guru, <employee number>}. (Smiths stay winning, I guess 💅)

This is accomplished as follows:

ets:select(emp_tab, ets:fun2ms(
                      fun(#emp{empno = E, surname = "Smith" }) ->
                              {guru,E};
                         (#emp{empno = E, empyear = Y}) when Y < 1997  ->
                              {inventory, E};
                         (#emp{empno = E, empyear = Y}) when Y > 2001  ->
                              {newbie, E};
                         (#emp{empno = E, empyear = Y}) -> % 1997 -- 2001
                              {rookie, E}
                      end)).

The result is as follows:

[{rookie,"011103"},
 {rookie,"041231"},
 {guru,"052341"},
 {guru,"076324"},
 {newbie,"122334"},
 {rookie,"535216"},
 {inventory,"789789"},
 {newbie,"963721"},
 {rookie,"989891"}]

---

✨ Useful BIFs ✨

okay so like… what else can you even do in there, bestie? 💅 short answer: go read the match specification docs in the ERTS User's Guide. but fr fr, here's the quick lil' tour of the most useful "built-in functions" you get to play with when your fun gets translated into a match spec by ets:fun2ms/1.

lowkey important, highkey a rule: you literally cannot call any function other than the ones that match specifications allow. no "normal" Erlang code is getting executed by the fun that ets:fun2ms/1 translates. the fun is capped exactly at match-spec power 😭 which is kinda sad, but that's the price you pay for ets:select/2 being so much faster than ets:foldl/foldr. periodt.

The head of the fun is a head that matches (or doesn't match) one parameter — one object from the table you're selecting out of. That object is always either a single variable (can be _) or a tuple, since that's what ETS, Dets, and Mnesia tables hold. And the match spec that ets:fun2ms/1 hands back? it works with dets:select/2 and mnesia:select/2 too, not just ets:select/2 🫶. Also using = in the head at the top-level is allowed (and honestly encouraged ✨).

The guard section can hold any guard expression Erlang supports. Here's the list of BIFs and expressions you can use:

• 🦋 Type tests: is_atom, is_float, is_integer, is_list, is_number, is_pid, is_port, is_reference, is_tuple, is_binary, is_function, is_record
• 🎀 Boolean operators: not, and, or, andalso, orelse
• ⭐ Relational operators: >, >=, <, =<, =:=, ==, =/=, /=
• 🌸 Arithmetic: +, -, *, div, rem
• 💜 Bitwise operators: band, bor, bxor, bnot, bsl, bsr
• 👑 The guard BIFs: abs, element, hd, length, node, round, size, byte_size, tl, trunc, binary_part, self

and here's a lil' plot twist for you: unlike with "handwritten" match specs, the is_record guard behaves exactly like it does in ordinary Erlang code. slay 💅

Semicolons (;) inside guards are allowed too — and the result is (as you'd expect) one "match specification clause" for every semicolon-separated chunk of the guard. The semantics are identical to Erlang's semantics. no cap.

The body of the fun is what builds your resulting value. When you're selecting out of tables you usually construct a fitting term right there, using ordinary Erlang term construction — tuple parentheses, list brackets, and variables you matched out in the head, plus the occasional constant if you're feeling cute 🎀. Whatever expressions are allowed in guards are allowed here too, but there are no special functions available except object and bindings (more on those further down), which return the whole matched object and all known variable bindings respectively.

ikyk but: the dbg variants of match specs are kind of imperative-flavored when it comes to the body — the ETS dialect is not. The fun body for ets:fun2ms/1 returns a result with zero side effects. And since matching (=) inside a match spec body isn't allowed (performance reasons, bestie, it's always performance reasons), the only thing left is, more or less… term construction. that's it. that's the whole vibe.

---

🐛 Example with dbg 🐛

This section is all about the slightly different match specs that dbg:fun2ms/1 translates 👀.

Same reasons for using the parse transformation apply to dbg — maybe even more so, because filtering with Erlang code while tracing is just not a good idea (unless it's afterwards, like if you trace to file). The concept is basically ets:fun2ms/1's twin, except you usually run it straight from the shell (which, btw, you can also do with ets:fun2ms/1).

Here's an example module to trace on:

-module(toy).

-export([start/1, store/2, retrieve/1]).

start(Args) ->
    toy_table = ets:new(toy_table, Args).

store(Key, Value) ->
    ets:insert(toy_table, {Key,Value}).

retrieve(Key) ->
    [{Key, Value}] = ets:lookup(toy_table, Key),
    Value.

So during model testing, the first test gives back {badmatch,16} in {toy,start,1}… WHY??? 😭

We're suspicious of that ets:new/2 call, 'cause we're matching hard on the return value, but we only care about the specific new/2 call where the first parameter is toy_table. So we start a default tracer on the node:

1> dbg:tracer().
{ok,<0.88.0>}

We turn on call tracing for all processes — we want a pretty restrictive trace pattern, so there's no need to only trace a few processes (usually it's not, anyway):

2> dbg:p(all,call).
{ok,[{matched,nonode@nohost,25}]}

Now we specify the filter: we wanna see calls that look like ets:new(toy_table, <something>):

3> dbg:tp(ets,new,dbg:fun2ms(fun([toy_table,_]) -> true end)).
{ok,[{matched,nonode@nohost,1},{saved,1}]}

See what happened there? 👀 The fun used with dbg:fun2ms/1 takes a single list as its parameter instead of a single tuple. That list matches the list of parameters handed to the traced function. You can also use just a single variable. The body of the fun expresses, in a more imperative way, the actions to take if the fun head (and the guards) match. Here we return true — only because the body of a fun literally can't be empty. The return value gets discarded anyway. 💀

The following trace output shows up during the test:

(<0.86.0>) call ets:new(toy_table, [ordered_set])

Now say we still haven't found the problem, and we wanna see what ets:new/2 actually returns. Slightly different trace pattern:

4> dbg:tp(ets,new,dbg:fun2ms(fun([toy_table,_]) -> return_trace() end)).

And during the test we get:

(<0.86.0>) call ets:new(toy_table,[ordered_set])
(<0.86.0>) returned from ets:new/2 -> 24

So the call to return_trace gives you a trace message when the function returns 🎀. It only applies to that specific function call that triggered the match spec (and matched the head/guards of the match spec). And ngl, this is by far the most common call you'll see in the body of a dbg match spec. iconic behavior honestly.

The test now fails with {badmatch,24} because the atom toy_table doesn't match the number returned for an unnamed table. So mystery solved 🔍 — the table is supposed to be named, and the arguments the test program supplies don't include named_table. We rewrite the start function:

start(Args) ->
    toy_table = ets:new(toy_table, [named_table|Args]).

With the exact same tracing turned on, we now get:

(<0.86.0>) call ets:new(toy_table,[named_table,ordered_set])
(<0.86.0>) returned from ets:new/2 -> toy_table

Okay so assume the module passes all testing and ships into the system 🚀. After a while, someone notices the toy_table table keeps growing while the system runs, and that there are a lot of elements with atoms as keys. We expected only integer keys, and so does the rest of the system — but clearly not the entire system 🙈. So we turn on call tracing and try to spot calls to the module with an atom as the key:

1> dbg:tracer().
{ok,<0.88.0>}
2> dbg:p(all,call).
{ok,[{matched,nonode@nohost,25}]}
3> dbg:tpl(toy,store,dbg:fun2ms(fun([A,_]) when is_atom(A) -> true end)).
{ok,[{matched,nonode@nohost,1},{saved,1}]}

We use dbg:tpl/3 to make sure we catch local calls too (assume the module has grown since that smaller version, and we're not sure the atom-inserting isn't happening locally). Rule of thumb, bestie: when in doubt, always use local call tracing. 💅

Now assume nothing happens when tracing this way. The function is never called with those parameters. So we conclude someone else (some other module) is doing it, and we realize we've gotta trace ets:insert/2 instead — and we want to see the calling function. You grab the calling function with the match spec function caller. And to get it into the trace message, you need the match spec function message. The filter call looks like this (looking for calls to ets:insert/2):

4> dbg:tpl(ets,insert,dbg:fun2ms(fun([toy_table,{A,_}]) when is_atom(A) ->
                                  message(caller())
                                end)).
{ok,[{matched,nonode@nohost,1},{saved,2}]}

Now the caller shows up in the "additional message" part of the trace output, and after a while this appears:

(<0.86.0>) call ets:insert(toy_table,{garbage,can}) ({evil_mod,evil_fun,2})

and there it is 😤 — function evil_fun of the evil_mod module, with arity 2, is the one causing all this trouble. caught in 4k.

This example walks you through the most-used calls in match specs for dbg. The other, way more esoteric calls are listed and explained over in Match specifications in Erlang in the ERTS User's Guide, 'cause they're beyond the scope of this description 🌈.

---

🚨 Warnings and Restrictions 🚨

The following warnings and restrictions apply to the funs used with ets:fun2ms/1 and dbg:fun2ms/1. bestie pls read these, i'm begging 🫶

WARNING: To use the pseudo functions that trigger the translation, make sure to include the header file ms_transform.hrl in the source code. If you fail to do so, you might get runtime errors instead of compile-time ones — because the expression can totally be valid as a plain Erlang program without any translation. sneaky. 💀

WARNING: The fun must be literally constructed inside the parameter list of the pseudo functions. You cannot bind the fun to a variable first and then hand it to ets:fun2ms/1 or dbg:fun2ms/1. For example, ets:fun2ms(fun(A) -> A end) works, but F = fun(A) -> A end, ets:fun2ms(F) does not. The latter gives you a compile-time error if the header is included — otherwise a runtime error. no cap.

There are many restrictions on the fun that gets translated into a match spec. To keep it simple: you can't use anything in the fun that you can't use in a match spec. Which means, among others, these restrictions:

• 🚫 Functions written in Erlang can't be called — not local functions, not global functions, not even real funs.
• 🚫 Everything written as a function call gets translated into a match spec call to a built-in function. So the call is_list(X) becomes {'is_list', '$1'} ('$1' is just an example, the numbering can vary ✨). If you try to call something that isn't a match spec built-in, it causes an error.
• 🚫 Variables that show up in the head of the fun get replaced by match spec variables in order of occurrence, so the fragment fun({A,B,C}) becomes {'$1', '$2', '$3'}, and so on. Every occurrence of such a variable inside the match spec gets replaced by a match spec variable the same way — so the fun fun({A,B}) when is_atom(A) -> B end is translated into [{{'$1','$2'},[{is_atom,'$1'}],['$2']}]. cute, right? 💅
• Vars that aren't in the head? They get imported from the environment and turned into match specification const expressions. Live from the shell, no cap:

1> X = 25.
25
2> ets:fun2ms(fun({A,B}) when A > X -> B end).
[{{'$1','$2'},[{'>','$1',{const,25}}],['$2']}]

• Matching with = in the body? Absolutely not. The only place = is allowed is the top-level in the head of the fun. Shell round two:

1> ets:fun2ms(fun({A,[B|C]} = D) when A > B -> D end).
[{{'$1',['$2'|'$3']},[{'>','$1','$2'}],['$_']}]
2> ets:fun2ms(fun({A,[B|C]=D}) when A > B -> D end).
Error: fun with head matching ('=' in head) cannot be translated into
match_spec
{error,transform_error}
3> ets:fun2ms(fun({A,[B|C]}) when A > B -> D = [B|C], D end).
Error: fun with body matching ('=' in body) is illegal as match_spec
{error,transform_error}

Every variable is bound in the head of a match specification, bestie, so the translator literally cannot allow multiple bindings. The one special case — matching on the top-level — makes the variable bind to '$_' in the resulting match specification. It's there so you have a more natural way to grab the whole matched object. Pseudo function object() does the same job, see below 🫶

These two translate exactly the same, no difference fr fr:

ets:fun2ms(fun({a,_} = A) -> A end).
ets:fun2ms(fun({a,_}) -> object() end).

• The special match spec variables '$_' and '$*' are reachable through the pseudo functions object() (for '$_') and bindings() (for '$*'). So like, you can translate this ets:match_object/2 call into an ets:select/2 call:

ets:match_object(Table, {'$1',test,'$2'}).

Which is the same as:

ets:select(Table, ets:fun2ms(fun({A,test,B}) -> object() end)).

In this little baby case though? The first one is probably the nicer read, honestly 🍓

Conceptually the ets:select/2 call looks like this in the resulting code:

ets:select(Table, [{{'$1',test,'$2'},[],['$_']}]).

And matching on the top-level of the fun head can be a more natural way to reach '$_' — scroll up, it's right there 👆

• Term constructions and literals get translated as much as is needed to land them in a valid match specification. So tuples become match specification tuple constructions (a one element tuple containing the tuple 💀 yes really) and constant expressions get used when importing variables from the environment. Records also get translated into plain tuple constructions, calls to element, and so on. The guard test is_record/2 is translated into match specification code using the three parameter version that's built into match specification — so is_record(A,t) becomes {is_record,'$1',t,5} if the record size of record type t is 5. Numbers don't lie ✨
• Language constructions like case, if, and catch? Not present in match specifications, so not allowed. Periodt. 🙅‍♀️
• If the header file ms_transform.hrl is not included, the fun is not translated, which can result in a runtime error (depending on whether the fun is valid in a pure Erlang context). So bestie, please make sure the header is included when you use ets and dbg:fun2ms/1 in compiled code 🙈

I'm so serious about that one. Missing header = surprise runtime error, and nobody wants that plot twist 😭

• Which pseudo function is doing the translating matters: if it's ets:fun2ms/1, the head of the fun must contain a single variable or a single tuple. If it's dbg:fun2ms/1, the head of the fun must contain a single variable or a single list. Two different flavours, don't mix them up 💅

---

The translation from funs to match specifications happens at compile time, so using these pseudo functions does not affect runtime performance. Zero. Nada. Iconic behavior honestly 👑

For more info about match specifications, go read the Match specifications in Erlang chapter in the ERTS User's Guide. It's giving required reading 🎀

---

Okay that's the whole page bestie!! 💖 This one came straight from Ericsson AB, shipping with stdlib 8.0.4, ms_transform(3) — and the footer didn't even give us a date, which is lowkey rude of it, but we move 🦋 Go forth and translate your funs, you absolute main character 🌸✨