LIBEXPECT(3)

🦋 libexpect(3) 🦋 — aka the Library Functions Manual entry you're about to be obsessed with

okay so bestie, buckle up, this page is LONG (it literally comes in three parts and this is part one, so don't you dare expect a sign-off at the end 💖). We're talking about the programmed dialogue library with interactive programs. Yes. It talks to interactive programs FOR you. Main character energy, no cap.

---

✨ NAME ✨

libexpect — programmed dialogue library with interactive programs 💅

---

🌸 DESCRIPTION 🌸

So this library is a whole collection of functions that let you use Expect in two different ways, and you get to pick your fighter:

• as a Tcl extension, or
• directly from C or C++ (without Tcl at all, living your raw unhinged life)

Adding Expect as a Tcl extension is very short and simple, so that's the one we're covering first. Slay.

---

⭐ SYNOPSIS ⭐

#include expect_tcl.h
Expect_Init(interp);

cc files... -lexpect5.20 -ltcl7.5 -lm

note bestie: library versions may differ in the actual release, so don't come for me if your numbers are different 🙈

The Expect_Init function adds expect commands to the named interpreter. It's polite about it — it avoids overwriting commands that already exist. HOWEVER (and this is the tea ☕) aliases beginning with "exp_" are always created for expect commands. So for example, "send" can be used as "exp_send". Periodt.

Generally, you should only call Expect commands via Tcl_Eval. Certain auxiliary functions may be called directly, and they're summarized below. They may be useful in constructing your own main. Look at the file exp_main_exp.c in the Expect distribution as a prototype main. Another prototype is tclAppInit.c in the Tcl source distribution. A prototype for working with Tk is in exp_main_tk.c in the Expect distribution. Prototypes everywhere, it's giving free samples at the mall 💀

int exp_cmdlinecmds;
int exp_interactive;
FILE *exp_cmdfile;
char *exp_cmdfilename;
int exp_tcl_debugger_available;

void exp_parse_argv(Tcl_Interp *,int argc,char **argv);
int  exp_interpreter(Tcl_Interp *);
void exp_interpret_cmdfile(Tcl_Interp *,FILE *);
void exp_interpret_cmdfilename(Tcl_Interp *,char *);
void exp_interpret_rcfiles(Tcl_Interp *,int my_rc,int sys_rc);
char *    exp_cook(char *s,int *len);
void (*exp_app_exit)EXP_PROTO((Tcl_Interp *);
void exp_exit(Tcl_Interp *,int status);
void exp_exit_handlers(Tcl_Interp *);
void exp_error(Tcl_Interp,char *,...);

Let's do a vibe check on each of these:

• exp_cmdlinecmds is 1 if Expect has been invoked with commands on the program command-line (using "-c" for example). 🎀
• exp_interactive is 1 if Expect has been invoked with the -i flag or if no commands or script is being invoked. Lowkey the "just hang out and chat" mode.
• exp_cmdfile is a stream from which Expect will read commands. 🌊
• exp_cmdfilename is the name of a file which Expect will open and read commands from. 📂
• exp_tcl_debugger_available is 1 if the debugger has been armed. 💀 armed and dangerous
• exp_parse_argv reads the representation of the command line. Based on what is found, any of the other variables listed here are initialized appropriately. It's the little matchmaker that sets everyone up. 🫶
• exp_interpreter interactively prompts the user for commands and evaluates them. It literally asks you what you want, iconic behaviour.
• exp_interpret_cmdfile reads the given stream and evaluates any commands found.
• exp_interpret_cmdfilename opens the named file and evaluates any commands found.
• exp_interpret_rcfiles reads and evalutes the .rc files. If my_rc is zero, then ~/.expectrc is skipped. If sys_rc is zero, then the system-wide expectrc file is skipped. Skipping is valid, we support that. 🙈
• exp_cook returns a static buffer containing the argument reproduced with newlines replaced by carriage-return linefeed sequences. The primary purpose of this is to allow messages to be produced without worrying about whether the terminal is in raw mode or cooked mode. (Yes, "cooked". Like, the terminal is cooked. 💅) If length is zero, it is computed via strlen.
• exp_error is a printf-like function that writes the result to interp->result.

---

🌈 SYNOPSIS 🌈

okay so now we're doing the C library part, the second flavour, the remix if you will ✨

#include <expect.h>

int
exp_spawnl(file, arg0 [, arg1, ..., argn] (char *)0);
char *file;
char *arg0, *arg1, ... *argn;

int
exp_spawnv(file,argv);
char *file, *argv[ ];

int
exp_spawnfd(fd);
int fd;

FILE *
exp_popen(command);
char *command;

extern int exp_pid;
extern int exp_ttyinit;
extern int exp_ttycopy;
extern int exp_console;
extern char *exp_stty_init;
extern void (*exp_close_in_child)();
extern void (*exp_child_exec_prelude)();
extern void exp_close_tcl_files();

cc files... -lexpect -ltcl -lm

---

💜 DESCRIPTION 💜 (part two, electric boogaloo)

exp_spawnl and exp_spawnv fork a new process so that its stdin, stdout, and stderr can be written and read by the current process. It's a whole situationship. file is the name of a file to be executed. The arg pointers are null-terminated strings. Following the style of execve(), arg0 (or argv[0]) is customarily a duplicate of the name of the file.

Four interfaces are available, bestie, choose wisely:

• exp_spawnl is useful when the number of arguments is known at compile time. 🔒
• exp_spawnv is useful when the number of arguments is not known at compile time. 🎲
• exp_spawnfd is useful when an open file descriptor is already available as a source. 🔌
• exp_popen is explained later on. The suspense... the drama... 🍓

If the process is successfully created, a file descriptor is returned which corresponds to the process's stdin, stdout and stderr. A stream may be associated with the file descriptor by using fdopen(). (This should almost certainly be followed by setbuf() to unbuffer the I/O. Like, do it. Do the setbuf.)

Closing the file descriptor will typically be detected by the process as an EOF. Once such a process exits, it should be waited upon (via wait) in order to free up the kernel process slot. (Some systems allow you to avoid this if you ignore the SIGCHLD signal). Don't leave zombie processes lying around, it's not cute. 🧟

exp_popen is yet another interface, styled after popen(). It takes a Bourne shell command line, and returns a stream that corresponds to the process's stdin, stdout and stderr. The actual implementation of exp_popen below demonstrates exp_spawnl. Look at her, she's doing so much:

FILE *
exp_popen(program)
char *program;
{
     FILE *fp;
     int ec;

     if (0 > (ec = exp_spawnl("sh","sh","-c",program,(char *)0)))
          return(0);
     if (NULL == (fp = fdopen(ec,"r+")) return(0);
     setbuf(fp,(char *)0);
     return(fp);
}

After a process is started, the variable exp_pid is set to the process-id of the new process. The variable exp_pty_slave_name is set to the name of the slave side of the pty.

The spawn functions uses a pty to communicate with the process. By default, the pty is initialized the same way as the user's tty (if possible, i.e., if the environment has a controlling terminal.) This initialization can be skipped by setting exp_ttycopy to 0. Skipping, as we established, is valid. 🦋

The pty is further initialized to some system wide defaults if exp_ttyinit is non-zero. The default is generally comparable to "stty sane". (We love a sane tty, that's the goal fr fr.)

The tty setting can be further modified by setting the variable exp_stty_init. This variable is interpreted in the style of stty arguments. For example, exp_stty_init = "sane"; repeats the default initialization. Full circle, we're home. 🏡

On some systems, it is possible to redirect console output to ptys. If this is supported, you can force the next spawn to obtain the console output by setting the variable exp_console to 1. Very "main character takes over the console". 👑

Between the time a process is started and the new program is given control, the spawn functions can clean up the environment by closing file descriptors. By default, the only file descriptors closed are ones internal to Expect and any marked "close-on-exec". Tidy. ✨

If needed, you can close additional file descriptors by creating an appropriate function and assigning it to exp_close_in_child. The function will be called after the fork and before the exec. (This also modifies the behavior of the spawn command in Expect.) So yes, side effects, keep that in mind bestie.

If you are also using Tcl, it may be convenient to use the function exp_close_tcl_files which closes all files between the default standard file descriptors and the highest descriptor known to Tcl. (Expect does this.) She's already handling it, you're welcome. 💅

The function exp_child_exec_prelude is the last function called prior to the actual exec in the child. You can redefine this for effects such as manipulating the uid or the signals. Last one out, turn off the lights. 🕯️

---

👑 IF YOU WANT TO ALLOCATE YOUR OWN PTY 👑

extern int exp_autoallocpty;
extern int exp_pty[2];

The spawn functions use a pty to communicate with the process. By default, a pty is automatically allocated each time a process is spawned. If you want to allocate ptys yourself, before calling one of the spawn functions, set exp_autoallocpty to 0, exp_pty[0] to the master pty file descriptor and exp_pty[1] to the slave pty file descriptor. You're doing it yourself, DIY queen 🎀

The expect library will not do any pty initializations (e.g., exp_stty_init will not be used). The slave pty file descriptor will be automatically closed when the process is spawned. After the process is started, all further communication takes place with the master pty file descriptor.

exp_spawnl and exp_spawnv duplicate the shell's actions in searching for an executable file in a list of directories. The directory list is obtained from the environment. It's giving PATH, but make it ✨dramatic✨.

---

✨ EXPECT PROCESSING ✨

okay bestie, so like — you could raw-dog this with read() to pull info from a process you spawned with exp_spawnl or exp_spawnv... but why would you when the library hands you these absolute icons instead? 💅 Here's the whole squad:

int
exp_expectl(fd,type1,pattern1,[re1,],value1,type2,...,exp_end);
int fd;
enum exp_type type;
char *pattern1, *pattern2, ...;
regexp *re1, *re2, ...;
int value1, value2, ...;

int
exp_fexpectl(fp,type1,pattern1,[re1,]value1,type2,...,exp_end);
FILE *fp;
enum exp_type type;
char *pattern1, *pattern2, ...;
regexp *re1, *re2, ...;
int value1, value2, ...;

enum exp_type {
exp_end,
exp_glob,
exp_exact,
exp_regexp,
exp_compiled,
exp_null,
};

struct exp_case {
char *pattern;
regexp *re;
enum exp_type type;
int value;
};

int
exp_expectv(fd,cases);
int fd;
struct exp_case *cases;

int
exp_fexpectv(fp,cases);
FILE *fp;
struct exp_case *cases;

extern int exp_timeout;
extern char *exp_match;
extern char *exp_match_end;
extern char *exp_buffer;
extern char *exp_buffer_end;
extern int exp_match_max;
extern int exp_full_buffer;
extern int exp_remove_nulls;

So what do they even do, you ask? 🦋 They wait. Like, they literally sit there until one of three things goes down: the output from a process matches one of your patterns, a time period you specified passes, or an EOF shows up. Very patient. Very demure. 🎀

First argument to each function is either a file descriptor or a stream — pick your fighter. Then come successive sets of arguments, and each set is like "here's a pattern, and here's the integer value to hand back when it matches." 💖

The type argument is one of four values (well, six total, but four real pattern kinds — stay with me). Let's break it down:

• exp_end — no more patterns, we're done here, closing time 💀
• exp_glob — the pattern is a glob-style string pattern
• exp_exact — the pattern is an exact string, no games
• exp_regexp — the pattern is a regexp-style string pattern
• exp_compiled — the pattern is a regexp-style string pattern and you're also handing over its compiled form
• exp_null — the pattern is a null (strictly for debugging vibes, and a string pattern still has to follow it, okay?)

Now here's the tea about compilation ☕ — if you don't provide the compiled form with exp_expectl and exp_fexpectl, then any pattern compilation done internally gets yeeted the second the function returns. Gone. Bye. 🫶

But exp_expectv and exp_fexpectv?? They compile patterns automatically and they keep them around. So you have to be the one to clean up — call free on each pattern. And you only need to discard them the last time the cases are used, so don't be out here freeing mid-conversation. 🙈

Regexp subpatterns that matched get stored in the compiled regexp. Assuming re holds a compiled regexp, the matched string lives in re->startp[0]. The match substrings (per the parentheses in your original pattern) live in re->startp[1], re->startp[2], and so on, all the way up to re->startp[9]. And the corresponding string ends? Those are re->endp[x] where x is that same index as the string start. Symmetry. We love to see it. ✨

The type exp_null matches if a null appears in the input. Important!! You have to set exp_remove_nulls to 0 to stop nulls from being automatically stripped. By default, exp_remove_nulls is set to 1 and nulls get stripped automatically. That's just the default, bestie.

exp_expectv and exp_fexpectv are your besties when you don't know how many patterns you'll have in advance. In that case the sets come in an array, and the end of the array is marked by a struct exp_case with type exp_end. For the rest of this discussion, these functions all get referred to generically as expect. Group chat name, basically. 💬

If a pattern matches, the corresponding integer value gets returned. The values don't have to be unique, but they should be positive so they don't get mistaken for EXP_EOF, EXP_TIMEOUT, or EXP_FULLBUFFER. Learn from my mistakes. 😭

Upon EOF or timeout, the value EXP_EOF or EXP_TIMEOUT gets returned. The default timeout period is 10 seconds, but you can change that by setting the variable exp_timeout. Setting it to -1 disables timeouts entirely. Setting it to 0 makes the expect function return immediately (i.e., poll) after one read().

⚠️ But bestie, listen — if you go with 0, it must be preceded by a function like select, poll, or an event manager callback to guarantee there's actually data to be read. Otherwise you're just out here reading vibes.

If the variable exp_full_buffer is 1, then EXP_FULLBUFFER gets returned if exp_buffer fills up with no pattern having matched. Packed house, nobody matched. 💀

When the expect function returns:

• exp_buffer points to the buffer of characters that was being considered for matching
• exp_buffer_end points to one past the last character in exp_buffer
• if a match occurred, exp_match points into exp_buffer where the match began
• exp_match_end points to one character past where the match ended

Every time new input arrives, it gets compared to each pattern in the order they're listed. So here's a little trick — you can test for the absence of a match by making the last pattern something guaranteed to appear, like a prompt. In situations where there's no prompt, you've gotta check for EXP_TIMEOUT, just like you would if you were interacting manually like a peasant. More philosophy and strategies on specifying expect patterns live in the docs on the expect program itself. See SEE ALSO below. 👑

Patterns are the usual C-shell-style regular expressions. For example, this little fragment right here looks for a successful login, like from a telnet dialogue:

switch (exp_expectl(
     exp_glob,"connected",CONN,
     exp_glob,"busy",BUSY,
     exp_glob,"failed",ABORT,
     exp_glob,"invalid password",ABORT,
     exp_end)) {
case CONN:     /* logged in successfully */
     break;
case BUSY:     /* couldn't log in at the moment */
     break;
case EXP_TIMEOUT:
case ABORT:    /* can't log in at any moment! */
     break;
default: /* problem with expect */
}

Asterisks (like in the example above) are a handy shorthand for omitting line-termination characters and other detail. 🌸

Patterns must match the entire output of the current process (since the previous read on the descriptor or stream). And heads up — more than 2000 bytes of output can force earlier bytes to be "forgotten". You can change that by setting the variable exp_match_max. Just know that excessively large values can slow down the pattern matcher, so don't go feral with it. 🐌

---

🌙 RUNNING IN THE BACKGROUND 🌙

extern int exp_disconnected;
int exp_disconnect();

It is genuinely possible to move a process into the background after it's already started running. Typical use case? Reading passwords and then dipping into the background to sleep before using those passwords to do the real work. Main character exit. 🚪✨

To move a process into the background: fork, call exp_disconnect() in the child process, and exit() in the parent process. This disassociates your process from the controlling terminal. If you wanna move a process into the background some other way, you must set the variable exp_disconnected to 1. That's what lets processes spawned after that point start correctly. 💅

---

🌈 MULTIPLEXING 🌈

By default, the expect functions block inside of a read on a single file descriptor. If you want to wait on patterns from multiple file descriptors, use select, poll, or an event manager — they'll tell you which file descriptor is ready to read. 👀

When a file descriptor is ready to read, you can use the expect functions to do one and only read by setting timeout to 0.

---

🎀 SLAVE CONTROL 🎀

void
exp_slave_control(fd,enable)
int fd;
int enable;

Pty trapping is normally done automatically by the expect functions. But if you want to issue an ioctl — for example — directly on the slave device, you should temporarily disable trapping. Sometimes you just gotta do it yourself. 🙈

Pty trapping can be controlled with exp_slave_control. The first argument is the file descriptor corresponding to the spawned process. The second argument is a 0 if trapping is to be disabled, and 1 if it's to be enabled.

---

💀 ERRORS 💀

All functions indicate errors by returning -1 and setting errno.

Errors that happen after the spawn functions fork (e.g., attempting to spawn a non-existent program) get written to the process's stderr, and will be read by the first expect. So like... don't be surprised when they show up in your output. 😭

---

⚡ SIGNALS ⚡

extern int exp_reading;
extern jmp_buf exp_readenv;

expect uses alarm() to timeout, so if you generate alarms during expect, it'll timeout prematurely. Not the vibe you want. ⏰

Internally, expect calls read() which can be interrupted by signals. If you define signal handlers, you can choose to restart or abort expect's internal read. The variable exp_reading is true if (and only if) expect's read has been interrupted.

• longjmp(exp_readenv,EXP_ABORT) will abort the read
• longjmp(exp_readenv,EXP_RESTART) will restart the read

Pick your lane. 🫶

---

📓 LOGGING 📓

extern int exp_loguser;
extern int exp_logfile_all
extern FILE *exp_logfile;

If exp_loguser is nonzero, expect sends any output from the spawned process to stdout. Since interactive programs typically echo their input, this usually suffices to show both sides of the conversation. Both sides! Représentation! 💅

If exp_logfile is also nonzero, this same output gets written to the stream defined by exp_logfile. And if exp_logfile_all is non-zero, exp_logfile gets written regardless of the value of exp_loguser. No gatekeeping. 🎀

---

🔍 DEBUGGING 🔍

okay real talk bestie — while I consider the library to be easy to use, I think the standalone expect program is much, much easier to use than working with the C compiler and its usual edit, compile, debug cycle. 😭

Unlike typical C programs, most of the debugging isn't getting the C compiler to accept your programs — rather, it's getting the dialogue correct. Also, translating scripts from expect to C is usually not necessary. For example, the speed of interactive dialogues is virtually never an issue. So please try the standalone expect program first. I suspect it's a more appropriate solution for most people than the library. Fr fr. 🫶

Nonetheless, if you feel compelled to debug in C, here are some tools to help you out:

extern int exp_is_debugging;
extern FILE *exp_debugfile;

While expect dialogues seem very intuitive, trying to codify them in a program can reveal many surprises in a program's interface. So a variety of debugging aids are available. They're controlled by the above variables, all 0 by default.

Debugging information internal to expect gets sent to stderr when exp_is_debugging is non-zero. This debugging info includes every character received, and every attempt made to match the current input against the patterns. In addition, non-printable characters get translated to a printable form — for example, a control-C appears as a caret followed by a C. If exp_logfile is non-zero, this information is also written to that stream. 👑

If exp_debugfile is non-zero, all normal and debugging information gets written to that stream, regardless of the value of exp_is_debugging. It said what it said. 💅

💅 CAVEATS 💅

okay bestie, gather round, because this part is important. the stream versions of the expect functions are MUCH slower than the file descriptor versions — and there's a real reason for it: there is literally no way to portably read an unknown number of bytes without the potential of timing out. so what happens instead? characters are read one at a time. one. at. a. time. 😭

so you are strongly encouraged to use the file descriptor versions of expect (although, tbh, automated versions of interactive programs don't usually demand high speed anyway, so it's not the end of the world 🫶).

---

but wait, you can actually get the best of both worlds ✨ — write with the usual stream functions and read with the file descriptor versions of expect, as long as you don't attempt to intermix other stream input functions (e.g., fgetc). to do this, pass fileno(stream) as the file descriptor each time. fortunately, there is little reason to use anything but the expect functions when reading from interactive programs. iconic behaviour, no notes. 💅

also, PSA: there is no matching exp_pclose to exp_popen (unlike popen and pclose, which DO have a matching pair). it only takes two functions to close down a connection (fclose() followed by waiting on the pid), but it is not uncommon to separate these two actions by large time intervals, so the function seems of little value. so like, fair. 🤷

---

⚠️ CRAY ALERT ⚠️ if you are running on a Cray running Unicos (all I know for sure from experience, fr fr), you must run your compiled program as root or setuid. the problem is that the Cray only allows root processes to open ptys. 😳

you should observe as much precautions as possible: if you don't need permissions, setuid(0) only immediately before calling one of the spawn functions and immediately set it back afterwards. do not be casual about this one bestie, i'm begging 🙏

---

normally, spawn takes little time to execute. but if you notice spawn taking a significant amount of time, it is probably encountering ptys that are wedged. 💀 a number of tests are run on ptys to avoid entanglements with errant processes. (these take 10 seconds per wedged pty.) running expect with the -d option will show if expect is encountering many ptys in odd states. and if you cannot kill the processes to which these ptys are attached, your only recourse may be to reboot. yeah. that escalated. 😭

---

🐛 BUGS 🐛

okay so this one's a known situation: the exp_fexpect functions don't work at all under HP-UX — it appears to be a bug in getc. 🫠

follow the advice (above) about using the exp_expect functions (which doesn't need to call getc). and if you fix the problem (before I do — please check the latest release) let me know. the author is literally asking. 🥺

---

🌈 SEE ALSO 🌈

an alternative to this library is the expect program itself. expect interprets scripts written in a high-level language which direct the dialogue. in addition, the user can take control and interact directly when desired. so if it is not absolutely necessary to write your own C program, it is much easier to use expect to perform the entire interaction. 👑

it is described further in the following references:

• "expect: Curing Those Uncontrollable Fits of Interactivity" by Don Libes, Proceedings of the Summer 1990 USENIX Conference, Anaheim, California, June 11-15, 1990.
• "Using expect to Automate System Administration Tasks" by Don Libes, Proceedings of the 1990 USENIX Large Installation Systems Administration Conference, Colorado Springs, Colorado, October 17-19, 1990.

and the whole friend group of related man pages:

• expect(1), alarm(3), read(2), write(2), fdopen(3), execve(2), execvp(3), longjmp(3), pty(4).

there are several examples C programs in the test directory of expect's source distribution which use the expect library. go peek, they're cute 🎀

---

👑 AUTHOR 👑

Don Libes, libes@nist.gov, National Institute of Standards and Technology

---

💖 ACKNOWLEDGEMENTS 💖

thanks to John Ousterhout (UCBerkeley) for supplying the pattern matcher. 🫶

design and implementation of the expect library was paid for by the U.S. government and is therefore in the public domain. however the author and NIST would like credit if this program and documentation or portions of them are used. so like, give credit where it's due bestie, it's free ✨

---

and that's the whole page, we made it!! 🎉 this was LIBEXPECT(3), dated 12 December 1991 — yes, the vintage is part of the charm, she's an icon and she's been serving since 1991 💅 periodt. now go forth and expect responsibly. 🦋