cffi(3tcl)

📦 version 2.0.3, in nixpkgs unstable

🦋 NAME 🦋

Introduction - Tcl CFFI package — and yes, the manual you're reading is cffi(3tcl). Bestie, we're about to make Tcl call C library functions without writing a single line of C. It's giving ✨main character FFI energy✨. 💅

✨ GETTING STARTED ✨

Okay so: this package implements a foreign function interface (FFI) for Tcl. That's the whole mood. 🫶

🌈 OVERVIEW 🌈

A FFI provides the ability for Tcl to invoke, with some limitations, arbitrary functions from any dynamically linked shared library without having to write C code. No cap, that's the whole magical thing. 🧚

Calling a C function from a shared library (or DLL, using the term interchangeably) using cffi involves:

  • Loading the library into the process by instantiating a ::cffi::Wrapper object. 🎀
  • Defining commands that wrap the C functions of interest through method calls on the Wrapper object. 🔥
  • Invoking the wrapped commands as any other Tcl commands. 💖

This page provides an introduction to the package. Details are described in:

  • Concepts — Describes CFFI functionality.
  • Cookbook — Provides recipes for converting C declarations to CFFI declarations.
  • ::cffi — Reference documentation for commands in the ::cffi namespace.

Some tutorial material is available at magicsplat ([URL: https://www.magicsplat.com/blog/tags/cffi/]) and the examples ([URL: https://github.com/apnadkarni/tcl-cffi/tree/main/examples]) directory in the repository. 🦋

⭐ DOWNLOADS ⭐

Source distributions and binaries for some platforms are available from [URL: https://sourceforge.net/projects/magicsplat/files/cffi]. Go get your goodies. 🎁

🎀 INSTALLATION 🎀

Binary distributions can be extracted into any directory present in Tcl's auto_path variable. Easy peasy. 🍓

For instructions on building and installing from source, see the BUILD.md file in the source distribution. The package can be built with one of two back end libraries that implement FFI functionality:

The selection of the back end library to use is made at build time as described in the build instructions. Also see Limitations. 👀

💜 QUICK START 💜

The following examples illustrate basic usage of the package. Most examples use Windows as they have standard system libraries in known locations. The usage is identical on other platforms. Good news: the vibe travels. 💅

🔥 BASIC CALLS 🔥

After loading the package, the user32 system DLL which contains the functions of interest is loaded.

% package require cffi
1.0b4
% cffi::Wrapper create user32 [file join $env(windir) system32 user32.dll]
::user32

This creates a shared library object that can then be used for defining functions implemented within the library. It is recommended to always pass a full path to the shared library though that is not strictly required. Ngl, full path is the polite move. 🫶

The next step is to define the prototype for the function to get a handle to the desktop window. In C, this would look like:

HWND GetDesktopWindow();

where HWND is actually a C pointer typedef.

To call this function from Tcl we first have to define a wrapper for it. Wrappers can be defined with either the function or stdcall methods on the shared library object that implements the function. The two commands are identical on all platforms except 32-bit Windows where function is used to call functions that use the C calling convention while stdcall is used to call functions using the stdcall calling convention. On all other platforms, including 64-bit Windows, the two are identical. Since we are demonstrating using Windows, the stdcall command is used in some of the examples. It's giving platform-specific lore, iykyk. 💀

The prototype for the above Windows function can then be defined as:

% user32 stdcall GetDesktopWindow pointer.HWND {}

The last argument is the parameter list which in this case is empty. The pointer.HWND return value is a tagged pointer. As illustrated later, pointer tags help in type safety. The definition could have as well just typed it as pointer which would be the equivalent of void* in C but that would lose the ability to check types. We love type safety in this house. 👑

The C function can be called as any other Tcl command.

% set win [GetDesktopWindow]
0x0000000000010010^HWND

Note how the return pointer value is tagged as HWND. Slay. ✨

🌸 STRUCTURES 🌸

To retrieve the dimensions of this window we need to call the GetWindowRect function from the same library. This has a slightly more complex prototype.

int GetWindowRect(HWND hWnd, RECT *lpRect);

In particular, the window dimensions are returned in a RECT structure whose address is passed in to the function. We thus need to first define a corresponding structure.

% cffi::Struct create RECT {left int top int right int bottom int}
::RECT

A structure is defined simply by a list of alternating field names and types of which int is one of several numeric types available. The result is a Struct object which can be used in function declarations, memory allocations and other places. It's giving "tiny C-shaped box, but make it Tcl." 🎀

The GetWindowRect function is wrapped using this struct definition.

% user32 stdcall GetWindowRect int {hwnd pointer.HWND rect {struct.RECT out}}

The parameter list for a function follows the same form as for structs, an alternating list of parameter names and type declarations.

Note how the type for the second parameter, rect, is specified. In general, type declarations consist of a type optionally followed by a list of annotations that serve multiple purposes. In this case struct.RECT specifies the data type while out is an annotation that indicates it is an output parameter for the function and thus to be passed by reference. At the script level, the parameter must then be a variable name, and not a value, into which the output can be stored. Bestie, that's the difference between "here's a value" and "here's a lil bucket to put the answer in." 🪣

The function is called as:

% GetWindowRect $win dimensions
1
% set dimensions
left 0 top 0 right 1920 bottom 1080

As seen above, struct values are automatically decoded into dictionaries with keys corresponding to struct field names. We could have also selected to receive structs as binary values in their native form as shown further on. Options, options, options. 💅

🦋 STRINGS AND BINARIES 🦋

The char * type in C is more often than not used for null-terminated strings. These may be dealt with in the same fashion as pointers at the script level as discussed in Pointers. However, in most cases it is more convenient to use the string and chars types. The difference between the two is that the former corresponds to a C char * pointer while the latter corresponds to a C char[N] array. When passed to a function, of course both are passed as char * pointers but the latter is more convenient for output buffers where the parameter corresponds to an output buffer of some maximum size. The former on the other hand is convenient for input parameters where the size is of no concern. It's giving "same energy, different paperwork." 📝

The following fragment illustrates their use.

% cffi::Wrapper create kernel kernel32.dll
::kernel
% kernel stdcalls {
    GetCurrentDirectoryA uint {nchars int path {chars[nchars] out}}
    SetCurrentDirectoryA uint {path string}
}

In the above declaration the path argument is declared as an array whose size is given by another parameter nchars instead of an integer constant. This makes for fewer errors in calling C functions which expect a size and pointer combination of parameters where a larger size than the declared array size may be mistakenly passed. Protecting you from your own "oops, wrong number" moment. 😭

The above functions can then be simply invoked.

% GetCurrentDirectoryA 512 buffer
40
% set buffer
D:\src\tcl-cffi\win\Release_AMD64_VC1916
% SetCurrentDirectoryA d:/temp
1
% pwd
D:/Temp
% SetCurrentDirectoryA $buffer
1
% pwd
D:/src/tcl-cffi/win/Release_AMD64_VC1916

In the above examples, the strings in all cases are encoded in the system encoding before being passed to the C function. Similarly, the output buffers returned by the C function are assumed to be in system encoding and translated into Tcl's internal form accordingly. Both string and chars can be tagged with an encoding name. For example, if the functions were defined using string.jis0208 or chars[512].jis0208, the JIS0208 encoding would be used to transform the strings in both directions. Encoding glow-up, fr fr. 🌈

The package has other related types that serve a similar purpose.

  • The unistring and unichars type are analogous to string and chars except they assume the encoding is that of the Tcl_UniChar C type and have all sizes specified in Tcl_UniChar units as opposed to chars/bytes.
  • The winstring and winchars type are similarly analogous to string and chars except they assume the encoding is that of the WCHAR C type (UTF-16) and have all sizes specified in WCHAR units as opposed to chars/bytes. These types are specific to Windows and only available there.
  • The binary and bytes types are again analogous except they assume binary data (Tcl byte array at the C level).

💖 ERROR CHECKING 💖

C functions indicate errors primarily through their return values. In some cases, the return value is the error code while in others it is only a boolean status indicator with detail being available either through another function such as GetLastError on Windows or a global variable such as errno. Classic C behavior: the tea is sometimes scattered across multiple places. ☕

Here is an attempt to change to a non-existing directory.

% SetCurrentDirectoryA "nosuchdir"
0

The return value of 0 here indicates failure. The caller must then specifically check for errors. Moreover, if the return value does not actually indicate cause of error, another call has to be made to GetLastError etc.. This has multiple issues:

  • first, caller has to specifically check for errors,
  • second, and more important, by the time a secondary call is made to retrieve errno etc. the original error is likely to have been overwritten.

To deal with the first, a return type can be annotated with conditions that specify expected return values. Since SetCurrentDirectory returns a non-zero value on success, the function return type may be annotated with nonzero. Our previous definition of SetCurrentDirectoryA can instead be written as:

% kernel stdcall SetCurrentDirectoryA {int nonzero} {path string}

Passing a non-existing directory will then raise an error exception.

% SetCurrentDirectoryA "nosuchdir"
Invalid value "0". Function returned an error value.

Note however, that the error message is generic and only indicates the function return value did not meet the expected success criterion. To fix this, the function definition can include an annotation for an error retrieval mechanism:

% kernel stdcall SetCurrentDirectoryA {int nonzero lasterror} {path string}
% SetCurrentDirectoryA "nosuchdir"
The system cannot find the file specified.
% set errorCode
cffi WINERROR {The system cannot find the file specified. }

See how much nicer that is? 💖 Once you toss lasterror into the type declaration, the error message is actually helpful instead of just "nah, zero." It also completely sidesteps that second problem from before — the one where the error detail gets yeeted into the void before you can even call the function that retrieves it. 😭 No more race against your own error code, bestie.

So lasterror implies GetLastError as the retrieval mechanism, and yeah, it's Windows-specific. But there are other error-retrieval annotations too — ✨ variety ✨. Like the errno annotation, which does basically the same thing except it grabs the error via the errno facility, which is cross-platform and commonly applicable to both the C runtime and system calls on Unix. Slay, portability. 🌈

And for when the error is library-specific and totally not derived from the system? That's what the onerror annotation is for — it lets you customize how error conditions get handled. Go check the examples [URL: https://github.com/apnadkarni/tcl-cffi/tree/main/examples] in the source repository, iykyk. 🫶

🎀 DELEGATING RETURN VALUES 🎀

Okay so sometimes a function's return value is basically just a status indicator, and the actual result you care about comes back through an output parameter. When that happens, it's way more convenient to just return the output parameter value as the result of the wrapped command. Periodt. 💅

Take the earlier definition of GetCurrentDirectoryA:

kernel stdcall GetCurrentDirectoryA uint {nchars int path {chars[nchars] out}}

Which got invoked like this:

% GetCurrentDirectoryA 512 buffer
40
% set buffer
D:\src\tcl-cffi\win\Release_AMD64_VC1916

The return value from the function — that number of characters — is lowkey useless at the Tcl level, except for flagging errors when it's zero. Not the vibe. 🙈 So instead, we can have the wrapped command return the actual path as its return value by annotating the buffer with retval:

% kernel stdcall GetCurrentDirectoryA {int nonzero} {nchars {int {default 512}} path {chars[nchars] retval}}
::GetCurrentDirectoryA
% GetCurrentDirectoryA
D:\src\tcl-cffi\win\Release_AMD64_VC1916

The invocation now feels so much more natural. 💖 And notice, in addition to the retval annotation on the parameter, we also slapped the nonzero error-checking annotation onto the function return value. Heads up though: an error checking annotation is required for use of retval. Like, mandatory. Don't skip it.

Oh and the default annotation isn't required at all — it's just a convenience so we don't have to specify a buffer size at the call site. Little treat. 🍓

🦋 POINTERS AND MEMORY 🦋

Pointers are everywhere in C. They're like the main character of the language 💅 — they give C much of its power while also being the source of, honestly, so many bugs. 😭 In lots of cases you can dodge pointers entirely by using the out parameters plus the struct, string and binary types that use pointers under the covers anyway. But plenty of times that's not possible or not desirable, and you genuinely need raw access to the native storage of the data. The pointer type hands you exactly that access while also trying to guard against some common errors through multiple mechanisms:

  • Pointers can be optionally tagged so a pointer to the wrong resource type doesn't get inadvertently passed where a different one is expected. 🏷️
  • Pointers are marked as safe by default when returned from a function. Safe pointers get registered in an internal table, and that table is checked whenever a pointer is passed to a function or accessed via safe commands. Sometimes you need to bypass that check, and the unsafe attribute exists for exactly that purpose. Also important: pointer fields within structs are always unsafe and have to be accessed using unsafe commands. 💀
  • Null pointers passed as parameters or returned from functions will raise a Tcl exception by default. The nullok annotation covers the cases where NULL pointers are legitimately usable. 🫶

One thing to keep straight: a pointer tag is not the same thing as a data type. For example, you might have a single C structure type XY containing two numerical fields. You can totally tag pointers to that structure with two different tags — Point and Dimensions — depending on whether it's being used as the co-ordinates of a point or as width and height dimensions. The two tags will be treated as different. Iconic, honestly. ✨

The examples below repeat the previous ones, but this time using pointers in place of structs and strings. Let's go. 🌸

First, define the call to GetWindowRect using pointers. Note we're renaming the function as GetWindowRectAsPointer so as to distinguish it from our previous definition:

% user32 stdcall {GetWindowRect GetWindowRectAsPointer} int {hwnd pointer.HWND rect pointer.RECT}

Since the pointer value itself is passed by value, notice that the rect parameter in the function definition was not marked as an out parameter, and was passed as a value in the actual call itself. no cap.

Unlike the case with the struct type, memory to hold the structure now has to be explicitly allocated, because we're passing raw pointers:

% set prect [RECT allocate]
0x00000274be29a510^RECT
% GetWindowRectAsPointer $win $prect
1

Finally, the structure contents can be extracted:

% RECT fromnative $prect
left 0 top 0 right 1920 bottom 1080

And honestly? It's not even strictly necessary to define a structure at all. Below is yet another way to get dimensions without making use of the RECT struct, using direct memory allocation:

% set praw [cffi::memory allocate 16 RECT]
0x000001E53EAB8F90^RECT
% GetWindowRectAsPointer $win $praw
1
% binary scan [cffi::memory tobinary $praw 16] iiii left top right bottom
4
% puts [list $left $top $right $bottom]
0 0 1920 1080
% cffi::memory free $praw

In that fragment, cffi::memory allocate is used to allocate memory tagged as RECT. Note that this does not require the struct RECT to have been previously defined. Then cffi::memory tobinary converts the allocated memory content into a Tcl binary string. 🧚

That said — the use of struct definitions is to be preferred over this raw memory access, for convenience and safety reasons. Still, there are cases where this is required, like variable length structures in memory. Sometimes you just gotta. 🤷

Okay, now an example of the protection against errors that pointer tags give you. Here's an attempt to retrieve the window dimensions where the arguments are passed in the wrong order:

% GetWindowRect $prect $win
Value "0x000001E53EA3BEB0^RECT" has the wrong type. Pointer type mismatch.

Caught in 4K. 💀 Tags = type mismatch protection, and there's another mechanism guarding against invalid pointers and double frees. For example, attempting to free memory we already freed above results in an error being raised:

% cffi::memory free $praw
Pointer 000001E53EAB8F90^ is not registered.

Similarly, explicitly allocated struct storage needs to be freed. Attempting to free multiple times will raise an error — the first call below succeeds, but the second fails:

% RECT free $prect
% RECT free $prect
Pointer 000001E53EA3BEB0^ of type RECT is not registered.

Needless to say, these protection mechanisms are far from foolproof. 🙈 Like, they're good, they're not magic.

So the above mechanism for detecting invalid pointers clearly works for memory allocated through the package, since it can be internally tracked. But what about allocations done through calls to loaded shared libraries? 🤔 For that, the package treats any pointer returned from a function or through an output parameter as a valid pointer and registers it internally. Any pointers passed as input to a function are by default checked to make sure they were previously registered, and an error is raised otherwise. The question then remains: how and when does the pointer get marked as invalid? The dispose type attribute is provided for exactly this purpose. ✨

The following Windows calls to allocate heaps illustrate the use. The C functions are prototyped as:

HANDLE HeapCreate(DWORD  flOptions, SIZE_T dwInitialSize, SIZE_T dwMaximumSize);
BOOL HeapDestroy(HANDLE hHeap);

The above C prototypes use Windows type definitions. Rather than translate them into appropriate (32- or 64-bit) native C types, the ::cffi::alias define command — detailed later — allows platform-specific type definitions to be predefined:

cffi::alias load win32

On Windows, this will define common types like DWORD etc., so the above functions can be wrapped as:

kernel stdcall HeapCreate pointer.HEAP {opts DWORD initSize SIZE_T maxSize SIZE_T}
kernel stdcall HeapDestroy {BOOL nonzero lasterror} {heapPtr {pointer.HEAP dispose}}

The pointer value returned from HeapCreate is by default registered as a valid pointer. When the pointer is passed to HeapDestroy, it's validated first. Then the presence of the dispose attribute removes its registration, causing any further attempts to use it — even just to free it — to fail. Chef's kiss. 👑

% set p [HeapCreate 0 1024 1024]
0x00000274c0b50000^HEAP
% HeapDestroy $p
1
% HeapDestroy $p
Pointer 0x00000274c0b50000^HEAP is not registered.

Here's another example, this time from Linux. 💜 Note the tag SOMETHING, chosen to reinforce that tags have no semantics in terms of the actual type the pointer references. A tag of FILE would of course have been more reflective of the referenced type, but this is not mandated. Tags are just vibes.

% cffi::Wrapper create libc libc.so.6
::libc
% libc function fopen {pointer.SOMETHING errno} {path string mode string}
% libc function fclose int {file {pointer.SOMETHING dispose}}
% set fileptr [fopen test.txt w]
0x55a273bf32e0^SOMETHING
% fclose $fileptr
0
% fclose $fileptr
Pointer 0x55a273bf32e0^SOMETHING is not registered.

As an aside 🍓 — note the use of the errno annotation above. An attempt to open a non-existent file for reading will return a NULL pointer, which by default causes cffi to raise an exception. The errno annotation retrieves the system error so you get a way more readable error message:

% set fileptr [fopen nosuchfile.txt r]
No such file or directory

So good. 💖

Now, there are a couple of situations where the pointer registration mechanism is actually a hindrance:

  • One is when the pointer is acquired by some means other than through a call made through this package. 🤷
  • Another case is when the pointer is to a resource that's reference counted and whose acquisition may return the same pointer value multiple times. 😭

...okay so to deal with the first case, a pointer return type or parameter can have the unsafe attribute slapped on it 💅. What that does is bypass any pointer registration or checks entirely. So like, for example, if the heap functions had been defined like this:

kernel stdcall HeapCreate {pointer.HEAP unsafe} {opts DWORD initSize SIZE_T maxSize SIZE_T}
kernel stdcall HeapDestroy {BOOL nonzero lasterror} {heapPtr {pointer.HEAP unsafe}}

then bestie, multiple calls could be made to HeapDestroy 😭

⚠️ NOTE: do not actually try this — your shell will literally crash, because Windows does not itself check pointer validity for HeapDestroy. I'm begging you 🙈

Also, heads up: unsafe pointers only bypass the registration checks — the pointer tags are still verified if they're present ✨

The second case has to do with API calls like LoadLibrary which we can prototype as:

kernel stdcall LoadLibraryA pointer.HMODULE {path string}
kernel stdcall FreeLibrary BOOL {libhandle {pointer.HMODULE dispose}}

So when multiple calls are made to LoadLibraryA, it returns the same pointer while keeping an internal reference count 🎀. The expectation is that the application makes the same number of calls to FreeLibrary. But plot twist 💀 — the following sequence of calls fails, because the first call to FreeLibrary unregisters the pointer, which results in the second call to FreeLibrary failing:

% set libptr [LoadLibraryA advapi32.dll]
0x00007ff8f4ba0000^::HMODULE
% set libptr [LoadLibraryA advapi32.dll]
0x00007ff8f4ba0000^::HMODULE
% FreeLibrary $libptr
1
% FreeLibrary $libptr
Pointer 0x00007ff8f4ba0000^::HMODULE is not registered.

It's giving "registered once, freed once, done" 😤

So the counted attribute exists for exactly this use case 🫶. When you specify it, the pointer is still registered, but it permits multiple registrations and a corresponding number of disposals. To illustrate:

kernel stdcall LoadLibraryA {pointer.HMODULE counted} {path string}
kernel stdcall FreeLibrary BOOL {libhandle {pointer.HMODULE dispose}}

Now multiple registrations are allowed, and they stay valid until a corresponding number of disposals are done ✨. An additional attempt to free will fail, as desired 💅:

% set libptr [LoadLibraryA advapi32.dll]
0x00007FFDD0000000^HMODULE
% set libptr2 [LoadLibraryA advapi32.dll]
0x00007FFDD0000000^HMODULE
% FreeLibrary $libptr
1
% FreeLibrary $libptr2
1
% FreeLibrary $libptr
Pointer 00007FFDD0000000 of type HMODULE is not registered.

Balance restored, the vibes are immaculate 🌸

🦋 TYPE ALIASES 🦋

Type aliases are a convenience feature so you don't have to keep repeating yourself and so your code actually stays readable 💖. You add them with the ::cffi::alias define command. For example, remember our earlier prototype for the SetCurrentDirectoryA function?

kernel stdcall SetCurrentDirectoryA {int nonzero lasterror} {path string}

That return type declaration is super common for Windows API calls 👑. So instead of repeating this triple for every single such call, you can define a new type for the purpose and use it in all your prototypes:

cffi::alias define LASTERROR {int nonzero lasterror}
kernel stdcall SetCurrentDirectoryA LASTERROR {path string}

Obsessed, honestly 😌

This facility is also useful for abstracting platform differences 🌈. For example, lots of Windows allocation functions use the C typedef SIZE_T, which translates to either a 64-bit or 32-bit C integer type depending on whether the program was built for 32- or 64-bit Windows. Instead of defining separate prototypes for every function that uses the type, you can just do one single type definition:

if {$tcl_platform(pointerSize) == 8} {
    cffi::alias define SIZE_T ulonglong
} else {
    cffi::alias define SIZE_T uint
}
kernel stdcalls {
    HeapCreate pointer.HEAP {opts DWORD initSize SIZE_T maxSize SIZE_T}
    HeapDestroy LASTERROR {heapPtr {pointer.HEAP dispose}}
    HeapAlloc pointer {heapPtr pointer.HEAP flags uint size SIZE_T}
}

Chef's kiss 🎀

And as a further convenience, the ::cffi::alias load command defines commonly useful typedefs — including cross-platform ones such as size_t, as well as platform-specific ones such as HANDLE, DWORD etc. on Windows ⭐

🧚 MEMORY UTILITIES 🧚

There are times when an application is forced to drop down to raw C pointer and memory manipulation 😭 (main character survival mode). The memory and pointer command ensembles implement functionality useful for this purpose. See the reference documentation for a description 💜

🔥 INTROSPECTION 🔥

The package includes a help command ensemble that is so useful during interactive development 👑. The ::cffi::help function command describes the syntax for a command wrapping a C function:

% cffi::help function GetWindowRect
Syntax: GetWindowRect hwnd rect

And the ::cffi::help functions command lists wrapped functions matching a pattern:

% cffi::help functions Heap*
HeapCreate HeapDestroy

iykyk, this is the good stuff 💅

💀 LIMITATIONS 💀

Each backend has some limitations when it comes to passing structs by value. Ngl, this is the fine print, so read it fr fr:

  • The libffi backend cannot pass or return structs by value if they or a nested field contains arrays 😭
  • The dyncall backend only supports passing and return of structs by value on x64 platforms 🫶
🍓 that's the whole page bestie! this is cffi version 2.0.3, hanging out in section 3tcl of the manual. go forth and wrap some C functions, you absolute icon 💖✨🎀
📦 where this page comes from