critcl_use(3tcl)

📦 version 3.1.18.1, in debian bookworm, ubuntu jammy, ubuntu noble

🎀 NAME 🎀

critcl_use — the whole "how do I actually use this thing" page, section 3tcl of the CriTcl manual. It's giving tutorial energy. Bestie, buckle up. 💖

✨ DESCRIPTION ✨

C Runtime In Tcl, aka CriTcl, is a whole system for compiling C code that's embedded inside Tcl on the fly — and then either loading the resulting objects straight into Tcl so you can use them immediately, or packaging them up for distribution. 🦋

Use CriTcl to make stuff faster by rewriting the routines that are your performance bottlenecks in C. It's the glow-up your hot loop deserves. 🔥

This document is a (hopefully) gentle introduction to Critcl, and it does that by walking through a series of small examples.

If you landed here directly through a search or whatever, and you're lowkey in need of an overview of the whole system, pls go read the Introduction To CriTcl first. No cap. 🙈

The examples here cover both how to embed C into Tcl with it, and how to build the distributable packages. So the intended audience is both writers of packages with embedded C, and people building such packages. To make life easier, the two themes each have their own section in this document, so every reader can quickly skip the part they're not interested in. Main character energy for everyone. 👑

The sources of Critcl, should you have gotten them, contain several larger examples show-casing various aspects of the system. These demonstration packages can all be found in the sub-directory "examples/" of the sources. 🌸

💅 EMBEDDING C 💅

This is the section for developers writing, or wishing to write, a package embedding C into Tcl via critcl.

I guess that we are allowed to assume that you, gentle reader, are here because you have written some Tcl code which is not fast enough (any more) and you wish to make it "go faster" by replacing parts (or all) of it with speedy C. ⚡

Another, and I believe reasonable, assumption to make would be that you have already investigated and ruled out or done things like changes to data structures and algorithms which reduce O(n*n) work to O(n*log n), O(n), or even O(1). Of course, nothing prevents you from forging ahead here even if you have not done such. Still, even in that case I would recommend that you consider investigating this line of making your code go faster as well. 🧚

Now, with these introductory words out of the way, lets jump into the meat of things.

🌈 A SIMPLE PROCEDURE 🌈

Starting simple, let us assume that the Tcl code in question is something like

proc math {x y z} {
    return [expr {(sin($x)*rand())/$y**log($z)}]
}

with the expression pretending to be something very complex and slow.

Converting this to C we get:

critcl::cproc math {double x double y double z} double {
    double up   = rand () * sin (x);
    double down = pow(y, log (z));
    return up/down;
}

Notable about this translation:

  1. All the arguments got type information added to them, here "double". Like in C the type precedes the argument name. Other than that it is pretty much a Tcl dictionary, with keys and values swapped. 💫
  2. We now also have to declare the type of the result, here "double", again.
  3. The reference manpage lists all the legal C types supported as arguments and results.

🎀 CUSTOM TYPES, INTRODUCTION 🎀

When writing bindings to external libraries, critcl::cproc is usually the most convenient way of writing the lower layers. This is however hampered by the fact that critcl on its own only supports a few standard (arguably the most import) standard types, whereas the functions we wish to bind most certainly will use much more, specific to the library's function.

The critcl commands argtype, resulttype and their adjuncts are provided to help here, by allowing a developer to extend critcl's type system with custom conversions. ✨

This and the three following sections will demonstrate this, from trivial to complex.

The most trivial use is to create types which are aliases of existing types, standard or other. As an alias it simply copies and uses the conversion code from the referenced types.

Our example is pulled from an incomplete project of mine, a binding to Jeffrey Kegler's libmarpa library managing Earley parsers. Several custom types simply reflect the typedef's done by the library, to make the critcl::cprocs as self-documenting as the underlying library functions themselves. 🦋

critcl::argtype Marpa_Symbol_ID     = int
critcl::argtype Marpa_Rule_ID       = int
critcl::argtype Marpa_Rule_Int      = int
critcl::argtype Marpa_Rank          = int
critcl::argtype Marpa_Earleme       = int
critcl::argtype Marpa_Earley_Set_ID = int

...

method sym-rank: proc {
    Marpa_Symbol_ID sym
    Marpa_Rank      rank
} Marpa_Rank {
    return marpa_g_symbol_rank_set (instance->grammar, sym, rank);
}

...

🧚 CUSTOM TYPES, SEMI-TRIVIAL 🧚

A more involved custom argument type would be to map from Tcl strings to some internal representation, like an integer code.

The first example is taken from the tclyaml package, a binding to the libyaml library. In a few places we have to map readable names for block styles, scalar styles, etc. to the internal enumeration.

critcl::argtype yaml_sequence_style_t {
    if (!encode_sequence_style (interp, @@, &@A)) return TCL_ERROR;
}

...

critcl::ccode {
    static const char* ty_block_style_names [] = {
        "any", "block", "flow", NULL
    };

    static int
    encode_sequence_style (Tcl_Interp* interp, Tcl_Obj* style,
                           yaml_sequence_style_t* estyle)
    {
        int value;
        if (Tcl_GetIndexFromObj (interp, style, ty_block_style_names,
                                 "sequence style", 0, &value) != TCL_OK) {
            return 0;
        }
        *estyle = value;
        return 1;
    }
}

...

method sequence_start proc {
    pstring anchor
    pstring tag
    int implicit
    yaml_sequence_style_t style
} ok {
    /* Syntax: <instance> seq_start <anchor> <tag> <implicit> <style> */
    ...
}

...

It should be noted that this code precedes the advent of the supporting generator package critcl::emap. Using the generator, the definition of the mapping becomes much simpler:

critcl::emap::def yaml_sequence_style_t {
    any   0
    block 1
    flow  2
}

Note that the generator will not only provide the conversions, but also define the argument and result types needed for their use by critcl::cproc. 💖

Another example of such a semi-trivial argument type can be found in the CRIMP package, which defines a Tcl_ObjType for image values. This not only provides a basic argument type for any image, but also derived types which check that the image has a specific format. Here we see for the first time non-integer arguments, and the need to define the C types used for variables holding the C level value, and the type of function parameters (Due to C promotion rules we may need different types).

critcl::argtype image {
    if (crimp_get_image_from_obj (interp, @@, &@A) != TCL_OK) {
        return TCL_ERROR;
    }
} crimp_image* crimp_image*

...

    set map [list <<type>> $type]
    critcl::argtype image_$type [string map $map {
        if (crimp_get_image_from_obj (interp, @@, &@A) != TCL_OK) {
            return TCL_ERROR;
        }
        if (@A->itype != crimp_imagetype_find ("crimp::image::<<type>>")) {
            Tcl_SetObjResult (interp,
                              Tcl_NewStringObj ("expected image type <<type>>",
                                                -1));
            return TCL_ERROR;
        }
    }] crimp_image* crimp_image*

...

👑 CUSTOM TYPES, SUPPORT STRUCTURES 👑

The adjunct command critcl::argtypesupport is for when the conversion needs additional definitions, for example a helper structure.

An example of this can be found among the standard types of critcl itself, the pstring type. This type provides the C function with not only the string pointer, but also the string length, and the Tcl_Obj* this data came from. As critcl::cproc's calling conventions allow us only one argument for the data of the parameter, a structure is needed to convey these three pieces of information. 📦

Thus the argument type is defined as

critcl::argtype pstring {
    @A.s = Tcl_GetStringFromObj(@@, &(@A.len));
    @A.o = @@;
} critcl_pstring critcl_pstring

critcl::argtypesupport pstring {
    typedef struct critcl_pstring {
        Tcl_Obj*    o;
        const char* s;
        int         len;
    } critcl_pstring;
}

In the case of such a structure being large we may wish to allocate it on the heap instead of having it taking space on the stack. If we do that we need another adjunct command, critcl::argtyperelease. This command specifies the code required to release dynamically allocated resources when the worker function returns, before the shim returns to the caller in Tcl.

To keep things simple our example is synthetic, a modification of pstring above, to demonstrate the technique. An actual, but more complex example is the code to support the variadic args argument of critcl::cproc. 🫶

critcl::argtype pstring {
    @A = (critcl_pstring*) ckalloc(sizeof(critcl_pstring));
    @A->s = Tcl_GetStringFromObj(@@, &(@A->len));
    @A->o = @@;
} critcl_pstring* critcl_pstring*

critcl::argtypesupport pstring {
    typedef struct critcl_pstring {
        Tcl_Obj*    o;
        const char* s;
        int         len;
    } critcl_pstring;
}
critcl::argtyperelease pstring {
    ckfree ((char*)) @A);
}

okay so heads up bestie 💖 the example above is literally the simplest case of an allocated argument — a conversion that cannot fail (it's just string retrieval, she's not out here doing tricks). If your conversion can fail though? 🫠 then you've got two options: either defer the allocation so it only happens on a successful conversion, or have the conversion code release the allocated memory itself in the failure path. Why? because on the failure path it will never reach the code you defined via critcl::argtyperelease. that's just how it is, no cap. 💅

🦋 CUSTOM TYPES, RESULTS 🦋

so everything we've talked about so far has been argument conversions — that's the Tcl → C direction, going in. Custom result types are the other way around: C → Tcl, coming back out. 🌈

ngl this one is usually easier, because most of the time errors shouldn't even be possible. 💀 and heads up: supporting structures, or allocating them on the heap, are not really required here and therefore not supported. periodt.

the example below got pulled from KineTcl. it's a variant of the builtin result type Tcl_Obj*, aka object. the builtin conversion assumes the object returned by the function has a refcount of 1 (or higher), with the function having held the reference, and it releases that reference after placing the value into the interp result. this conversion? totally different vibe — it assumes the value has a refcount of 0, and therefore that decrementing it is forbidden, lest it be released much too early and crash the system. 😭

critcl::resulttype KTcl_Obj* {
    if (rv == NULL) { return TCL_ERROR; }
    Tcl_SetObjResult(interp, rv);
    /* No refcount adjustment */
    return TCL_OK;
} Tcl_Obj*

this type of definition also shows up in Marpa, and recent hacking hacking on CRIMP introduced it there as well. which is why this definition became a builtin type starting with version 3.1.16, under the names Tcl_Obj*0 and object0. ✨ iconic behaviour honestly.

okay so going back to errors and their handling — of course, if a function we're wrapping signals them in-band, then the conversion of such results has to deal with that. happens for example in KineTcl, where we find:

critcl::resulttype XnStatus {
    if (rv != XN_STATUS_OK) {
        Tcl_AppendResult (interp, xnGetStatusString (rv), NULL);
        return TCL_ERROR;
    }
    return TCL_OK;
}

critcl::resulttype XnDepthPixel {
    if (rv == ((XnDepthPixel) -1)) {
        Tcl_AppendResult (interp,
                          "Inheritance error: Not a depth generator",
                          NULL);
        return TCL_ERROR;
    }
    Tcl_SetObjResult (interp, Tcl_NewIntObj (rv));
    return TCL_OK;
}

💅 HANDLING A VARIABLE NUMBER OF ARGUMENTS 💅

in A Simple Procedure we showed you how easy a translation to C can be. but is it still that easy when we introduce something moderately complex, like handling a variable number of arguments? 🦋 which is literally a feature you need to handle commands with options and optional arguments.

well... starting with version 3.1.16, critcl::cproc does have full support for optional arguments, args-style variadics, and default values, extending its range to everything covered by the builtin proc. the only thing not truly supported are options (i.e. flag arguments) of any kind. 🙈

for the moment, and for the example, let us pretend that we can use critcl::cproc only if the number of arguments is fully known beforehand, i.e. at the time of declaration. then we have to use critcl::ccommand to handle the translation of the procedure shown below:

proc math {args} {
    set sum 0
    foreach y $args { set sum [expr {$sum + $y}] }
    return $sum
}

its advantage: access to the low-level C arguments representing the Tcl arguments of the command. full control over argument conversion, argument validation, etc. 👑

its disadvantage: access to the low-level C arguments representing the Tcl arguments of the command. assuming the burden of having to write argument conversion, argument validation, etc. 😭 where critcl::cproc handles the task of converting from Tcl to C values (for arguments) and back (for the result), with critcl::command it is the developer who has to write all this code. not me having to do all that work fr fr.

under our restriction, the translation of the example is:

critcl::ccommand math {cd ip oc ov} {
    double sum = 0;
    double y;

    oc --;
    while (oc) {
        if (Tcl_GetDoubleFromObj (ip, ov[oc], &y) != TCL_OK) {
            return TCL_ERROR;
        }
        sum += y;
        oc --;
    }

    Tcl_SetObjResult (ip, Tcl_NewDoubleObj (sum));
    return TCL_OK:
}

notable about this translation:

  1. as promised/threatened, all the conversions between the Tcl and C domains are exposed, and the developer should know her way around Tcl's C API. 💀
  2. the four arguments "cd ip oc ov" are our names for the low-level arguments holding
    1. ClientData (reference)
    2. Tcl_Interp (reference)
    3. Number of arguments, and
    4. Array of argument values, each a Tcl_Obj*.

this list of arguments, while not optional in itself, is allowed to be empty, and/or to contain empty strings as argument names. if we do that, critcl will supply standard names for the missing pieces, namely:

  1. clientdata
  2. interp
  3. objc
  4. objv

now, letting go of our pretenses regarding the limitations of critcl::cproc — because of the support it does have for args-style variadics (since version 3.1.16) — we can write a much simpler translation. 🎀

critcl::cproc math {double args} double {
    double sum = 0;

    args.c --;
    while (args.c) {
        sum += args.v[args.c];
        args.c --;
    }
    return sum;
}

🌸 DATA AS A TCL COMMAND 🌸

here we assume that we have a Tcl procedure which returns a fixed string. in the final product we are going to C to hide this string from the casual user. 🙈

proc somedata {} {
      return {... A large blob of characters ...}
}

the translation of this is simple and easy:

package require critcl

critcl::cdata somedata {... A large blob of characters ...}

there is nothing really notable here. it just works. ✨

🔥 BLOCKS OF ARBITRARY C 🔥

often, just defining Tcl commands in C — like we demonstrated in A Simple Procedure, Handling A Variable Number Of Arguments, and Data As A Tcl Command — is not really enough. for example we may have several of our new C commands using the same code over and over, and we wish to avoid this duplication. or we wish to pull in declarations and definitions from some external library. 🫶

in both cases we require the ability to embed an unstructured block of C code which can contain whatever we want — defines, functions, includes, etc. — without being directly tied to Tcl commands. the command critcl::code provides us with exactly that. 💅

as our example now, an excerpt taken from real code, the top of the "sha1c.tcl" critcl file in the sha1 module of Tcllib:

critcl::ccode {
    #include "sha1.h"
    #include <stdlib.h>
    #include <assert.h>

    static
    Tcl_ObjType sha1_type; /* fast internal access representation */

    static void
    sha1_free_rep(Tcl_Obj* obj)
    {
        SHA1_CTX* mp = (SHA1_CTX*) obj->internalRep.otherValuePtr;
        Tcl_Free(mp);
    }

    ...
}

we see here the beginning of the C code defining a custom Tcl_ObjType holding the data of a SHA1 context used during the incremental calculation of sha1 hashes. 🧚

💖 CONSTANT VALUES 💖

while one might believe that there is no need for commands which return constant values, that is not true. commands reporting on compile-time configuration — like version numbers, available features, etc. — are at least one use case for such commands.

and the reason for creating critcl commands to support them? convenience to the user, yes, but further than that: the ability to optimize the internals, i.e. the generated code. 👑

a cproc would be easy to write, but it incurs overhead due to a superfluous work function. a ccommand has no overhead, except that of the user having to write the argument checking and result conversion. 😭

using critcl::cconst is both convenient and without code overhead. our example is a function found in package tcl-linenoise, that is, if cconst had existed at the time of writing. it returns a configuration value reporting to the policy layer if an extended mode for hidden input is available from the bound linenoise, or not.

critcl::cconst linenoise::hidden_extended boolean 1

🎀 LIFTING CONSTANTS 🎀

when writing a critcl-based package to make a third-party library available to scripts, we do not only have to make the relevant functions available as commands — often we also have to know all the various constants, flags, etc. that these functions take.

rather than writing such magic numbers directly, we would greatly prefer to use symbolic names instead. and instead of providing one or more commands to list and map the magic numbers to strings, critcl only provides a single command which allows the export of C defines and enumeration values, mapping them to Tcl variables of the given names, whose values are the associated magic numbers. ✨

this is good enough because the developers of the third-party library were very likely like us and wanted to use symbolic names instead of magic numbers. which in C are declared via defines and enumeration types. we just have to lift them up. 🦋

our example comes from cryptkit, a Tcl binding to cryptlib, a cryptography library. the command

critcl::cdefines CRYPT_* ::crypt

maps all Cryptlib specific #defines and enums into the namespace ::crypt, telling critcl to create aliases to the symbols. 🔥

similarly

critcl::cdefines {
    NULL
    TRUE
    FALSE
    TCL_OK
    TCL_ERROR
} ::crypt

maps the listed defines into the namespace ::crypt. 🌈

okay so genuinely important thing to clock: these commands do not create the defines at the C level. They literally only lift material that already exists. Where's it coming from? The headers of the third-party library, that's the usual case, but also from Blocks of arbitrary C. 💅

And the corollary, bestie: what is not there cannot be lifted. Every name and pattern you list that has no actual C code declaring it just gets ignored, i.e. not mapped. Periodt. ✨

🌸 FINDING HEADER FILES 🌸

There's a notable thing in the example from the section about Blocks of arbitrary C, and it's this line:

#include "sha1.h"

Where is this header even coming from?? 👀 Go look at the Tcllib module and you'll find the header is actually a sibling of the sha1c.tcl file that holds the embedded C code. But critcl does not know that. It has to be told. 🫶

Without that knowledge it will still invoke the compiler just fine — and the compilation will fail, because the header isn't on any of the include paths the compiler is using, so it simply won't be found. 💀

That's what critcl::cheaders is for. It lets us either tell the compiler the path(s) where the needed headers can be found:

critcl::cheaders -I/path/to/headers/

or tell it directly which headers we're using and where they live:

critcl::cheaders sha1.h

And now critcl knows sha1.h is important, and that it lives beside the .critcl file which referenced it (because of the relative path used). Note that this does not absolve us of the need to #include the header through a critcl::ccode block. This only tells critcl where it lives, so it can configure the compiler with the proper include paths to actually find it on use. ✨

Further note: a C development environment is usually already configured to find all the system headers, which obviates the need for a critcl::cheaders declaration when those are used. For those, a plain #include in a critcl::ccode block is good enough. In other words, the second form of invoking critcl::cheaders is pretty much only for headers which accompany the .critcl file. 💖

🎀 SEPARATE C SOURCES 🎀

In all of the examples shown so far the C code was fully embedded in a .critcl file. But if the C part is large, it can make sense to break it out of the .critcl file into one or more separate proper .c file(s). 🌈

critcl::csources is the command that makes this code known to the original .critcl file again. It accepts the paths to the .c files as arguments, and glob patterns as well. Our example comes from the struct::graph package in Tcllib — its core C functions live in separate files, and the .critcl code makes them known via:

namespace eval ::struct {
    # Supporting code for the main command.
    critcl::cheaders graph/*.h
    critcl::csources graph/*.c

    ...
}

Which tells critcl that these files are in the subdirectory graph relative to the location of graph_c.tcl, which is the relevant .critcl file. 🦋

This example also demonstrates again the use of critcl::cheaders, which we already saw in the section Finding header files. 💅

💜 FINDING EXTERNAL LIBRARIES 💜

When creating a package exposing some third-party library to Tcl, Finding header files is only the first part — it's what gets you a failure-free compilation. We also have to find the library/ies themselves so they can be linked to our package. That's what's described here. The last issue, Lifting constants from C to Tcl for the use by scripts, is handled in a separate section and example. 👑

The relevant command is critcl::clibraries. Its basic semantics are like those of critcl::cheaders, i.e. it enables us to tell the linker the path(s) where the required libraries can be found, using

critcl::clibraries -L/path/to/libraries/

name them

critcl::clibraries -lfoo

or tell it directly which libraries we are using and where they live:

critcl::clibraries /path/to/library/foo.so
This last way of using it should be avoided however, as it intermingles searching and naming, plus the name is platform dependent. 🙈

For OS X we additionally have the critcl::framework command, which enables us to name the frameworks used by our package. Note that this command can be used unconditionally — if the build target is not OS X, it is ignored. ✨

The commands critcl::cflags and critcl::ldflags enable you to provide custom options to the compile and link phases for a .critcl file. 🔥

This usually becomes necessary if the C code in question comes from an external library we're writing a Tcl binding for, with multiple configurations to select, non-standard header locations, and other things. Among the latter, especially platform-specific settings, for example byteorder.

This makes critcl::check an important adjunct command, as this is the API for Checking The Environment, and then selecting the compile & link flags to use. 💖

I currently have no specific example to demonstrate these commands. 😭

🌸 HAVING BOTH C AND TCL FUNCTIONALITY 🌸

Often enough only pieces of a package require recoding in C to boost the whole system. Or, alternatively, the package in question consists of a low-level layer in C with a Tcl layer above it encoding policies and routing to the proper low-level calls, creating a nicer (high-level) API to the low-level functionality, etc. 🦋

For all of this we have to be able to write a package which contains both C and Tcl — nevermind the fact that the C parts are embedded in Tcl. ✨

The easiest way to structure such a package is to have several files, each with a different duty. First, a .critcl file containing the embedded C, and second one or more .tcl files providing the Tcl parts. Then use the critcl::tsources command in the .critcl file to link the two parts together, declaring the .tcl files as necessary companions of the C part. 💅

package require critcl

critcl::tsources your-companion.tcl ; # Companion file to use

... embedded C via critcl commands ...

With a declaration as shown above the companion file will be automatically sourced when the C parts are made available, thus making the Tcl parts available as well. 💖

👑 USING C WITH TCL FUNCTIONALITY AS FALLBACK 👑

There is one special case of Having both C and Tcl functionality which deserves its own section. 🌈

The possibility of not having the fast C code on some platform, and using a slower Tcl implementation of the functionality. In other words, a fallback which keeps the package working in the face of failure to build the C parts. A more concrete example of this would be a module implementing the SHA hash, in both C and Tcl, and using the latter if and only if the C implementation is not available. 🎀

There are two major possibilities in handling such a situation.

  1. Keep all the pieces separated. In that scenario our concrete example would be spread over three packages. Two low-level packages sha::c and sha::tcl containing the two implementations of the algorithm, and, thirdly, a coordinator package sha which loads either of them, based on availability.

The Tcllib bundle of packages contains a number of packages structured in this manner, mostly in the struct module.

Writing the C and Tcl parts should be simple by now, with all the examples we had so far. The only non-trivial part is the coordinator, and even that if and only if we wish to make it easy to write a testsuite which can check both branches, C, and Tcl without gymnastics. So, the most basic coordinator would be

set sha::version 1
if {[catch {
    package require sha::c $sha::version
}]} {
    package require sha::tcl $sha::version
}
package provide sha $sha::version

It tries to load the C implementation first, and falls back to the Tcl implementation if that fails. The code as is assumes that both implementations create exactly the same command names, leaving the caller unaware of the choice of implementations. ✨

A concrete example of this scheme can be found in Tcllib's md5 package. While it actually uses the Trf as its accelerator, and not a critcl-based package, the principle is the same. It also demonstrates the need for additional glue code when the C implementation doesn't exactly match the signature and semantics of the Tcl implementation. 💅

This basic coordinator can be easily extended to try more than two packages to get the needed implementation. For example, the C implementation may not just exist in a sha::c package, but also be bundled somewhere else. Tcllib, for example, has a tcllibc package which bundles all the C parts of its packages which have them in a single binary. 🫶

Another direction to take it in is to write code which allows the loading of multiple implementations at the same time, and then switching between them at runtime. Doing this requires effort to keep the implementations out of each others way, i.e. they cannot provide the same command names anymore, and a more complex coordinator as well, which is able to map from the public command names to whatever is provided by the implementation. 👑

The main benefit of this extension is that it makes testing the two different implementations easier — simply run through the same set of tests multiple times, each time with a different implementation active. The disadvantage is the additional complexity of the coordinator's internals. As a larger example of this technique here is the coordinator modules/struct/queue.tcl handling the C and Tcl implementations of Tcllib's struct::queue package:

# queue.tcl --
#       Implementation of a queue data structure for Tcl.

package require Tcl 8.4
namespace eval ::struct::queue {}

## Management of queue implementations.

# ::struct::queue::LoadAccelerator --
#       Loads a named implementation, if possible.
proc ::struct::queue::LoadAccelerator {key} {
    variable accel
    set r 0
    switch -exact -- $key {
        critcl {
            # Critcl implementation of queue requires Tcl 8.4.
            if {![package vsatisfies [package provide Tcl] 8.4]} {return 0}
            if {[catch {package require tcllibc}]} {return 0}
            set r [llength [info commands ::struct::queue_critcl]]
        }
        tcl {
            variable selfdir
            if {
                [package vsatisfies [package provide Tcl] 8.5] &&
                ![catch {package require TclOO}]
            } {
                source [file join $selfdir queue_oo.tcl]
            } else {
                source [file join $selfdir queue_tcl.tcl]
            }
            set r 1
        }
        default {
            return -code error "invalid accelerator/impl. package $key: must be one of [join [KnownImplementations] {, }]"
        }
    }
    set accel($key) $r
    return $r
}

# ::struct::queue::SwitchTo --
#       Activates a loaded named implementation.

proc ::struct::queue::SwitchTo {key} {
    variable accel
    variable loaded

    if {[string equal $key $loaded]} {
        # No change, nothing to do.
        return
    } elseif {![string equal $key ""]} {
        # Validate the target implementation of the switch.

        if {![info exists accel($key)]} {
            return -code error "Unable to activate unknown implementation \"$key\""
        } elseif {![info exists accel($key)] || !$accel($key)} {
            return -code error "Unable to activate missing implementation \"$key\""
        }
    }

    # Deactivate the previous implementation, if there was any.

    if {![string equal $loaded ""]} {
        rename ::struct::queue ::struct::queue_$loaded
    }

    # Activate the new implementation, if there is any.

    if {![string equal $key ""]} {
        rename ::struct::queue_$key ::struct::queue
    }

    # Remember the active implementation, for deactivation by future
    # switches.

    set loaded $key
    return
}

# ::struct::queue::Implementations --
#       Determines which implementations are
#       present, i.e. loaded.

proc ::struct::queue::Implementations {} {
    variable accel
    set res {}
    foreach n [array names accel] {
        if {!$accel($n)} continue
        lappend res $n
    }
    return $res
}

# ::struct::queue::KnownImplementations --
#       Determines which implementations are known
#       as possible implementations.

proc ::struct::queue::KnownImplementations {} {
    return {critcl tcl}
}

proc ::struct::queue::Names {} {
    return {
        critcl {tcllibc based}
        tcl    {pure Tcl}
    }
}

## Initialization: Data structures.

namespace eval ::struct::queue {
    variable  selfdir [file dirname [info script]]
    variable  accel
    array set accel   {tcl 0 critcl 0}
    variable  loaded  {}
}

## Initialization: Choose an implementation,
## most preferred first. Loads only one of the
## possible implementations. And activates it.

namespace eval ::struct::queue {
    variable e
    foreach e [KnownImplementations] {
        if {[LoadAccelerator $e]} {
            SwitchTo $e
            break
        }
    }
    unset e
}

## Ready

namespace eval ::struct {
    # Export the constructor command.
    namespace export queue
}

package provide struct::queue 1.4.2

So in this implementation the coordinator literally renames the commands of the low-level packages to the public commands, which makes the future dispatch as fast as if the commands had these names anyway — but it also forces a spike of bytecode recompilation if switching is ever done at the runtime of an application, and not just used for testing. And it can possibly disrupt introspection by the commands, especially if they move between different namespaces. 🙈

A different implementation would be to provide the public commands as procedures which consult a variable to determine which of the loaded implementations is active, and then call on its commands. This doesn't disrupt introspection, nor does it trigger bytecode recompilation on switching. But it takes more time to dispatch to the actual implementation, in <em>every</em> call of the public API for the package in question. 💀

A concrete example of this scheme can be found in Tcllib's crc32 package. go look bestie 🫶

  1. Mix the pieces together. Please note that while I am describing how to make this work I strongly prefer and recommend to use the previously shown approach using separate files/packages. It is much easier to understand and maintain. With this warning done, lets go into the nuts and bolts.

If we care only about mode "compile & run" things are easy:

package require critcl

if {![critcl::compiling]} {
    proc mycommand {...} {
        ...
    }

} else {
    critcl::cproc mycommand {...} {
        ...
    }
}

The command critcl::compiling tells us whether we have a compiler available or not, and in the latter case we implement our command in Tcl. 💅

Now what happens when we invoke mode "generate package"? ... compiler failure ... ... ok - C code - everything fine ... fail - no package ? or just no C code ? declare self as tsource, to be used ? ... platform-specific C/Tcl -- uuid.

🫶 UNLAZY PACKAGES 🫶

By default critcl is a bit inconsistent between modes "compile & run" and "generate package". The result of the latter is a standard Tcl package which loads and sources all of its files immediately when it is required. Whereas "compile & run" defers actual compilation, linking, and loading until the first time one of the declared commands is actually used, making this very lazy. 😴

This behaviour can be quite unwanted if Tcl companion files, or other users of the C commands use introspection to determine the features they have available. Just using [info commands] doesn't cut it, the auto_index array has to be checked as well, making things quite inconvenient for the users. 😭

To fix this issue at the source, instead of in each user, be it inside of the package itself, or other packages, we have the command critcl::load. Used as the last command in a ".critcl" file it forces the compile, link, and load trinity, ensuring that all C commands are available immediately. ✨

package require critcl

... Declare C procedures, commands, etc.

critcl::load ; # Force build and loading.

Note that is not allowed, nor possible to use critcl commands declaring anything after critcl::load has been called. I.e., code like

package require critcl

... Declare C procedures, commands, etc.

critcl::load ; # Force build and loading.

... More declarations of C code, ...
critcl::code { ... }

will result in an error. 🚨 The only package-related commands still allowed are

  1. critcl::done
  2. critcl::failed
  3. critcl::load

as these only query information, namely the build status, and are protected against multiple calls. 💖

✨ CHECKING YOUR C ✨

Like I said (multiple times, bestie, sorry not sorry) — by default critcl defers the compile and link steps for a file until it's actually needed. And "needed" means: the first command of the .critcl file in question actually getting invoked. 💅

So that's not just the lazily-loading-the-package's-functionality thing. It also means that when you're developing in mode compile & run, you straight up don't see any errors in your code until you're actually trying to run some demonstration. 😭

If we do not wish to have such a delay, we gotta be able to force at least the execution of the compile step.

And that's literally what critcl::failed is for. When called it forcibly builds the C code for the .critcl file it is part of, and returns a boolean value signaling failure (true) or success (false).

package require critcl

... Declare C procedures, commands, etc.

if {[critcl::failed]} {
    ... signal error
}

It's related and similar to critcl::load, the command to overcome the lazy loading, as shown in section Unlazy Packages. 🦋

⚠️ It is not allowed, nor possible, to use critcl commands declaring anything after critcl::failed has been called — which makes it pretty much the last critcl command in a .critcl file. periodt.

Code like

package require critcl

... Declare C procedures, commands, etc.

if {[critcl::failed]} { ... }

... More declarations of C code, ...
critcl::code { ... }

will result in an error. 💀 The only package-related commands still allowed are

  1. critcl::done
  2. critcl::failed
  3. critcl::load

as these only query information, namely the build status, and are protected against multiple calls. 🫶

🌈 WHICH TCL ? 🌈

When building the shared library from the embedded C sources, one of the things critcl does for us is provide the Tcl headers — especially the stubs declarations. 👑

By default these are the Tcl 8.4 headers and stubs, which covers 90% of the cases. But what about when the package in question is meant for use with Tcl 8.5 or higher, using C-level features of this version of Tcl? 🤔

Use critcl::tcl to declare to critcl the minimum version of Tcl required to operate the package. This can be either 8.4, 8.5, or 8.6, and critcl then supplies the proper headers and stubs. ✨

package require critcl
critcl::tcl 8.5

... Declare your code ...

💖 MAKING A WIDGET 💖

... requires compiling against the Tk headers, and linking with Tk's stubs. For our convenience we have a simple, single command to activate all the necessary machinery, with critcl supplying the header files and stubs C code — instead of having to make it work on our own via critcl::cflags, critcl::ldflags, critcl::cheaders, critcl::csources.

That command is critcl::tk. 🎀

package require critcl
critcl::tk ; # And now critcl knows to put in the Tk headers and other support.

... Declare your code ...

Please note that this doesn't release you from the necessity of learning Tk's C API and how to use it to make a widget work. Sorry. 😭

🔍 CHECKING THE ENVIRONMENT 🔍

... may be necessary when creating a binding to some third-party library. The headers for this library may be found in non-standard locations, ditto for the library/ies itself. We may not have the headers and/or library on the build host. Types with platform-dependent sizes and definitions. Endianness issues. Any number of things. 💀

TEA-based packages can use autoconf and various predefined macros to deal with all this. We have the Power Of Tcl (tm) and critcl::check. 💅

This command takes a piece of C code as argument, like critcl::ccode. Instead of saving it for later, it however tries to compile it immediately, using the current settings, and then returns a boolean value reporting on the success (true) or failure (false). From there we can then branch to different declarations. 🌸

As an example, let's check for the existence of some header "FOO.h":

package require critcl

if {[critcl::check {
    #include <FOO.h>
}]} {
    ... Code for when FOO.h is present.
} else {
    ... Code for when FOO.h is not present.
}

Should we, on the other hand, wish to search for the header ourselves, in non-standard locations, we have the full power of Tcl available, i.e. loops, the file and glob commands, etc., which can then be followed by a critcl::cheader command to declare the location we found (See also Finding header files). 🦋

A nice extension to critcl would be a package collecting procedures for common tasks like that, sort of like an autoconf for Tcl. critcl::config seems to be a nice name for such a package.

Obvious adjunct commands which can be driven by results from critcl::check are

  • critcl::cflags
  • critcl::cheaders
  • critcl::clibraries
  • critcl::framework
  • critcl::ldflags

Less obvious, yet still valid, are also

  • critcl::ccode
  • critcl::ccommand
  • critcl::cdata
  • critcl::cproc
  • critcl::csources
  • critcl::ctsources

and pretty much everything else you can imagine. 🔥

📜 LICENSE INVOKED 📜

When writing packages it is always good manners to provide prospective users with the license the package is under, so that they can decide whether they truly want to use the package, or not. 💜

As critcl-based packages often consist of only a single file, a nice way of doing that is to embed the license in that file. By using a critcl command, namely critcl::license, this information is then also available to the critcl application, which can put it into a standard location, i.e. "license.terms", of the generated packages. 👑

I currently have no specific example to demonstrate the command. 🙈

🌈 BUILDING CRITCL PACKAGES 🌈

Okay bestie, this section is for the developers 💖 — the ones who have to actually generate packages out of .critcl files, i.e. real binaries you can deploy. It's giving build pipeline energy. Let's go. ✨

GETTING HELP ... 🆘

... is easy, babes. Just run

critcl -help

and it prints the basics of how to use the application to stdout. That's it. That's the whole vibe check. 🫶

PRE-FILLING THE RESULT CACHE 🎀

So critcl's default mode is this: you hand it a bunch of .critcl files, it builds their binaries, and it leaves them behind in the result cache. Then later, when those files actually get used, the compile and link steps can be straight up skipped — which means way shorter load times. It's giving pre-gaming, but for compilation. 🧚

The command line for that is

critcl foo.tcl

or, if you wanna process multiple files,

critcl foo.tcl bar.tcl ...
One thing to be aware of, bestie: if critcl finds that the cache already contains the results for your input files, no building will happen. None. Zero. If you're sure those results are outdated, use the option -force to force(sic!) critcl to rebuild the binaries.
critcl -force foo.tcl

And for debugging purposes it may be handy to peek at the generated intermediate .c files too 👀. You can stop them from being removed from the cache by specifying the option -keep.

critcl -keep foo.tcl

These can be combined, of course. Mix and match, slay. 💅

BUILDING A PACKAGE 📦

To build the binary package for a .critcl file — instead of Pre-Filling The Result Cache — simply specify the option -pkg.

critcl -pkg foo.tcl

This will generate a package named foo. A simpler alternative to the above is

critcl -pkg foo

and the application will automatically assume that the input file to look for is "foo.tcl". Iconic, honestly. ✨

But what when the name of the input file is not the name of the package to build? Bestie, we can handle that too:

critcl -pkg foo bar.tcl

The argument foo specifies the name, and "bar.tcl" is the file to process. 🎯

Going back to the very first example, it is of course possible to use an absolute path to specify the file to process:

critcl -pkg /path/to/foo.tcl

The package name derived from that is still foo. The path does not get to name your baby. 💅

BUILDING AND INSTALLING A PACKAGE 🛠️

Here we're assuming you know the basics of how to build a package. If not, please read the section Building A Package first. (No skipping, babe. 🙈)

By default critcl will place all newly-made packages in the subdirectory "lib" of the current working directory. I.e. running

critcl -pkg foo

will create the directory "lib/foo" which contains all the files of the package. Cozy. 🏠

When this behaviour is unwanted, the option -libdir is available — it allows the explicit specification of the destination location to use.

critcl -pkg -libdir /path/to/packages foo

A common use might be to not only build the package in question, but to also immediately install it directly in the path where the user's tclsh will be able to find it. Assuming, for example, that the tclsh in question is installed at "/path/to/bin/tclsh", with the packages searched for under "/path/to/lib" ([info library]), the command

critcl -pkg -libdir /path/to/lib foo

will build the package and place it in the directory "/path/to/lib/foo". Build and install in one breath. Main character energy. 👑

BUILDING FOR DEBUGGING 🔍

Here we're assuming you know the basics of how to build a package. If not, please read the section Building A Package first. (Again, no skipping. 🫶)

An important issue: when there is trouble with the package, debugging becomes necessary — an evil. 😭 Critcl supports this through the -debug option. Using it enables various build modes which help with that.

For example, to activate the Tcl core's built-in memory debugging subsystem, build your package with

critcl -pkg -debug memory foo

The resulting binary for package foo will use Tcl's debug-enabled (de)allocation functions, making them visible to Tcl's memory command. This of course assumes that the Tcl core used was also built for memory debugging. No cap — if it wasn't, this ain't gonna do what you want.

Further, build your package with

critcl -pkg -debug symbols foo

to see foo's symbols (types, functions, variables, etc.) when inspecting a "core" file it is involved in with a symbolic debugger. 🦋

To activate both memory debugging and symbols use either

critcl -pkg -debug all foo

or

critcl -pkg -debug symbols -debug memory foo

Two paths, same slay. ✨

RETARGETING THE BINARIES 🎯

The configuration settings critcl uses to drive the compiler, linker, etc. are by default selected based on the platform it is run on, to generate binaries which properly work on this platform. Sensible. 💅

There is one main use-case for overriding this selection, and it's done with the option -target:

  1. Cross-compilation. The building of binaries for a platform T while critcl actually runs on platform B. The standard configuration of critcl currently has settings for two cross-compilation targets. So, to build 32bit Windows binaries on a Linux host which has the Xmingw cross-compilation development environment installed, use
critcl -pkg -target mingw32 foo

Similarly, building a package for use on ARM processors while critcl is running in an Intel environment, use

critcl -pkg -target linux-arm foo
Note that both configurations assume that the cross-compiling compiler, linker, etc. are found first in the PATH. Sort your PATH out, bestie. 🙈

CUSTOM CONFIGURATIONS 🧰

The compiler configurations coming with critcl currently cover all hosts having gcc installed (the foremost among these are Linux and OS X), plus the native compilers of the more common unix-like operating systems, i.e. Solaris, HP-UX, and AIX, and, on the non-unix side, Windows. That's the whole roster. 🌈

Developers using operating systems and compilers outside of this range will either have to install a gcc-based development environment — i.e. get into the covered range — or write their own custom configuration and then tell critcl about it.

The latter is the easier part, given that critcl supports the option -config, whose argument is the path to the file containing the custom configuration(s). I.e.

critcl -config /path/to/config ...

will run critcl with the custom configuration in "/path/to/config", with the other options and arguments as explained in previous sections. Depending on the choice of name for the new configuration(s), this may or may not require a -target option to select the configuration needed. 🤔

For the former — the writing of the custom configuration — the reader is referred to the section "Configuration Internals" of the CriTcl Package Reference for the necessary details. This is an advanced topic, pretty much out of scope for this tutorial beyond what was already said. Respectfully, that's above our pay grade here. 💜

CUSTOM HEADER PATH 📂

Sometimes the use of critcl::headers might not be enough for a package to find its headers. Maybe they're outside of the paths checked by the setup code. 😭 To help, the application recognizes the option -I, which allows the user to supply a single additional include path to use during the build phase of the package.

Simply use

critcl -I /path/to/header ...

and the specified header will be handed to the package to be built. One path, one job. 🎀

INTROSPECTION OF TARGETS AND CONFIGURATIONS 🔮

To see a list containing the names of all the available configurations, run

critcl -targets

The configuration settings for either the default or user-chosen target can be inspected on stdout with

critcl -show

and

critcl -show -target TARGET

The raw contents of the configuration file used by critcl are dumped to stdout with

critcl -showall

All of the above can of course be combined with custom configuration files. Peak transparency, honestly. 👀✨

AUTHORS 👑

Jean Claude Wippler, Steve Landers, Andreas Kupries

BUGS, IDEAS, FEEDBACK 🐛💡

This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report them at https://github.com/andreas-kupries/critcl/issues. Ideas for enhancements you may have for either package, application, and/or the documentation are also very welcome and should be reported at https://github.com/andreas-kupries/critcl/issues as well. Come through, bestie. 🫶

KEYWORDS 🔑

C code, Embedded C Code, code generator, compile & run, compiler, dynamic code generation, dynamic compilation, generate package, linker, on demand compilation, on-the-fly compilation

CATEGORY 🏷️

Glueing/Embedded C code

Copyright (c) Jean-Claude Wippler

Copyright (c) Steve Landers

Copyright (c) 2011-2018 Andreas Kupries

And that's a wrap, bestie!! 🎉 This whole page is from the doc source, version 3.1.18.1, and it's the critcl_use(3tcl) manual page. You just read the entire thing. Not everyone can say that. 💅✨🦋 Go build something iconic. 🫶

📦 where this page comes from