TAP::H...Beyond(3perl)

✨ NAME ✨

Test::Harness::Beyond — beyond make test, bestie. This is the TAP::H...Beyond(3perl) page from the Perl Programmers Reference Guide, and we're gonna take the humble test run and make it serve.

---

🎀 Beyond make test 🎀

Okay so. Test::Harness is the whole vibe behind running your test scripts — it runs them, reads their output, and reports back whether you slayed or flopped. When you type make test (or ./Build test) for a module, Test::Harness is usually the thing doing the running. Not every module uses it, but the majority absolutely do. Periodt.

So to actually explore what Test::Harness can do, we gotta switch from make test to the prove command (which ships with Test::Harness, iconic). For all the examples below you'll also want a recent version of Test::Harness installed — 3.14 is current as of when this was written. Just a heads up. 🫶

For the examples I'm assuming we're working with a 'normal' Perl module distribution. Specifically: typing make or ./Build makes the built, ready-to-install module code show up below ./blib/lib and ./blib/arch, and there's a directory called t holding our tests. Test::Harness is not hardwired to that setup, it's just that assuming it saves me from explaining where every file lives for every single example. Efficiency. 💅

Back to prove — like make test, it runs a test suite, but it gives you so much more control over which tests run, in what order, and how their results get reported. Typically make test runs all the test scripts below the t directory. To do the same thing with prove I type:

prove -rb t

The switches here: -r recurses into any directories below t, and -b adds ./blib/lib and ./blib/arch to Perl's include path so the tests can find the code they're testing. Now listen — if I'm testing a module where an earlier version is already installed, I have to be careful with the include path so I'm not accidentally running my tests against the installed version instead of the shiny new one I'm working on. 🙈

And unlike make test, typing prove does not automatically rebuild my module. If I forget to make before prove, I'll be testing against older versions of those files — which inevitably leads to confusion, every time, no cap. So either I get into the habit of typing:

make && prove -rb t

or — if I have no XS code that needs building — I just use the modules below lib instead:

prove -Ilib -r t

So far I've shown you literally nothing that make test doesn't already do. Let's fix that. 🔥

---

💾 Saved State 💾

If I've got failing tests in a suite with more than a handful of scripts, and it takes more than a few seconds to run, it rapidly becomes tedious to run the whole thing over and over while I'm tracking down problems. So annoying. 😭

I can tell prove to just run the tests that are failing, like this:

prove -b t/this_fails.t t/so_does_this.t

That speeds things up, but now I have to make a note of which tests are failing and remember to run those specific ones. Instead — main character move — I use prove's --state switch and let it keep track of failing tests for me. First I do a complete run of the test suite and tell prove to save the results:

prove -rb --state=save t

That stores a machine readable summary of the test run in a file called .prove in the current directory. Then if I have failures, I can run just the failing scripts like this:

prove -b --state=failed

I can also tell prove to save the results again so it updates its idea of which tests failed:

prove -b --state=failed,save

And the second one of my failing tests passes, it gets removed from the failed list. Eventually I fix them all and prove can find no failing tests to run at all:

Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
Result: NOTESTS

Bestie that NOTESTS is a whole mood. 🦋

Now — as I work on one particular part of my module, it's most likely that the tests covering that code will fail. I'd love to run the whole suite but have it prioritize these 'hot' tests. So I tell prove:

prove -rb --state=hot,save t

All the tests run, but the ones that failed most recently run first. If no tests have failed since I started saving state, all tests run in their normal order. It's full test coverage plus early notification of failures. Obsessed. ✨

The --state switch supports a bunch of options. For example, to run failed tests first followed by all remaining tests ordered by the timestamps of the test scripts — and save the results — I can use:

prove -rb --state=failed,new,save t

See the prove documentation (type prove --man) for the full list of state options.

Heads up: when I tell prove to save state it writes a file called .prove (_prove on Windows) in the current directory. It's a YAML document, so it's quite easy to write your own tools that work on the saved test state — but the format isn't officially documented, so it might change without (much) warning in the future. You've been warned. 💀

---

🏎️ Parallel Testing 🏎️

If my tests take too long to run, I may be able to speed them up by running multiple test scripts in parallel. This is particularly effective if the tests are I/O bound or if I've got multiple CPU cores. I tell prove to run my tests in parallel like this:

prove -rb -j 9 t

The -j switch enables parallel testing; the number that follows it is the maximum number of tests to run in parallel.

Now, sometimes tests that pass when run sequentially will fail when run in parallel. For example, if two different test scripts use the same temporary file, or attempt to listen on the same socket, I'll have problems running them in parallel. If I see unexpected failures I need to check my tests to work out which of them are trampling on the same resource, then rename temporary files or add locks as appropriate. Fix it, don't just vibe through it. 🙈

And to get the most performance benefit, I want the test scripts that take the longest to run to start first — otherwise I'll be sitting there waiting for the one test that takes nearly a minute to finish after all the others are already done. Lame. So I use the --state switch to run the tests in slowest to fastest order:

prove -rb -j 9 --state=slow,save t

---

🌈 Non-Perl Tests 🌈

The Test Anything Protocol (http://testanything.org/) isn't just for Perl. Just about any language can be used to write tests that output TAP — there are TAP-based testing libraries for C, C++, PHP, Python and many others. And if I can't find a TAP library for my language of choice, it's easy to generate valid TAP myself. It looks like this:

1..3
ok 1 - init OK
ok 2 - opened file
not ok 3 - appended to file

The first line is the plan — it specifies the number of tests I'm going to run, so it's easy to check that the test script didn't exit before running all the expected tests. The following lines are the test results: ok for pass, not ok for fail. Each test has a number and, optionally, a description. And that's it. Any language that can produce output like that on STDOUT can be used to write tests. Slay. 💅

Recently I've been rekindling a two-decades-old interest in Forth. Evidently I have a masochistic streak that even Perl can't satisfy. 😭 I want to write tests in Forth and run them using prove — you can find my gforth TAP experiments at https://svn.hexten.net/andy/Forth/Testing/. I can use the --exec switch to tell prove to run the tests using gforth like this:

prove -r --exec gforth t

Alternately, if the language used to write my tests allows a shebang line, I can use that to specify the interpreter. Here's a test written in PHP:

#!/usr/bin/php
<?php
  print "1..2\n";
  print "ok 1\n";
  print "not ok 2\n";
?>

If I save that as t/phptest.t, the shebang line will make sure it runs correctly right along with all my other tests. Cute. 🎀

---

🔀 Mixing it up 🔀

Subtle interdependencies between test programs can mask problems — for example, an earlier test may neglect to remove a temporary file that affects the behaviour of a later test. Sneaky. To find this kind of problem I use the --shuffle and --reverse options to run my tests in random or reversed order.

---

🛠️ Rolling My Own 🛠️

If I need a feature that prove doesn't provide, I can easily write my own. Iconic behaviour, honestly.

Typically you'll want to change how TAP gets input into and output from the parser. App::Prove supports arbitrary plugins, and TAP::Harness supports custom formatters and source handlers that you can load using either prove or Module::Build — there are many examples to base yours on. For more details see App::Prove, TAP::Parser::SourceHandler, and TAP::Formatter::Base.

And if writing a plugin isn't enough, you can write your own test harness. One of the motives for the 3.00 rewrite of Test::Harness was literally to make it easier to subclass and extend. King move. 👑

The Test::Harness module is a compatibility wrapper around TAP::Harness. For new applications I should use TAP::Harness directly. And as we'll see, prove uses TAP::Harness.

When I run prove it processes its arguments, figures out which test scripts to run, and then passes control to TAP::Harness to run the tests, parse, analyse and present the results. So by subclassing TAP::Harness I can customise many aspects of the test run.

Me? I want to log my test results in a database so I can track them over time. To do that I override the summary method in TAP::Harness. I start with a simple prototype that dumps the results as a YAML document:

package My::TAP::Harness;

use base 'TAP::Harness';
use YAML;

sub summary {
  my ( $self, $aggregate ) = @_;
  print Dump( $aggregate );
  $self->SUPER::summary( $aggregate );
}

1;

Now I need to tell prove to use my My::TAP::Harness. If My::TAP::Harness is on Perl's @INC include path I can do:

prove --harness=My::TAP::Harness -rb t

If I don't have My::TAP::Harness installed on @INC, I need to provide the correct path to perl when I run prove:

perl -Ilib `which prove` --harness=My::TAP::Harness -rb t

I can also bake these options into my own version of prove. It's pretty simple — most of the work of prove is handled by App::Prove. The important code in prove is just:

use App::Prove;

my $app = App::Prove->new;
$app->process_args(@ARGV);
exit( $app->run ? 0 : 1 );

And if I write a subclass of App::Prove I can customise any aspect of the test runner while inheriting all of prove's behaviour. Here's myprove:

#!/usr/bin/env perl use lib qw( lib );      # Add ./lib to @INC
use App::Prove;

my $app = App::Prove->new;

# Use custom TAP::Harness subclass
$app->harness( 'My::TAP::Harness' );

$app->process_args( @ARGV ); exit( $app->run ? 0 : 1 );

Now I can run my tests like this:

./myprove -rb t

---

🧚 Deeper Customisation 🧚

Now that I know how to subclass and replace TAP::Harness, I can replace any other part of the harness. To do that I need to know which classes are responsible for which functionality. Here's a brief guided tour — the default class for each component is shown in parentheses. Normally any replacements I write will be subclasses of these default classes. 💖

okay so when I run my tests, TAP::Harness is out here assembling a whole little squad for me. It makes a scheduler (TAP::Parser::Scheduler) to figure out what order the tests run in, an aggregator (TAP::Parser::Aggregator) to scoop up all the results and analyse them, and a formatter (TAP::Formatter::Console) to actually show me those results. 💅

And if I'm running tests in parallel? There might also be a multiplexer (TAP::Parser::Multiplexer) in the mix — that's the component that lets multiple tests run at the same time. slay

Once all those helpers exist, TAP::Harness starts running tests. For every single test it spins up a brand-new parser (TAP::Parser), and that parser's whole job is running the test script and parsing what it spits out. 🎀

So if I wanna swap any of these components out, I just call one of these harness methods with the name of the replacement class:

aggregator_class
formatter_class
multiplexer_class
parser_class
scheduler_class

Like, say I want to replace the aggregator — easy:

$harness->aggregator_class( 'My::Aggregator' );

Or, alternate route: I can hand the names of my substitute classes straight to the TAP::Harness constructor:

my $harness = TAP::Harness->new(
  { aggregator_class => 'My::Aggregator' }
);

Now if I need to go even deeper into the internals of the harness, I can replace the classes that TAP::Parser uses to execute test scripts and tokenise their output. Before running a test script, TAP::Parser creates a grammar (TAP::Parser::Grammar) to decode the raw TAP into tokens, a result factory (TAP::Parser::ResultFactory) to turn those decoded TAP results into objects, and — depending on whether it's running a test script or reading TAP from a file, scalar or array — a source or an iterator (TAP::Parser::IteratorFactory). 💖

Each of these objects can be swapped out by calling one of these parser methods:

source_class
perl_source_class
grammar_class
iterator_factory_class
result_factory_class

---

✨ Callbacks ✨

So instead of subclassing the components I want to change, there's another option: I can attach callbacks to the default classes. TAP::Harness exposes these ones:

• parser_args — tweak the parameters used to create the parser
• made_parser — just made a new parser
• before_runtests — about to run tests
• after_runtests — have run all tests
• after_test — have run an individual test script

TAP::Parser supports callbacks too, bestie. 🦋 bailout, comment, plan, test, unknown, version and yaml all get called for the corresponding TAP result types. ALL gets called for all results. ELSE gets called for all results that don't have a named callback installed. And EOF gets called once at the end of each TAP stream. 👑

To install a callback I pass the name of the callback plus a subroutine reference to TAP::Harness or TAP::Parser's callback method:

$harness->callback( after_test => sub {
  my ( $script, $desc, $parser ) = @_;
} );

I can also pass callbacks straight to the constructor:

my $harness = TAP::Harness->new({
  callbacks => {
          after_test => sub {
      my ( $script, $desc, $parser ) = @_;
      # Do something interesting here
          }
  }
});

When it comes to changing how the test harness behaves, there are — periodt — more than one way to do it. Which way is best totally depends on my requirements. General rule: if I only want to observe test execution without changing the harness' behaviour (like logging test results to a database) then callbacks are the move. But if I want the harness to actually behave differently, subclassing gives me way more control. 💀

---

🧚 Parsing TAP 🧚

But maybe I don't need a whole entire test harness? If I already have a TAP test log sitting there that needs parsing, all I really need is TAP::Parser and the various classes it depends upon. Here's the code to run a test and parse its TAP output:

use TAP::Parser;

my $parser = TAP::Parser->new( { source => 't/simple.t' } );
while ( my $result = $parser->next ) {
  print $result->as_string, "\n";
}

Alternatively, I can pass an open filehandle as the source and have the parser read from that instead of trying to run a test script:

open my $tap, '<', 'tests.tap'
  or die "Can't read TAP transcript ($!)\n";
my $parser = TAP::Parser->new( { source => $tap } );
while ( my $result = $parser->next ) {
  print $result->as_string, "\n";
}

This approach is lowkey perfect if I need to convert my TAP based test results into some other representation. See TAP::Convert::TET (http://search.cpan.org/dist/TAP-Convert-TET/) for an example of this approach. 🌈

---

🌸 Getting Support 🌸

The Test::Harness developers hang out on the tapx-dev mailing list[1]. For discussion of general, language independent TAP issues there's the tap-l[2] list. And finally there's a wiki dedicated to the Test Anything Protocol[3]. Contributions to the wiki, patches and suggestions are all welcome — they want the lore, fr fr. 🫶

[1] http://www.hexten.net/mailman/listinfo/tapx-dev

[2] http://testanything.org/mailman/listinfo/tap-l

[3] http://testanything.org/

---

and that's a wrap on this page, bestie 🎀 this was perl v5.44.0, dated 2026-10-17, and yeah the header was doing that classic TAP::H...Beyond(3perl) truncation thing — iconic of it, honestly. go write some tests 💖✨