Test::...Runner(3pm)

✨ NAME ✨

omg bestie, you're reading the Test::...Runner(3pm) page from the User Contributed Perl Documentation 💅 and it's all about Test::LectroTest::TestRunner — the configurable TAP-compatible engine for running LectroTest property checks. No cap, that's literally the whole vibe.

---

🌸 VERSION 🌸

version 0.5001 — iconic, periodt. 👑

---

🦋 SYNOPSIS 🦋

Bestie, here's the quick little taste test:

use Test::LectroTest::TestRunner;

my @args = trials => 1_000, retries => 20_000;
my $runner = Test::LectroTest::TestRunner->new( @args );

# test a single property and print details upon failure
my $result = $runner->run( $a_single_lectrotest_property );
print $result->details unless $result->success;

# test a suite of properties, w/ Test::Harness::TAP output
my $num_successful = $runner->run_suite( @properties );
print "# All passed!" if $num_successful == @properties;

---

💜 DESCRIPTION 💜

STOP! 🛑 bestie, listen. If you just want to write and run simple tests, go see Test::LectroTest. That's the low-effort slay. But if you really want to learn about the property-checking apparatus or turn its control knobs, then read on. You're in the right place, iykyk. ✨

This module provides Test::LectroTest::TestRunner, a class of objects that tests properties by running repeated random trials. Create a TestRunner, configure it, and then call its run or run_suite methods to test properties individually or in groups. Main character energy fr fr. 💅

---

🧚 METHODS 🧚

The following methods are available.

✨ new(named-params) ✨

my $runner = new Test::LectroTest::TestRunner(
  trials      => 1_000,
  retries     => 20_000,
  scalefn     => sub { $_[0] / 2 + 1 },
  verbose     => 1,
  regressions => "/path/to/regression_suite.txt",
);

Creates a new Test::LectroTest::TestRunner and configures it with the given named parameters, if any. Typically, you need only provide the trials parameter because the other values are reasonable for almost all situations. Here is what each parameter means:

• trials — The number of trials to run against each property checked. The default is 1_000.
• retries — The number of times to allow a property to retry trials (via $tcon->retry) during the entire property check before aborting the check. This is used to prevent infinite looping, should the property retry every attempt.
• scalefn — A subroutine that scales the sizing guidance given to input generators.

The TestRunner starts with an initial guidance of 1 at the beginning of a property check. For each trial (or retry) of the property, the guidance value is incremented. This causes successive trials to be tried using successively more complex inputs. The scalefn subroutine gets to adjust this guidance on the way to the input generators. Typically, you would change the scalefn subroutine if you wanted to change the rate and which inputs grow during the course of the trials.

• verbose — If this parameter is set to true (the default) the TestRunner will use verbose output that includes things like label frequencies and counterexamples. Otherwise, only one-line summaries will be output. Unless you have a good reason to do otherwise, leave this parameter alone because verbose output is almost always what you want.
• record_failures — If this parameter is set to a file's pathname (or a FailureRecorder object), the TestRunner will record property-check failures to the file (or recorder). (This is an easy way to build a regression-testing suite.) If the file cannot be created or written to, this parameter will be ignored. Set this parameter to undef (the default) to turn off recording.
• playback_failures — If this parameter is set to a file's pathname (or a FailureRecorder object), the TestRunner will load previously recorded failures from the file (or recorder) and use them as additional test cases when checking properties. If the file cannot be read, this option will be ignored. Set this parameter to undef (the default) to turn off recording.
• regressions — If this parameter is set to a file's pathname (or a FailureRecorder object), the TestRunner will load failures from and record failures to the file (or recorder). Setting this parameter is a shortcut for, and exactly equivalent to, setting record_failures and <playback_failures> to the same value, which is typically what you want when managing a persistent suite of regression tests.

This is a write-only accessor.

You can also set and get the values of the configuration properties using accessors of the same name. For example:

$runner->trials( 10_000 );

🔥 run(property) 🔥

$results = $runner->run( $a_property );
print $results->summary, "\n";
if ($results->success) {
    # celebrate!
}

Checks whether the given property holds by running repeated random trials. The result is a Test::LectroTest::TestRunner::results object, which you can query for fine-grained information about the outcome of the check.

The run method takes an optional second argument which gives the test number. If it is not provided (usually the case), the next number available from the TestRunner's internal counter is used.

$results = $runner->run( $third_property, 3 );

Additionally, if the TestRunner's playback_failures parameter is defined, this method will play back any relevant failure cases from the given playback file (or FailureRecorder).

Additionally, if the TestRunner's record_failures parameter is defined, this method will record any new failures to the given file (or FailureRecorder).

👑 run_suite(properties...) 👑

my $num_successful = $runner->run_suite( @properties );
if ($num_successful == @properties) {
    # celebrate most jubilantly!
}

Checks a suite of properties, sending the results of each property checked to STDOUT in a form that is compatible with Test::Harness::TAP. For example:

1..5
ok 1 - Property->new disallows use of 'tcon' in bindings
ok 2 - magic Property syntax disallows use of 'tcon' in bindings
ok 3 - exceptions are caught and reported as failures
ok 4 - pre-flight check catches new w/ no args
ok 5 - pre-flight check catches unbalanced arguments list

By default, labeling statistics and counterexamples (if any) are included in the output if the TestRunner's verbose property is true. You may override the default by passing the verbose named parameter after all of the properties in the argument list:

my $num_successes = $runner->run_suite( @properties,
                                        verbose => 1 );
my $num_failed = @properties - $num_successes;

🫶 HELPER OBJECTS 🫶

So there's two kinds of objects that TestRunner pulls in as helpers, and bestie, listen — neither one is meant to be made by you. Nope. A TestRunner just spawns them for you when it needs them, like a lil fairy godmother but for property checks. ✨

They get their own lil subsections below, so let's go.

---

🌸 Test::LectroTest::TestRunner::results 🌸

my $results = $runner->run( $a_property );
print "Property name: ", $results->name, ": ";
print $results->success ? "Winner!" : "Loser!";

This is the object you get handed back from run. It's basically the receipt for your property check — it's holding every single piece of info about how things went down, and it's got these methods for you:

• success — Boolean moment. True if the property checked out fine and dandy; false if it did not. 💀
• summary — Gives you a one-line summary of how the property check turned out. And no, it does not end with a newline. Example:

ok 1 - Property->new disallows use of 'tcon' in bindings

• details — Returns all the relevant tea about the property-check outcome, laid out as a series of lines. The last line does get a newline, unlike summary. The details are identical to the summary (minus that terminating newline) unless label frequencies are present or a counterexample is present — in those cases the details get these extras and the summary does not. Example:

1..1
not ok 1 - 'my_sqrt meets defn of sqrt' falsified in 1 attempts
# Counterexample:
# $x = '0.546384454460178';

• name — Returns the name of the property these results are about. 🏷️
• number — The number assigned to the property that got checked.
• counterexample — Returns the counterexample that "broke" the code being tested, if there is one. If there isn't, you get an empty string. And if any notes got attached to the failing trial, they'll be included too.
• labels — Label counts, bestie. If any labels got applied to trials during the property check, this value is a reference to a hash that maps each combination of labels to the count of trials that had that particular combo. Otherwise it's undef, no thoughts head empty.

Heads up tho: each trial is counted only once — for the most-specific combination of labels applied to it. For example, consider this labeling logic:

Property {
  ##[ x <- Int ]##
  $tcon->label("negative") if $x < 0;
  $tcon->label("odd")      if $x % 2;
  1;
}, name => "negative/odd labeling example";

So for a particular trial: if x was 2 (positive and even), that trial gets no labels at all. If x was 3 (positive and odd), it gets labeled "odd". If x was -2 (negative and even), it gets labeled "negative". And if x was -3 (negative and odd), it gets labeled "negative & odd". Slay, math. 💅

• label_frequencies — Returns a string with a line-by-line accounting of the labels applied during the series of trials:

print $results->label_frequencies;

The output for that looks like:

25% negative
25% negative & odd
25% odd

If no labels got applied, you just get an empty string back. 🤷

• exception — Returns the text of the exception or error that caused the series of trials to be aborted, if the trials got aborted because LectroTest caught an exception or error. Otherwise: empty string.
• attempts — Returns the count of trials performed. 🔢
• incomplete — If the series of trials got halted before it finished (like when the retry count ran out, rip), this method returns the reason why. Otherwise it returns an empty string.

Note that a series of trials counts as complete if a counterexample was found. So finding the bug = finished. Iconic behavior honestly.

---

🦋 Test::LectroTest::testcontroller 🦋

During a live property-check trial, the variable $tcon is available to your Properties. It lets you label the current trial, or ask for it to be re-tried with new inputs. It's giving remote control for your trial.

Here's what's available:

• retry

Property {
  ##[ x <- Int ]##
  return $tcon->retry if $x == 0;
}, ... ;

Stops the current trial and tells the TestRunner to re-try it with new inputs. Usually you use this to reject a particular case of inputs that doesn't make for a good or valid test. It's not required, but you'll probably want to call $tcon->retry as part of a return statement so the rest of your property's logic doesn't keep running — because those results would get thrown out anyway if it ran to completion.

The return value of $tcon->retry is itself meaningless. It's the side-effect of calling it that makes the current trial get tossed and re-tried. No cap.

• label(string)

Property {
  ##[ x <- Int ]##
  $tcon->label("negative") if $x < 0;
  $tcon->label("odd")      if $x % 2;
}, ... ;

Applies a label to the current trial. At the end of the trial, all the labels get gathered together and the trial gets dropped into a bucket wearing the combined label. See the discussion of "labels" above for more. 🪣

• trivial

Property {
  ##[ x <- Int ]##
  $tcon->trivial if $x == 0;
}, ... ;

Applies the label "trivial" to the current trial. It's literally identical to calling label with "trivial" as the argument. Same thing, cuter name. ✨

• note(string...)

Property {
  ##[ s <- String( charset=>"A-Za-z0-9" ) ]##
  my $s_enc     = encode($s);
  my $s_enc_dec = decode($s_enc);
  $tcon->note("s_enc     = $s_enc",
              "s_enc_dec = $s_enc_dec");
  $s eq $s_enc_dec;
}, name => "decode is encode's inverse" ;

Adds a note (or notes, plural, we love a variadic) to the current trial. If the trial fails, those notes get emitted as part of the counterexample. Like so:

1..1
not ok 1 - property 'decode is encode's inverse' \
    falsified in 68 attempts
#     Counterexample:
#     $s = "0";
#     Notes:
#     $s_enc     = "";
#     $s_enc_dec = "";

Notes help you debug your code when something goes sideways. Think of them as lil debugging hints to your future self. For example, you can use notes to record the output of each stage of a multi-stage test — that way if the test fails, you can see what happened at every stage without having to plug the counterexample into your code under a debugger. Self-care via logging. 🫶

If you want to include complicated values or data structures in your notes, peep the "dump" method right below, which might be a better fit.

• dump(value, name)

Property {
  ##[ s <- String ]##
  my $s_enc     = encode($s);
  my $s_enc_dec = decode($s_enc);
  $tcon->dump($s_enc, "s_enc");
  $tcon->dump($s_enc_dec, "s_enc_dec");
  $s eq $s_enc_dec;
}, name => "decode is encode's inverse" ;

Adds a note to the current trial where the given value gets dumped. The value goes through Data::Dumper, so it can be complex and full of weird control characters and whatever else. If you supply a name, that name gets used to name the dumped value. Returns value as its result.

If the trial fails, the note (and any other notes) get emitted as part of the counterexample.

See "note" above for more. 🌈

---

👀 SEE ALSO 👀

• Test::LectroTest::Property — explains in detail what you can actually put inside your property specifications.
• Test::LectroTest::RegressionTesting — explains how to test for regressions and corner cases using LectroTest.
• Test::Harness:TAP — documents the Test Anything Protocol, Perl's simple text-based interface between testing modules like Test::LectroTest and the test harness Test::Harness.

---

👑 AUTHOR 👑

Tom Moertel (tom@moertel.com)

---

🌟 INSPIRATION 🌟

The LectroTest project was inspired by Haskell's QuickCheck module by Koen Claessen and John Hughes: http://www.cs.chalmers.se/~rjmh/QuickCheck/.

---

💖 COPYRIGHT and LICENSE 💖

Copyright (c) 2004-13 by Thomas G Moertel. All rights reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

---

And that's the whole page, bestie — this was perl v5.34.0, dated 2022-06-21, from the User Contributed Perl Documentation for Test::...Runner(3pm). Thanks for reading all the way to the end, you're literally the main character. Periodt. 💅✨