ent(1)

📦
version 1.2debian · 2 versions
, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble

🦋 ent(1) — the pseudorandom number sequence test 💖

okay bestie, gather round, because we're about to vibe-check your random numbers and honestly? it's giving forensics. this is ent, the pseudorandom number sequence test, and she is thorough. ✨

🌸 NAME 🌸

ent — pseudorandom number sequence test. that's it, that's the girl. she runs tests on streams of bytes and tells you if they're actually random or just pretending. main character energy, no cap.

🎀 SYNOPSIS 🎀

ent [options] [file]

✨ DESCRIPTION ✨

(There's an ENT logo here, iconic, love that for her. 🫶)

ent runs a whole bunch of tests on the stream of bytes in file — or on standard input if you didn't name a file, she's not picky — and then writes the results to standard output. for example:

Entropy = 7.980627 bits per character.

Optimum compression would reduce the size
of this 51768 character file by 0 percent.

Chi square distribution for 51768 samples is 1542.26, and randomly
would exceed this value 0.01 percent of the times.

Arithmetic mean value of data bytes is 125.93 (127.5 = random).
Monte Carlo value for Pi is 3.169834647 (error 0.90 percent).
Serial correlation coefficient is 0.004249 (totally uncorrelated = 0.0).

and here's what all those numbers actually mean, because i know you were gonna ask:

🔮 ENTROPY 🔮

the information density of the file's contents, measured in bits per character. the results up there came from processing a JPEG-compressed image file, and they show the file is extremely dense in information — basically random. so compression is not gonna shrink it, sorry. by contrast, the C source code of the program itself has entropy of about 4.9 bits per character, meaning optimal compression of that file would reduce its size by 38%. [Hamming, pp. 104-108]

🌈 CHI-SQUARE TEST 🌈

the chi-square test is the most commonly used test for randomness of data, and she is extremely sensitive to errors in pseudorandom sequence generators — like, she will notice. the chi-square distribution is calculated for the stream of bytes in the file and expressed as an absolute number plus a percentage that says how frequently a truly random sequence would exceed the calculated value. we read that percentage as how much the sequence is suspected of being non-random:

  • percentage greater than 99% or less than 1% — the sequence is almost certainly not random 💀
  • percentage between 99% and 95%, or between 1% and 5% — the sequence is suspect
  • percentage between 90% and 95%, or between 5% and 10% — the sequence is "almost suspect"

and note: that JPEG file? very dense in information, yes, but far from random as the chi-square test reveals. not the same thing bestie, not even close.

running this test on the output of various pseudorandom sequence generators is genuinely interesting. the low-order 8 bits returned by the standard Unix rand(1) function, for example, yields:

Chi square distribution for 500000 samples is 0.01, and randomly
would exceed this value 99.99 percent of the times.

while an improved generator [Park & Miller] reports:

Chi square distribution for 500000 samples is 212.53, and randomly
would exceed this value 95.00 percent of the times.

so the standard Unix generator (or at least the low-order bytes it hands back) is unacceptably non-random, while the improved generator is much better but still non-random enough to cause concern for demanding applications. now contrast both of those software generators with the chi-square result of a genuine random sequence created by timing radioactive decay events[1]:

Chi square distribution for 32768 samples is 237.05, and randomly
would exceed this value 75.00 percent of the times.

see [Knuth, pp. 35-40] for more on the chi-square test. an interactive chi-square calculator[2] is available at this site. ✨

🧮 ARITHMETIC MEAN 🧮

literally just the sum of all the bytes in the file (bits if you passed the -b option) divided by the file length. if the data is close to random this should land around 127.5 (0.5 for -b option output). if the mean drifts away from that, your values are consistently high or consistently low. simple as.

🥧 MONTE CARLO VALUE FOR PI 🥧

each successive sequence of six bytes gets used as 24 bit X and Y coordinates inside a square. if the distance of that randomly-generated point is less than the radius of a circle inscribed within the square, the six-byte sequence counts as a "hit". the percentage of hits can be used to calculate the value of Pi. for very large streams (this approximation converges very slowly, like girl take a seat) the value will approach the correct value of Pi if the sequence is close to random. a 32768 byte file created by radioactive decay yielded:

Monte Carlo value for Pi is 3.139648438 (error 0.06 percent).

🔗 SERIAL CORRELATION COEFFICIENT 🔗

this one measures how much each byte in the file depends on the previous byte. for random sequences this value (which can be positive or negative) will, of course, be close to zero. a non-random byte stream like a C program will yield a serial correlation coefficient on the order of 0.5. wildly predictable data, like uncompressed bitmaps, will exhibit serial correlation coefficients approaching 1. see [Knuth, pp. 64-65] for more details.

💅 OPTIONS 💅

  • -b — the input is treated as a stream of bits rather than of 8-bit bytes. the statistics reported reflect the properties of the bitstream.
  • -c — print a table of the number of occurrences of each possible byte (or bit, if the -b option is also specified) value, plus the fraction of the overall file made up by that value. printable characters in the ISO-8859-1 (Latin-1) character set are shown along with their decimal byte values. in non-terse output mode, values with zero occurrences are not printed.
  • -f — fold upper case letters to lower case before computing statistics. folding is done based on the ISO-8859-1 (Latin-1) character set, with accented letters correctly processed. she's precise like that. 💖
  • -t — terse mode: output is written in Comma Separated Value (CSV) format, suitable for loading into a spreadsheet and easily read by any programming language. see Terse Mode Output Format below for additional details.
  • -u — print how-to-call information.

📂 FILES 📂

if no file is specified, ent obtains its input from standard input. output is always written to standard output. always. no exceptions, that's the rule. 🎀

🧚 TERSE MODE 🧚

terse mode is selected by specifying the -t option on the command line. terse mode output is written in Comma Separated Value (CSV) format, which can be directly loaded into most spreadsheet programs and is easily read by any programming language. each record in the CSV file begins with a record type field, which identifies the content of the following fields. if the -c option is not specified, the terse mode output will consist of two records, as follows:

0,File-bytes,Entropy,Chi-square,Mean,Monte-Carlo-Pi,Serial-Correlation
1,file_length,entropy,chi_square,mean,Pi_value,correlation

where the italicised values in the type 1 record are the numerical values for the quantities named in the type 0 column title record. if the -b option is specified, the second field of the type 0 record will be File-bits, and the file_length field in the type 1 record will be given in bits instead of bytes. if the -c option is specified, additional records are appended to the terse mode output which contain the character counts:

2,Value,Occurrences,Fraction
3,v,count,fraction
...

if the -b option is specified, only two type 3 records will appear for the two bit values v=0 and v=1. otherwise, 256 type 3 records are included, one for each possible byte value. the second field of a type 3 record indicates how many bytes (or bits) of value v appear in the input, and fraction gives the decimal fraction of the file which has value v (which is equal to the count value of this record divided by the file_length field in the type 1 record).

🙈 BUGS 🙈

heads up bestie: the "optimal compression" shown for the file is computed from the byte- or bit-stream entropy, and thus reflects compressibility based on a reading frame of the chosen width (8-bit bytes, or individual bits if the -b option is specified). algorithms which use a larger reading frame, such as the Lempel-Ziv [Lempel & Ziv] algorithm, may achieve greater compression if the file contains repeated sequences of multiple bytes. so don't come for her about it, she's being honest about her frame. 😭

👑 COPYING 👑

this software is in the public domain. permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, without any conditions or restrictions. this software is provided "as is" without express or implied warranty.

original text and program by John Walker[3] October 20th, 1998.

modifications by Wesley J. Landaker < wjl@icecavern.net ⟨mailto:wjl@icecavern.net⟩ >, released under the same terms as above.

🔥 SEE ALSO 🔥

Introduction to Probability and Statistics[4]

and that's the whole page babes 💜 this version of the ent(1) manual page is dated 22 November 2020 — go forth and chi-square responsibly, bestie. slay. ✨🎀🦋

📦 where this page comes from