SORT(1POSIX)

okay bestie, welcome to SORT(1POSIX), the POSIX Programmer's Manual entry that's about to sort, merge, or sequence-check your text files like the main character she is. ๐Ÿ’…โœจ

โœจ PROLOG โœจ

Real talk: this manual page is part of the POSIX Programmer's Manual. The Linux implementation of this interface may differ โ€” so if you're on Linux, consult the corresponding Linux manual page for the details of Linux behavior โ€” or the interface may not be implemented on Linux at all. ๐Ÿ’€

๐ŸŒธ NAME ๐ŸŒธ

sort โ€” sort, merge, or sequence check text files

๐Ÿฆ‹ SYNOPSIS ๐Ÿฆ‹

sort [-m] [-o output] [-bdfinru] [-t char] [-k keydef]... [file...]

sort [-c|-C] [-bdfinru] [-t char] [-k keydef] [file]

๐Ÿ’– DESCRIPTION ๐Ÿ’–

okay so sort shall perform one of these three functions, and she's very serious about it:

  1. Sort lines of all the named files together and write the result to the specified output.
  2. Merge lines of all the named (presorted) files together and write the result to the specified output.
  3. Check that a single input file is correctly presorted.

Comparisons shall be based on one or more sort keys extracted from each line of input โ€” or, if no sort keys are specified, the entire line up to, but not including, the terminating <newline> โ€” and shall be performed using the collating sequence of the current locale. If this collating sequence does not have a total ordering of all characters (see the Base Definitions volume of POSIX.1โ€2017, Section 7.3.2, LC_COLLATE), any lines of input that collate equally should be further compared byte-by-byte using the collating sequence for the POSIX locale. iykyk. ๐ŸŒˆ

๐Ÿ‘‘ OPTIONS ๐Ÿ‘‘

The sort utility shall conform to the Base Definitions volume of POSIX.1โ€2017, Section 12.2, Utility Syntax Guidelines, except for Guideline 9, and the -k keydef option should follow the -b, -d, -f, -i, -n, and -r options. In addition, '+' may be recognized as an option delimiter as well as '-'. periodt.

The following options shall be supported:

  • -c โ€” Check that the single input file is ordered as specified by the arguments and the collating sequence of the current locale. Output shall not be sent to standard output. The exit code shall indicate whether or not disorder was detected or an error occurred. If disorder (or, with -u, a duplicate key) is detected, a warning message shall be sent to standard error indicating where the disorder or duplicate key was found. ๐Ÿ’€
  • -C โ€” Same as -c, except that a warning message shall not be sent to standard error if disorder or, with -u, a duplicate key is detected. Lowkey the quiet sibling.
  • -m โ€” Merge only; the input file shall be assumed to be already sorted. No cap.
  • -o output โ€” Specify the name of an output file to be used instead of the standard output. This file can be the same as one of the input files. Iconic.
  • -u โ€” Unique: suppress all but one in each set of lines having equal keys. If used with the -c option, check that there are no lines with duplicate keys, in addition to checking that the input file is sorted. Slay.

The following options shall override the default ordering rules. When ordering options appear independent of any key field specifications, the requested field ordering rules shall be applied globally to all sort keys. When attached to a specific key (see -k), the specified ordering options shall override all global ordering options for that key. The way that this is giving "global vs local" energy. โœจ

  • -d โ€” Specify that only <blank> characters and alphanumeric characters, according to the current setting of LC_CTYPE, shall be significant in comparisons. The behavior is undefined for a sort key to which -i or -n also applies. ๐Ÿšจ undefined behavior, bestie, be careful.
  • -f โ€” Consider all lowercase characters that have uppercase equivalents, according to the current setting of LC_CTYPE, to be the uppercase equivalent for the purposes of comparison. Case-insensitive-ish, fr fr.
  • -i โ€” Ignore all characters that are non-printable, according to the current setting of LC_CTYPE. The behavior is undefined for a sort key for which -n also applies. Another undefined behavior warning, so don't mix it with -n unless you're ready. ๐Ÿ˜ญ
  • -n โ€” Restrict the sort key to an initial numeric string, consisting of optional <blank> characters, optional <hyphen-minus> character, and zero or more digits with an optional radix character and thousands separators (as defined in the current locale), which shall be sorted by arithmetic value. An empty digit string shall be treated as zero. Leading zeros and signs on zeros shall not affect ordering. Math girlie behavior. ๐Ÿ”ฅ
  • -r โ€” Reverse the sense of comparisons. Flip it and reverse it. ๐Ÿ’…

The treatment of field separators can be altered using the options:

  • -b โ€” Ignore leading <blank> characters when determining the starting and ending positions of a restricted sort key. If the -b option is specified before the first -k option, it shall be applied to all -k options. Otherwise, the -b option can be attached independently to each -k field_start or field_end option-argument (see below). So it's giving "global unless attached" vibes. ๐Ÿฆ‹
  • -t char โ€” Use char as the field separator character; char shall not be considered to be part of a field (although it can be included in a sort key). Each occurrence of char shall be significant (for example, <char><char> delimits an empty field). If -t is not specified, <blank> characters shall be used as default field separators; each maximal non-empty sequence of <blank> characters that follows a non-<blank> shall be a field separator. ๐ŸŽ€

Sort keys can be specified using the options:

  • -k keydef โ€” The keydef argument is a restricted sort key field definition. The format of this definition is:
field_start[type][,field_end[type]]

where field_start and field_end define a key field restricted to a portion of the line (see the EXTENDED DESCRIPTION section), and type is one or more modifiers from the list of characters 'b', 'd', 'f', 'i', 'n', 'r'. The 'b' modifier shall behave like the -b option, but shall apply only to the field_start or field_end to which it is attached. The other modifiers shall behave like the corresponding options, but shall apply only to the key field to which they are attached; they shall have this effect if specified with field_start, field_end, or both. If any modifier is attached to a field_start or to a field_end, no option shall apply to either. Implementations shall support at least nine occurrences of the -k option, which shall be significant in command line order. If no -k option is specified, a default sort key of the entire line shall be used. Nine! At least! She ate. ๐Ÿ‘‘

When there are multiple key fields, later keys shall be compared only after all earlier keys compare equal. Except when the -u option is specified, lines that otherwise compare equal shall be ordered as if none of the options -d, -f, -i, -n, or -k were present (but with -r still in effect, if it was specified) and with all bytes in the lines significant to the comparison. The order in which lines that still compare equal are written is unspecified. Unspecified? Not me loving chaos. ๐Ÿ™ˆ

๐Ÿ“ OPERANDS ๐Ÿ“

The following operand shall be supported:

  • file โ€” A pathname of a file to be sorted, merged, or checked. If no file operands are specified, or if a file operand is '-', the standard input shall be used. If sort encounters an error when opening or reading a file operand, it may exit without writing any output to standard output or processing later operands. So if a file acts up, she might just dip. ๐Ÿ’€

๐Ÿงš STDIN ๐Ÿงš

The standard input shall be used only if no file operands are specified, or if a file operand is '-'. See the INPUT FILES section. ๐Ÿซถ

๐ŸŒธ INPUT FILES ๐ŸŒธ

The input files shall be text files, except that the sort utility shall add a <newline> to the end of a file ending with an incomplete last line. She's fixing your last line, you're welcome. ๐Ÿ’–

๐Ÿ’œ ENVIRONMENT VARIABLES ๐Ÿ’œ

The following environment variables shall affect the execution of sort:

  • LANG โ€” Provide a default value for the internationalization variables that are unset or null. (See the Base Definitions volume of POSIX.1โ€2017, Section 8.2, Internationalization Variables for the precedence of internationalization variables used to determine the values of locale categories.)
  • LC_ALL โ€” If set to a non-empty string value, override the values of all the other internationalization variables.
  • LC_COLLATE โ€” Determine the locale for ordering rules.
  • LC_CTYPE โ€” Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments and input files) and the behavior of character classification for the -b, -d, -f, -i, and -n options.
  • LC_MESSAGES โ€” Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error.
  • LC_NUMERIC โ€” Determine the locale for the definition of the radix character and thousands separator for the -n option.
  • NLSPATH โ€” Determine the location of message catalogs for the processing of LC_MESSAGES.

โญ ASYNCHRONOUS EVENTS โญ

Default. Nothing dramatic, bestie. ๐Ÿ’…

๐ŸŒˆ STDOUT ๐ŸŒˆ

Unless the -o or -c options are in effect, the standard output shall contain the sorted input. That's the whole vibe. โœจ

๐ŸŽ€ STDERR ๐ŸŽ€

The standard error shall be used for diagnostic messages. When -c is specified, if disorder is detected (or if -u is also specified and a duplicate key is detected), a message shall be written to the standard error which identifies the input line at which disorder (or a duplicate key) was detected. A warning message about correcting an incomplete last line of an input file may be generated, but need not affect the final exit status. So she might warn you, but she's not gonna tank the exit status over it. ๐Ÿซถ

๐Ÿ’… OUTPUT FILES ๐Ÿ’…

If the -o option is in effect, the sorted input shall be written to the file output. Slay. โœจ

โœจ EXTENDED DESCRIPTION โœจ

okay bestie this is the part where we learn how key fields actually get carved out of a line, and it's giving geometry class but make it useful ๐Ÿ’…

So this notation:

-k field_start[type][,field_end[type]]

defines a key field that starts at field_start and ends at field_end inclusive โ€” unless field_start falls past the end of the line, or lands after field_end, in which case congrats, your key field is empty ๐Ÿซ . If field_end is justโ€ฆ not there? Then it means the last character of the line. Periodt.

A field is a maximal sequence of non-separating characters and, when -t isn't in the picture, any field separator that comes before it.

The field_start chunk of the keydef option-argument looks like:

field_number[.first_character]

Fields and the characters inside them are numbered starting at 1. The field_number and first_character pieces get read as positive decimal integers, and they say which character is the first one used in the sort key. Skip .first_character entirely and it refers to the first character of the field. Easy.

The field_end chunk looks like:

field_number[.last_character]

field_number works exactly like it did up in field_start. The last_character piece is read as a non-negative decimal integer and says which character is the last one used in the sort key. If last_character evaluates to zero, or .last_character is omitted, it refers to the last character of the field named by field_number.

And if -b or the b type modifier is in effect, characters within a field get counted from the first non-<blank> in that field. (This applies to first_character and last_character separately, so don't go assuming one applies to the other ๐Ÿ™ˆ.)


๐Ÿšช EXIT STATUS ๐Ÿšช

Here's what you get back:

  • 0 โ€” all input files were output successfully, or you used -c and the input file was correctly sorted. slay.
  • 1 โ€” under -c, the file was not ordered the way you specified, or if you used -c and -u together, two input lines were found with equal keys.
  • >1 โ€” an error occurred. ๐Ÿ’€

๐Ÿ’ฅ CONSEQUENCES OF ERRORS ๐Ÿ’ฅ

The default requirements apply, except that if sort hits an error while opening or reading a file operand, it may exit without writing any output to standard output or processing later operands. So like, no promises about what made it out the door bestie.

The following sections are informative. (Read: helpful tea, not hard law. ๐Ÿต)

๐ŸŒธ APPLICATION USAGE ๐ŸŒธ

okay so the default value for -t is <blank>, and it has genuinely different properties from, say, -t"<space>". Watch this. If a line contains:

<space><space>foo

then here's how it gets treated with default separation vs. specifically picking a <space>:

  • Field 1 โ€” default: <space><space>foo; with -t "<space>": empty
  • Field 2 โ€” default: empty; with -t "<space>": empty
  • Field 3 โ€” default: empty; with -t "<space>": foo

The leading field separator is itself included in a field when -t isn't used. For example, this command returns an exit status of zero, meaning the input was already sorted:

sort -c -k 2 <<eof
y<tab>b
x<space>a
eof

(that's assuming a <tab> precedes the <space> in the current collating sequence, iykyk). But the field separator is not included in a field when you explicitly set it via -t. This is historical practice, and it's what lets you do stuff like:

sort -t "|" -k 2n <<eof
Atlanta|425022|Georgia
Birmingham|284413|Alabama
Columbia|100385|South Carolina
eof

where that second field can be correctly sorted numerically without the non-numeric field separator dragging it down. iconic behavior honestly ๐Ÿ‘‘

The wording up in the OPTIONS section makes it clear that -b, -d, -f, -i, -n, and -r have to come before the first sort key if you want them to apply to all the specified keys. The way it's described in this volume of POSIX.1โ€2017 matches historical practice, not historical documentation. And the results are unspecified if you go specifying these options after a -k option. so maybe don't ๐Ÿซถ

The -f option might not work as expected in locales where there isn't a one-to-one mapping between an uppercase and a lowercase letter. just a heads up.

When you're using sort to process pathnames, it's recommended that LC_ALL, or at least LC_CTYPE and LC_COLLATE, are set to POSIX or C in the environment โ€” because pathnames can contain byte sequences that don't form valid characters in some locales, in which case the utility's behavior would be undefined ๐Ÿ˜ญ. In the POSIX locale every byte is a valid single-byte character, so this whole problem justโ€ฆ doesn't happen. chef's kiss.

If the collating sequence of the current locale doesn't have a total ordering of all characters, that can affect how sort behaves in these ways:

  • Since sort -u suppresses lines with duplicate keys, it suppresses lines that collate equally but are not identical.
  • The output of sort (without -u) can contain identical lines that are not adjacent, if it doesn't implement the recommended further byte-by-byte comparison of lines that collate equally. This affects using sort with comm and uniq โ€” go peek at the APPLICATION USAGE for those utilities.

๐ŸŒˆ EXAMPLES ๐ŸŒˆ

  1. This one sorts the contents of infile with the second field as the sort key:
sort -k 2,2 infile
  1. This one sorts, in reverse order, the contents of infile1 and infile2, drops the output into outfile, and uses the second character of the second field as the sort key (assuming the first character of the second field is the field separator):
sort -r -o outfile -k 2.2,2.2 infile1 infile2
  1. This one sorts the contents of infile1 and infile2 using the second non-<blank> of the second field as the sort key:
sort -k 2.2b,2.2b infile1 infile2
  1. This one prints the System V password file (user database) sorted by the numeric user ID โ€” that's the third <colon>-separated field:
sort -t : -k 3,3n /etc/passwd
  1. And this one prints the lines of the already sorted file infile, suppressing all but one occurrence of lines having the same third field:
sort -um -k 3.1,3.0 infile

โœจ RATIONALE โœจ

Okay bestie, story time ๐Ÿ“– โ€” some old historical docs were like โ€œoh, -um with one input file keeps the first in each set of lines with equal keys.โ€ But that was literally just an implementation artifact, not a standardized thing. So we don't do that here. ๐Ÿ’…

The -z option? Omitted, bestie. It's not standard practice on most systems, and it's inconsistent with using sort to sort several files individually and then merge them together. The historical text about -z seemed to require implementations to figure out the proper buffer length during the sort phase of operation, but not during the merge. So yeah, it got left out. ๐Ÿšซ

The -y option was omitted because of non-portability. The -M option, which showed up in System V, was omitted because of non-portability in international usage. Not portable = not invited to the party. ๐Ÿ™…โ™€๏ธ

There's an undocumented -T option in some implementations. It's used to specify a directory for intermediate files. Implementations are encouraged to support the TMPDIR environment variable instead of adding an option to support this functionality. So if you see -T out in the wild, that's what it's doing. ๐Ÿ‘€

The -k option was added to satisfy two objections. First: the zero-based counting used by sort is not consistent with other utility conventions. Second: it did not meet syntax guideline requirements. So -k swooped in to fix the vibe. ๐Ÿฆธโ™€๏ธ

Historical documentation says setting -n implies -b. But the description of -n already states that optional leading <blank>s are tolerated in doing the comparison. If -b is enabled, rather than implied, by -n, this has unusual side-effects. When a character offset is used in a column of numbers (for example, to sort modulo 100), that offset is measured relative to the most significant digit, not to the column. Based upon a recommendation from the author of the original sort utility, the -b implication has been omitted from this volume of POSIX.1โ€2017, and an application wishing to achieve the previously mentioned side-effects has to code the -b flag explicitly. So no sneaky implications โ€” be explicit, bestie. ๐Ÿ“

Earlier versions of this standard allowed the -o option to appear after operands. Historical practice allowed all options to be interspersed with operands. This version of the standard allows implementations to accept options after operands but conforming applications should not use this form. So don't do it if you wanna be conforming. ๐Ÿšฉ

Earlier versions of this standard also allowed the -number and +number options. These options are no longer specified by POSIX.1โ€2008 but may be present in some implementations. So if you see them, they're ghosts of POSIX past. ๐Ÿ‘ป

Historical implementations produced a message on standard error when -c was specified and disorder was detected, and when -c and -u were specified and a duplicate key was detected. An earlier version of this standard contained wording that did not make it clear that this message was allowed and some implementations removed this message to be sure that they conformed to the standard's requirements. Confronted with this difference in behavior, interactive users that wanted to be sure that they got visual feedback instead of just exit code 1 could have used a command like:

sort -c file || echo disorder

whether or not the sort utility provided a message in this case. But, it was not easy for a user to find where the disorder or duplicate key occurred on implementations that do not produce a message, especially when some parts of the input line were not part of the key and when one or more of the -b, -d, -f, -i, -n, or -r options or keydef type modifiers were in use. POSIX.1โ€2008 requires a message to be produced in this case. POSIX.1โ€2008 also contains the -C option giving users the ability to choose either behavior. So you get to pick your fighter. ๐ŸฅŠ

When a disorder or duplicate is found when the -c option is specified, some implementations print a message containing the first line that is out of order or contains a duplicate key; others print a message specifying the line number of the offending line. This standard allows either type of message. So both are valid, no cap. โœ…

Implementations are encouraged to perform the recommended further byte-by-byte comparison of lines that collate equally, even though this may affect efficiency. The impact on efficiency can be mitigated by only performing the additional comparison if the current locale's collating sequence does not have a total ordering of all characters (if the implementation provides a way to query this) or by only performing the additional comparison if the locale name associated with the LC_COLLATE category has an '@' modifier in the name (since locales without an '@' modifier should have a total ordering of all characters โ€” see the Base Definitions volume of POSIX.1โ€2017, Section 7.3.2, LC_COLLATE). Note that if the implementation provides a stable sort option as an extension (usually -s), the additional comparison should not be performed when this option has been specified. Efficiency hack: only do the extra compare when needed. ๐Ÿ’ก


๐Ÿ”ฎ FUTURE DIRECTIONS ๐Ÿ”ฎ

A future version of this standard may require that if the collating sequence of the current locale does not have a total ordering of all characters, any lines of input that collate equally when comparing them as whole lines are further compared byte-by-byte using the collating sequence for the POSIX locale. So like, maybe future POSIX will make that mandatory. Manifesting it. โœจ


๐Ÿ‘€ SEE ALSO ๐Ÿ‘€

  • comm, join, uniq
  • The Base Definitions volume of POSIX.1โ€2017, Section 7.3.2, LC_COLLATE, Chapter 8, Environment Variables, Section 12.2, Utility Syntax Guidelines
  • The System Interfaces volume of POSIX.1โ€2017, toupper()

Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1-2017, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, Copyright (C) 2018 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. In the event of any discrepancy between this version and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Standard is the referee document. The original Standard can be obtained online at http://www.opengroup.org/unix/online.html . Any typographical or formatting errors that appear in this page are most likely to have been introduced during the conversion of the source files to man page format. To report such errors, see https://www.kernel.org/doc/man-pages/reporting_bugs.html .


That's the whole vibe, bestie ๐Ÿ’– โ€” this page is from IEEE/The Open Group, 2017, SORT(1POSIX). Periodt. ๐Ÿ‘‘

๐Ÿ“ฆ where this page comes from