man-pages/man1p/uniq.1p

.\" Copyright (c) 2001-2003 The Open Group, All Rights Reserved 
.TH "UNIQ" P 2003 "IEEE/The Open Group" "POSIX Programmer's Manual"
.\" uniq 
.SH NAME
uniq \- report or filter out repeated lines in a file
.SH SYNOPSIS
.LP
\fBuniq\fP \fB[\fP\fB-c|-d|-u\fP\fB][\fP\fB-f\fP \fIfields\fP\fB][\fP\fB-s\fP
\fIchar\fP\fB][\fP\fIinput_file\fP \fB[\fP\fIoutput_file\fP\fB]]\fP
.SH DESCRIPTION
.LP
The \fIuniq\fP utility shall read an input file comparing adjacent
lines, and write one copy of each input line on the output.
The second and succeeding copies of repeated adjacent input lines
shall not be written.
.LP
Repeated lines in the input shall not be detected if they are not
adjacent.
.SH OPTIONS
.LP
The \fIuniq\fP utility shall conform to the Base Definitions volume
of IEEE\ Std\ 1003.1-2001, Section 12.2, Utility Syntax Guidelines.
.LP
The following options shall be supported:
.TP 7
\fB-c\fP
Precede each output line with a count of the number of times the line
occurred in the input.
.TP 7
\fB-d\fP
Suppress the writing of lines that are not repeated in the input.
.TP 7
\fB-f\ \fP \fIfields\fP
Ignore the first \fIfields\fP fields on each input line when doing
comparisons, where \fIfields\fP is a positive decimal
integer. A field is the maximal string matched by the basic regular
expression: 
.sp
.RS
.nf

\fB[[:blank:]]*[^[:blank:]]*
\fP
.fi
.RE
.LP
If the \fIfields\fP option-argument specifies more fields than appear
on an input line, a null string shall be used for
comparison.
.TP 7
\fB-s\ \fP \fIchars\fP
Ignore the first \fIchars\fP characters when doing comparisons, where
\fIchars\fP shall be a positive decimal integer. If
specified in conjunction with the \fB-f\fP option, the first \fIchars\fP
characters after the first \fIfields\fP fields shall be
ignored. If the \fIchars\fP option-argument specifies more characters
than remain on an input line, a null string shall be used
for comparison.
.TP 7
\fB-u\fP
Suppress the writing of lines that are repeated in the input.
.sp
.SH OPERANDS
.LP
The following operands shall be supported:
.TP 7
\fIinput_file\fP
A pathname of the input file. If the \fIinput_file\fP operand is not
specified, or if the \fIinput_file\fP is \fB'-'\fP ,
the standard input shall be used.
.TP 7
\fIoutput_file\fP
A pathname of the output file. If the \fIoutput_file\fP operand is
not specified, the standard output shall be used. The
results are unspecified if the file named by \fIoutput_file\fP is
the file named by \fIinput_file\fP.
.sp
.SH STDIN
.LP
The standard input shall be used only if no \fIinput_file\fP operand
is specified or if \fIinput_file\fP is \fB'-'\fP . See
the INPUT FILES section.
.SH INPUT FILES
.LP
The input file shall be a text file.
.SH ENVIRONMENT VARIABLES
.LP
The following environment variables shall affect the execution of
\fIuniq\fP:
.TP 7
\fILANG\fP
Provide a default value for the internationalization variables that
are unset or null. (See the Base Definitions volume of
IEEE\ Std\ 1003.1-2001, Section 8.2, Internationalization Variables
for
the precedence of internationalization variables used to determine
the values of locale categories.)
.TP 7
\fILC_ALL\fP
If set to a non-empty string value, override the values of all the
other internationalization variables.
.TP 7
\fILC_COLLATE\fP
.sp
Determine the locale for ordering rules.
.TP 7
\fILC_CTYPE\fP
Determine the locale for the interpretation of sequences of bytes
of text data as characters (for example, single-byte as
opposed to multi-byte characters in arguments and input files) and
which characters constitute a <blank> in the current
locale.
.TP 7
\fILC_MESSAGES\fP
Determine the locale that should be used to affect the format and
contents of diagnostic messages written to standard
error.
.TP 7
\fINLSPATH\fP
Determine the location of message catalogs for the processing of \fILC_MESSAGES
\&.\fP 
.sp
.SH ASYNCHRONOUS EVENTS
.LP
Default.
.SH STDOUT
.LP
The standard output shall be used only if no \fIoutput_file\fP operand
is specified. See the OUTPUT FILES section.
.SH STDERR
.LP
The standard error shall be used only for diagnostic messages.
.SH OUTPUT FILES
.LP
If the \fB-c\fP option is specified, the output file shall be empty
or each line shall be of the form:
.sp
.RS
.nf

\fB"%d %s", <\fP\fInumber of duplicates\fP\fB>, <\fP\fIline\fP\fB>
\fP
.fi
.RE
.LP
otherwise, the output file shall be empty or each line shall be of
the form:
.sp
.RS
.nf

\fB"%s", <\fP\fIline\fP\fB>
\fP
.fi
.RE
.SH EXTENDED DESCRIPTION
.LP
None.
.SH EXIT STATUS
.LP
The following exit values shall be returned:
.TP 7
\ 0
The utility executed successfully.
.TP 7
>0
An error occurred.
.sp
.SH CONSEQUENCES OF ERRORS
.LP
Default.
.LP
\fIThe following sections are informative.\fP
.SH APPLICATION USAGE
.LP
The \fIsort\fP utility can be used to cause repeated lines to be adjacent
in the input
file.
.SH EXAMPLES
.LP
The following input file data (but flushed left) was used for a test
series on \fIuniq\fP:
.sp
.RS
.nf

\fB#01 foo0 bar0 foo1 bar1
#02 bar0 foo1 bar1 foo1
#03 foo0 bar0 foo1 bar1
#04
#05 foo0 bar0 foo1 bar1
#06 foo0 bar0 foo1 bar1
#07 bar0 foo1 bar1 foo0
\fP
.fi
.RE
.LP
What follows is a series of test invocations of the \fIuniq\fP utility
that use a mixture of \fIuniq\fP options against the
input file data. These tests verify the meaning of \fIadjacent\fP.
The \fIuniq\fP utility views the input data as a sequence of
strings delimited by \fB'\\n'\fP . Accordingly, for the \fIfields\fPth
member of the sequence, \fIuniq\fP interprets unique or
repeated adjacent lines strictly relative to the \fIfields\fP+1th
member.
.IP " 1." 4
This first example tests the line counting option, comparing each
line of the input file data starting from the second
field:
.sp
.RS
.nf

\fBuniq -c -f 1 uniq_0I.t
    1 #01 foo0 bar0 foo1 bar1
    1 #02 bar0 foo1 bar1 foo0
    1 #03 foo0 bar0 foo1 bar1
    1 #04
    2 #05 foo0 bar0 foo1 bar1
    1 #07 bar0 foo1 bar1 foo0
\fP
.fi
.RE
.LP
The number \fB'2'\fP , prefixing the fifth line of output, signifies
that the \fIuniq\fP utility detected a pair of repeated
lines. Given the input data, this can only be true when \fIuniq\fP
is run using the \fB-f\ 1\fP option (which shall cause
\fIuniq\fP to ignore the first field on each input line).
.LP
.IP " 2." 4
The second example tests the option to suppress unique lines, comparing
each line of the input file data starting from the
second field:
.sp
.RS
.nf

\fBuniq -d -f 1 uniq_0I.t
#05 foo0 bar0 foo1 bar1
\fP
.fi
.RE
.LP
.IP " 3." 4
This test suppresses repeated lines, comparing each line of the input
file data starting from the second field:
.sp
.RS
.nf

\fBuniq -u -f 1 uniq_0I.t
#01 foo0 bar0 foo1 bar1
#02 bar0 foo1 bar1 foo1
#03 foo0 bar0 foo1 bar1
#04
#07 bar0 foo1 bar1 foo0
\fP
.fi
.RE
.LP
.IP " 4." 4
This suppresses unique lines, comparing each line of the input file
data starting from the third character:
.sp
.RS
.nf

\fBuniq -d -s 2 uniq_0I.t
\fP
.fi
.RE
.LP
In the last example, the \fIuniq\fP utility found no input matching
the above criteria.
.LP
.SH RATIONALE
.LP
Some historical implementations have limited lines to be 1080 bytes
in length, which does not meet the implied {LINE_MAX}
limit.
.SH FUTURE DIRECTIONS
.LP
None.
.SH SEE ALSO
.LP
\fIcomm\fP , \fIsort\fP
.SH COPYRIGHT
Portions of this text are reprinted and reproduced in electronic form
from IEEE Std 1003.1, 2003 Edition, Standard for Information Technology
-- Portable Operating System Interface (POSIX), The Open Group Base
Specifications Issue 6, Copyright (C) 2001-2003 by the Institute of
Electrical and Electronics Engineers, Inc and The Open Group. In the
event of any discrepancy between this version and the original IEEE and
The Open Group Standard, the original IEEE and The Open Group Standard
is the referee document. The original Standard can be obtained online at
http://www.opengroup.org/unix/online.html .