bashsortinguniq

How to remove duplicate lines in a file?

I understand that the general approach is to use something like

$ sort file1.txt | uniq > file2.txt

But I was wondering if there was a way to do this without needing separate source and destination files, even if it means it can't be a one-liner.

Solution

  • Simply use the -o and -u options of sort:

    sort -o file -u file

    You don't need even to use a pipe for another command, such as uniq.