bashseduniq

remove consecutive multi-line duplicates (with bash)


In a text file, I want to remove duplicates spanning two lines. Meaning in four consecutive lines the first two are the same as the last two. I only want to keep the first (or last) two lines. I want to preserve the order of lines in the file.

Example

Consider a file input.txt where foo\nbar is repeated and baz\nboo is repeated, each in consecutive two-line blocks.

1
foo
bar
foo
bar
2
3
baz
boo
baz
boo
4

desired contents:

1
foo
bar
2
3
baz
boo
4

Things considered: uniq, sed

The same task is fairly simple for removing single line duplicates: uniq input.txt. However, man uniq doesn't suggest that there is an option to get it to work for my use case.

I also had a look at sed, but couldn't get it to work. EDIT: didn't try anything specific as these docs only consider searching and replacing within a two-lines block, not four lines.


Solution

  • If you want to accept a perl solution then:

    perl -0777 -pe 's/(.+\R.+\R)\1/$1/g' file
    
    1
    foo
    bar
    2
    3
    baz
    boo
    4