In a text file, I want to remove duplicates spanning two lines. Meaning in four consecutive lines the first two are the same as the last two. I only want to keep the first (or last) two lines. I want to preserve the order of lines in the file.
Consider a file input.txt
where foo\nbar
is repeated and baz\nboo
is repeated, each in consecutive two-line blocks.
1
foo
bar
foo
bar
2
3
baz
boo
baz
boo
4
desired contents:
1
foo
bar
2
3
baz
boo
4
uniq
, sed
The same task is fairly simple for removing single line duplicates: uniq input.txt
. However, man uniq
doesn't suggest that there is an option to get it to work for my use case.
I also had a look at sed
, but couldn't get it to work. EDIT: didn't try anything specific as these docs only consider searching and replacing within a two-lines block, not four lines.
If you want to accept a perl
solution then:
perl -0777 -pe 's/(.+\R.+\R)\1/$1/g' file
1
foo
bar
2
3
baz
boo
4