How do I delete BOTH lines of a duplicate pair in a text file in bash?

I want to delete both lines of a duplicate pair in a text file.

There is no way to do this using sort -u file or awk '!a[$0]++' file which both delete duplicated lines.

Somehow I would have to capture the deleted lines and run sed to use this list and then delete the already deleted (is there some way to get this output?). The pattern of the duplicated lines is not predictable.

I am looking for a shorter way than writing a complete bash program. It seems like a useful tool and there should be an easy way to do it.

Easy to delete one of the duplicates. I need to delete both and can't find a way to do it. Encountered during parsing.

1 Answer

You can use the -u command-line argument to the uniq utility, which does precisely what you want:

−u Suppress the writing of lines that are repeated in the input.

You still need to sort the input, of course:

sort file | uniq -u