Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

what the best way to remove duplicate IP's from file

I use the command:

        sort file | uniq

but I am not sure if this is the best way , maybe I missed something?

remark: my file contain two fields

example of file
share|improve this question
up vote 6 down vote accepted

I believe something as simple as 'sort -u ' should work for you

#sort -u /tmp/test

Check the 'sort' manpage for more info:

-u, --unique
with -c, check for strict ordering; without -c, output only the first of an equal run
share|improve this answer
Note: This does almost the exact same thing as sort | uniq and should be just as fast unless you're running the commands many times. – Chris S May 2 '13 at 14:02



This basically compares lines on a file in vi

share|improve this answer
how we can be sure that your code is better then sort file | uniq ? – yael May 2 '13 at 13:11
You can´t, but if you are already in an editor like vi you can kill two birds with one stone since you do not have to return to the shell. This is the great part of *nixes - you can do the same task at least in two different ways :) – german_guy May 2 '13 at 14:18
@yael If you care that much you can benchmark it on your workload, but honestly though you're investing more effort in this than it's worth. sort | uniq works, it works on every *nix variant I've ever encountered (including ones where sort doesn't understand -u), and it's generally "fast enough" on all but the most insanely huge data sets. Sometimes it's better to not worry about optimization :-) – voretaq7 May 2 '13 at 19:29

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.