Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So, my understanding of the ephemeral port system is that there's a tuple of four values identifying each connection: {source ip, source port, dest ip, dest port}, with source port being one of the ephemeral ports on a system. A connection cannot have the same tuple as the any other, so if you're making a bunch of simultaneous connections from your machine to the same port on another, you can only make as many connections as you have free ephemeral ports, as that's the only parameter which can be changed. This all makes sense to me (but please correct me if I'm wrong on any of it).

However, theoretically, if you were to use two different source ip addresses that would double the number of connections you could potentially make. I decided I wanted to test this, so I wrote a client/server test in erlang which makes and holds as many connections as it can. Using only one ip address, my connections capped out at about 52k. The ip address I used was 127.0.0.1.

Then I modified the script to also use the local network address of the machine (172.16.202.132). It was definitely making connections on both ips:

# lsof -Pnl +M -i4
beam.smp  7528     1000  947u  IPv4 3441692      0t0  TCP 172.16.202.132:32064->172.16.202.132:8888 (ESTABLISHED)
beam.smp  7528     1000  948u  IPv4 3441695      0t0  TCP 127.0.0.1:37225->127.0.0.1:8888 (ESTABLISHED)

(repeated seemingly endlessly)

But once again my connections capped out at 52k. I modified the two programs so that the server and client would use port 8889 as well as 8888. I was almost certain this would give me more, but once again I capped out at 52k. Connections were being properly made:

# lsof -Pnl +M -i4
beam.smp  7528     1000  946u  IPv4 3441689      0t0  TCP 172.16.202.132:26620->172.16.202.132:8889 (ESTABLISHED)
beam.smp  7528     1000  947u  IPv4 3441692      0t0  TCP 172.16.202.132:32064->172.16.202.132:8888 (ESTABLISHED)
beam.smp  7528     1000  948u  IPv4 3441695      0t0  TCP 127.0.0.1:37225->127.0.0.1:8888 (ESTABLISHED)
beam.smp  7528     1000  949u  IPv4 3441698      0t0  TCP 127.0.0.1:27965->127.0.0.1:8889 (ESTABLISHED)

(repeated seemingly endlessly)

Can anyone shed some light on why this may be occuring? I'm using Ubuntu 10.04, erlang R13B03. Here's the code I'm using for the client/server as well:

Server:

-module(contest).
-compile(export_all).
-define(TCP_OPTS, [binary, {packet, raw}, {nodelay, true}, {reuseaddr, true}, {active, false},{keepalive,true}]).

start() ->
    erlang:register(counter,spawn(fun()->?MODULE:counter(0) end)),

    %Gets the listen socket, generates acceptor threads
    case gen_tcp:listen(8888, ?TCP_OPTS) of
    {ok, Listen1} -> 
        ?MODULE:gen_accepts(10,Listen1)
    end,

    case gen_tcp:listen(8889, ?TCP_OPTS) of
    {ok, Listen2} -> 
        ?MODULE:gen_accepts(10,Listen2)
    end,

    ?MODULE:supervisor_loop({Listen1,Listen2}).

%Serves the purpose of keeping the listen socket open
%indefinitely
supervisor_loop(LS) ->
    receive
    _ -> ?MODULE:supervisor_loop(LS)
    end.    

%Generates I acceptor threads which constantly listen for
%new connections. Upon getting one, a new acceptor thread
%is spawned and the one which receieved a connection 
%continues on to process the connection
gen_accepts(0,_) -> ok;
gen_accepts(I,LS) ->
    spawn(?MODULE,accept_loop,[LS]),
    ?MODULE:gen_accepts(I-1,LS).

%Acceptor loop which spawns off sock processors when connections
%come in
accept_loop(Listen) ->
    case gen_tcp:accept(Listen) of
        {ok, Socket} ->
                Pid = spawn(fun()->?MODULE:process_sock(Socket) end),
                gen_tcp:controlling_process(Socket,Pid),
        whereis(counter)!plus;
        {error,_} -> ok
        end,
    ?MODULE:accept_loop(Listen).

%Holds socket, doesn't do anything
process_sock(Sock) ->
    receive
    _ -> process_sock(Sock)
        end.

counter(C) ->
    receive
    plus ->
        io:fwrite("~p\n",[C+1]),
        counter(C+1)
    end.

Client:

-module(flooder).
-compile(export_all).

start() -> 
    spawn(fun()->start("172.16.202.132",8888)end),
    spawn(fun()->start("172.16.202.132",8889)end).

start(Ip,Port) ->
    spawn(?MODULE,connect,[Ip,Port]),
    timer:sleep(2),

    case Ip of
    "127.0.0.1" -> ?MODULE:start("172.16.202.132",Port);
    "172.16.202.132" -> ?MODULE:start("127.0.0.1",Port)
    end.

connect(Ip,Port) ->
    case gen_tcp:connect(
        Ip,
        Port,
        [list,{active,true}]
    ) of
    {ok, Sock} -> io:fwrite("Connected on ~s\n",[Ip]),loop(Sock);
    {error,E} -> eMessage("connect",E)
    end.

loop(Sock) ->
    receive
    _ -> loop(Sock)
    end.

eMessage(W,E) ->
    io:fwrite("~w at ~s: ~s\n",[self(),W,E]).
share|improve this question

1 Answer 1

Your OS has a maximum number of IPv4 connections, regardless of how many IP addresses that you're listening on. That's probably the limit you're running into. There can also be limits per-process and per-user. Check all of those.

share|improve this answer
    
My net.ipv4.netfilter.ip_conntrack_max is set to 1048576, and ulimit -n yields 999999, so it isn't either of those. Are there any other settings I should be looking into? –  Mediocre Gopher May 20 '11 at 17:34
    
1/2 of your conntrack_max is 52k and change. You're using double the ports, right, since you're making and accepting the connections on the same server? Am I understanding your scenario correctly? Reading your post is tough for me, because I am not a programmer and your code doesn't mean much to be for the purposes of thinking about your problem, to be honest. –  mfinni May 20 '11 at 18:29
    
Yes, I'm hosting the client and server on the same box, but doesn't a listening socket only take one port, no matter how many external connections there are to it? –  Mediocre Gopher May 20 '11 at 18:57
    
Also, my ip_local_port_range is 8100-61000, so that's where the 52k probably comes from –  Mediocre Gopher May 20 '11 at 19:00
    
Also also, half my ip_conntrack_max would be 500k-ish –  Mediocre Gopher May 20 '11 at 19:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.