2

I'm looking for an explanation of the totsck column for the "sar -n SOCK" output

09:44:06 PM    totsck    tcpsck    udpsck    rawsck   ip-frag    tcp-tw
09:44:09 PM       580        18         5         0         0         1
09:44:10 PM       580        18         5         0         0         0
09:44:11 PM       580        18         5         0         0         0
Average:          580        18         5         0         0         1

It's obviously not the sum of the tcp/udp/raw sockets. The only other explanation I've come around is that it's sockets plus

 sysctl fs.file-nr 

but on my test box that's

fs.file-nr = 5632   0   803168

Precise explanation much appreciated.

Thanks

Edit 2: So apparently totsck is equivalent to

cat /proc/net/sockstat

which leads to the question what is counted there. I found this but in the end it only recommends asking the guys who wrote that particular piece of kernel code.

Edit (for domain socket accounting):

[root@fedora16 fs]# netstat --protocol unix| wc -l
413
[root@fedora16 fs]# sar -n SOCK 1 1
Linux 3.3.1-5.fc16.x86_64 (fedora16)    06/21/2012  _x86_64_    (4 CPU)

10:03:25 PM    totsck    tcpsck    udpsck    rawsck   ip-frag    tcp-tw
10:03:26 PM       598         6         5         0         0         3
Average:          598         6         5         0         0         3
1
  • The link to the discussion on sockstat implemetnation is broken.
    – Otheus
    Feb 8, 2023 at 22:34

2 Answers 2

7

There's also UNIX domain sockets (STREAM and DGRAM) that are accounted for in total number of sockets used by the system as it seems. UNIX domain sockets are referenced by processes as inodes in the file system. There's a lot of stuff that still uses UNIX domain sockets for various purposes so sar picks that up. Check that output of netstat -a to see how many UNIX domain sockets are open on your system.

fs.file-nr is the number of maximum file handles and while important has nothing to with what you are seeing there on the sar output.

Edit: Please consider that sar basically reads /proc/net/sockstat and makes an average over that count or reports historical values. It seems that /proc/net/sockstat gets the data from two places (kernel source for 2.6.27) and the locations are net/socket.c line: 2324 and net/ipv4/proc.c line 54 and following and the total number comes from the first locations while the rest is from the second. Going through the net structure also reveals what sockets are counted/accounted for and printed into the proc file system.

 79  * @SOCK_STREAM: stream (connection) socket
 80  * @SOCK_DGRAM: datagram (conn.less) socket>  
 81  * @SOCK_RAW: raw socket
 82  * @SOCK_RDM: reliably-delivered message>
 83  * @SOCK_SEQPACKET: sequential packet socket
 84  * @SOCK_DCCP: Datagram Congestion Control Protocol socket
5
  • Is there any command with which I can count the unix domain sockets (like netstat -t | wc -l for the tcp sockets)?
    – moodywoody
    Jun 21, 2012 at 12:00
  • Check the output of netstat --protocol unix | wc -l.
    – pfo
    Jun 21, 2012 at 12:02
  • Cheers and already thanks, but still doesn't seem to add up (see edit in OP)
    – moodywoody
    Jun 21, 2012 at 12:05
  • I've just skimmed through the source to see how it's calculated and edited my answer accordingly.
    – pfo
    Jun 21, 2012 at 12:52
  • Cheers mate, quality answer
    – moodywoody
    Jun 21, 2012 at 13:00
0

OK, so I had this problem too, with the conclusion that all kernels have this apparent flaw / bug. I've tested this on 3.10, 4.2, 5.1, and 6.2 kernels, and the code from before 2.6 to 4.16 is unchanged.

So it seems we have leaky sockets!

Consider the following commands:

ss -Han |wc -l ;\
awk 'NR==1 { print $NF }' /proc/net/sockstat

The output of each command should be about the same (at least 1 socket is created by the first command). But for many of my servers, the output of each command is vastly different.

sar gets its information from the /proc/net/sockstat, just as we see in the second command. This information is provided by code in the kernel socket driver, explained below.

It's possible both ss and netstat do not provide an exhaustive list of sockets. That would be a bug or lack of documentation on their part.

It could be the bug is in the code of net/socket.c. However, I could not find it. That code handles the allocation and deallocation of all sockets in the system. When it creates a new one, it increments a counter. When it destroys one, it decrements a counter. Pretty simple, should not have room for error.

However, the code does have some hidden complexity in regards to multi-CPU systems. I found that my single-CPU VMs had overall lower discrepencies than my other systems. But these systems are allocated only 1 CPU because they are test systems and not heavily used. The disparity on my single-CPU systems is < 10%. But for multi-cpu VMs, there is a wide-variation, from < 1% to a factor of 10!

The per-CPU code in net/socket.c looks like this when reporting:

        for_each_possible_cpu(cpu)
            counter += per_cpu(sockets_in_use, cpu);

The incrementers and decrementers look like:

percpu_add(sockets_in_use,1);
...
percpu_sub(sockets_in_use,1);

This code hasn't fundamentally changed from before 2005 until 2017, in 4.16 with net-namespaces. But I still experience the discrepancy in versions up to 6.2, so perhaps underneath it's the same code.

These per-cpu macros hide tons of details. I don't understand it all, but maybe I will, after I study this page. Ultimately, it should act like a cpu-number indexed array. I imagine this construct is needed so that the kernel running on each CPU can update its own counter to avoid costly spin-locking and possibly cache-flushing. The macros ensure these add/sub operations are atomic. Each counter is an int. No bounds checking are performed. Integer math will work correctly even with overflow/underflow across multiple CPUs. And it's very well-known code used all over the place.

Another possibility is that one or more of the memory areas used by the socket counters are corrupted by other code in the system.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .