8

In virtually every FreeBSD network tuning document I can find:

# /boot/loader.conf
net.inet.tcp.tcbhashsize=4096

This is usually paired with some unhelpful statement like "TCP control-block hash table tuning" or "Set this to a reasonable value." man 4 tcp isn't much help either:

tcbhashsize         Size of the TCP control-block hash table (read-only).
                    This may be tuned using the kernel option TCBHASHSIZE
                    or by setting net.inet.tcp.tcbhashsize in the
                    loader(8).

The only document I can find that touches on this mysterious thing is the Protocol Control Block Lookup subsection beneath Transport Layer in Optimizing the FreeBSD IP and TCP Stack, but its description is more about potential bottlenecks in using it. It seems tied to matching new TCP segments to their listening sockets, but I'm not sure how.

What exactly is the TCP Control Block used for? Why would you want to set its hash size to 4096 or any other particular number?

2
  • +1, very interesting question! Mar 22, 2012 at 20:13
  • AFAIK, all information for delivering packet to appropriate socket is available via inpcb only. Mar 23, 2012 at 18:31

1 Answer 1

3

It's more like computer science question. Especially if you want to dig into hash tables and big-O notations.

The answer is:
If you are handling many TCP sessions on sever you really want to look up connection's tcp parameters in O(1) time instead of O(n). FreeBSD uses chaining to resolve hash table collisions. So if there is lots of connection there will be lots of collisions and so instead of O(1) hash table lookup you'll need to do an linear chain lookup with O(n) complexity.

Parameter you mentioned - tcbhashsize is basically number of buckets in hash table.
On our servers it's set to pretty high values like 16384 and even higher. With that setting we are handling about 60,000 connections per server.

Each entry in hash table by itself currently on x86_64 uses 252 bytes (tcp_inpcb) + 688 bytes(tcpcb) of kernel memory for each entry (kmem size is 512G in amd64 since 7.2+ IIRC). It can be viewed via vmstat -z.

About structure of TCP Control block you can read FreeBSD sources: tcp_var.h or read TCP/IP Illustrated, Volume 2: The Implementation By Gary R. Wright, W. Richard Stevens

6
  • It was all blurry but now with this obscure C header everything is clear ;)
    – gparent
    Mar 23, 2012 at 16:42
  • I understand why increasing the number of buckets in a hashtable would help performance of lookups in those buckets, I didn't realize this was actually what this value was doing. If this is a table of buckets, then I suppose the TCPCB really is where socket information is stored so TCP segments can be matched up to the proper receiver. Can you confirm this? Also, part of the purpose of these sites is to aggregate information, so "Read the source" or "Read a book" answers aren't very useful.
    – sh-beta
    Mar 23, 2012 at 16:50
  • How did you come to your tuning of 16384? Why that? And what are you sacrificing for that value (I assume kernel memory, but how much?)? If it were a free performance win, I like to think it'd be the default. Surely it costs something.
    – sh-beta
    Mar 23, 2012 at 16:52
  • In my opinion this value should be set somewhat close to number of concurrent connections this server is willing to handle. PS. Do you really want become an expert in some area without reading sources/books? =) Mar 23, 2012 at 18:28
  • 1
    @SaveTheRbtz I abhor this notion that, if you use a technology, you need to either stop asking questions or become so expert in the code that you can recite the precise purpose of each individual struct and function in the network stack. The purpose of StackExchange is exchanging knowledge. I'm an expert in some things and not others. That line is determined by my work where I have to carefully choose where I spend my time. But that doesn't mean I'm content to simply accept tuning "advice" that appears to have been unthinkingly copied and pasted from blog to blog.
    – sh-beta
    Mar 23, 2012 at 21:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .