[Ref: Hitting the PF state table limit]
The PF State tables set the limit of connections that have been authorised,
and thus limits the number new connections that the firewall will
accept. You may have excess bandwidth available, but if there are no free
capacity in the State Tables, then your firewall becomes a bottle-neck.
The configured limits for state information is accessible through "pfctl"
# pfctl -sm
states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 200000
The above limits pre-sets the allocated memory the the defined structures,
such that they are always available, and it also limits growth of the said
data structures. If your firewall traffic exceeds the above settings, then
performance will be effected.
It is now important to monitor the effects of your traffic on the counters
for the above limits. The generic "-s info*" output gives us clues to
where to further investigate potential bottle-necks in our firewall.
# pfctl -si
Status: Enabled for XXXXXXXXXXXXXXXX Debug: Urgent
State Table Total Rate
current entries 34
searches 96379206 15.2/s
inserts 726196 0.1/s
removals 726162 0.1/s
On the above gateway, connected to two infrequently used laptops,
the current entries is very low relative to the hard limit 10000 above.
Obviously, the current entries will fluxuate due to use,
and on a busier gateway may fluxuate significantly.
[Ref: Hitting the PF state table limit,
Open BSD state hard-limit reached]
An important counter to monitor from pfctl -si is the "memory"
counter. The same details should be availble through systat pf
From an active gateway linking our 6 sites, we get the following from a standard
install, no modification to state tables.
# pfctl -si | grep memory
memory 209230 0.1/s
The counter highlights how often PF has failed at least one of the "pool(9)"
The higher the number, the higher the frequency of incidences where packets arriving
at the firewall have most likely been dropped due to one of the hardware limits.
Our above example shows 209,230 times the memory limit was hit.
The next review is to check with Kernel memory allocations, using "vmstat".
To narrow our search down to the effects on the pf state table we check the entry
for pfstatepl.
Below, we grab the lines with state or Fail (so we can get the column headers)
# vmstat -m | grep -E "state|Fail"
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
pfstatepl 296 213123877 209235 5075 1050 0 1050 1050 0 2308 526
pfstatekeypl
pfstateitempl
pfstatepl is the label for memory allocated for the *struct pf_state*
(/usr/src/sys/net/pf_ioctl.c) The failures do seem to be significant.
pfctl -vvsi | grep congestion