Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have an OpenBSD server used as firewall using the famous pf. The firewall is connected to the Internet form one side and to a local network form the other. we are experiencing a connection outage due to the fact that the pf is attaining its maximum number of states (which is 20000), this happens for less than hour then thinks goes back to normal. Is there anyway of determining the hosts that are opening these states. Does increasing the number of states in pf.conf helps?

Thanks a lot

share|improve this question
add comment

2 Answers 2

up vote 4 down vote accepted

There are a number of things you can do here.

To see which hosts are responsible for the large number of state table entries you can do pfctl -vs state.

To add more state table entries you can do what you suggested (set limit states to a bigger number), but if there is an underlying issue you probably don't want to do that.

You can also consider adjusting the state timeout values (set timeout), possibly using adaptive timeouts, in order to get rid of old/stale states more quickly.

See the manpage for pf.conf and the manpage for pfctl for more information

share|improve this answer
add comment

[Ref: Hitting the PF state table limit]

The PF State tables set the limit of connections that have been authorised, and thus limits the number new connections that the firewall will accept. You may have excess bandwidth available, but if there are no free capacity in the State Tables, then your firewall becomes a bottle-neck.

The configured limits for state information is accessible through "pfctl"

# pfctl -sm
states        hard limit    10000   
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   200000

The above limits pre-sets the allocated memory the the defined structures, such that they are always available, and it also limits growth of the said data structures. If your firewall traffic exceeds the above settings, then performance will be effected.

It is now important to monitor the effects of your traffic on the counters for the above limits. The generic "-s info*" output gives us clues to where to further investigate potential bottle-necks in our firewall.

# pfctl -si
Status: Enabled for XXXXXXXXXXXXXXXX          Debug: Urgent

State Table                          Total             Rate
  current entries                       34
  searches                        96379206           15.2/s
  inserts                           726196            0.1/s
  removals                          726162            0.1/s

On the above gateway, connected to two infrequently used laptops, the current entries is very low relative to the hard limit 10000 above. Obviously, the current entries will fluxuate due to use, and on a busier gateway may fluxuate significantly.

[Ref: Hitting the PF state table limit, Open BSD state hard-limit reached]

An important counter to monitor from pfctl -si is the "memory" counter. The same details should be availble through systat pf

From an active gateway linking our 6 sites, we get the following from a standard install, no modification to state tables.

# pfctl -si | grep memory
  memory                          209230            0.1/s

The counter highlights how often PF has failed at least one of the "pool(9)" The higher the number, the higher the frequency of incidences where packets arriving at the firewall have most likely been dropped due to one of the hardware limits.

Our above example shows 209,230 times the memory limit was hit.

The next review is to check with Kernel memory allocations, using "vmstat". To narrow our search down to the effects on the pf state table we check the entry for pfstatepl.

Below, we grab the lines with state or Fail (so we can get the column headers)

# vmstat -m | grep -E "state|Fail"
Name      Size  Requests  Fail   InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg  Idle
pfstatepl  296  213123877 209235  5075  1050     0  1050 1050     0   2308   526
pfstatekeypl 
pfstateitempl

pfstatepl is the label for memory allocated for the *struct pf_state* (/usr/src/sys/net/pf_ioctl.c) The failures do seem to be significant.

pfctl -vvsi | grep congestion

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.