2

we have a syslog-ng server acting as an intermediary logger.

the server receives data from all devices on the network (from many different sources, on both UDP & TCP ports), filters them a bit, and forwards the data to a SIEM (splunk).

when looking at ifconfig stats, i can see this :

[root@xxxxxxxx ~]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:50:56:00:00:00
          inet addr:1.1.1.1    Bcast:1.1.1.1    Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:60451021996 errors:0 dropped:0 overruns:0 frame:0
          TX packets:158501574 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:5000
          RX bytes:12508573129969 (11.3 TiB)  TX bytes:220148785267 (205.0 GiB)

this part is quite shocking to me :

          RX bytes:12508573129969 (11.3 TiB)  TX bytes:220148785267 (205.0 GiB)

obviously, the server received around 11 Terrabytes of data since the last boot (15 days uptime) but "only" transferred a little part of it ? (250 Gbs).

is there a way for me to check how is this possible ? i know there's a bunch of UDP dropped messages, but is it taken in consideration ? (dropped is displayed 0).

i expect a little less TX (due to the fact that we filter some messages), but not that much (10% max).

For information, Netstat on UDP shows :

Udp:
 10903564401 packets received
 8401685 packets to unknown port received.
 49356622070 packet receive errors
 43665773 packets sent
 RcvbufErrors: 34287641

any tips on how to investigate this ? thanks !

3
  • 1
    It's not just a fifth, but rather factor forty. My question: Are you sure Splunk is accessed via eth0? Next question, how are the logs processed before they are forwarded - you do talk about filtering; any way to look into this? Troubleshooting suggestion: You could use other tools to validate the data, such as venerable iptraf, which also tells you how much traffic comes from and goes to which IP addresses. Feb 19, 2021 at 0:49
  • thanks for your answer (and you are right it's not fifth, i will edit the message, sorry for that). Yes i control every filter on the server, which is why i know that "not that many data is filtered out. yes there is only 1 network interface, but we send on dozens of different ports to Splunk, so ideally i would like to see the traffic incoming & outgoing on every port, i have to look into this in order to see the bandwidth usage per "port" and see which feed is causing issues
    – olivierg
    Feb 19, 2021 at 10:25
  • 1
    Given the rcvbuferrors, could it be that your syslog server is not keeping up with the flow of data? What are your input and output throttle limits set to? Are you also seeing locked sockets increment in netstat -s? If not and syslog is keeping up, then maybe this is just sysctl socket buffer tuning. Are you forwarding to splunk with udp or tcp? Syslog-ng itself also has buffer settings and if using tcp, there could be rate limits on the splunk forwarder / listener.
    – Aaron
    Feb 19, 2021 at 16:39

2 Answers 2

2

To be clear, RX and TX are Receive and Transmit (not transfer).
Ie, traffic sent to this host (RX on its network interface) and sent from this host (TX on its network interface), respectively.

      RX packets:60451021996 errors:0 dropped:0 overruns:0 frame:0
      TX packets:158501574 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:5000
      RX bytes:12508573129969 (11.3 TiB)  TX bytes:220148785267 (205.0 GiB)

In your example above, that network interface has received much more data than it has transmitted.

If the main traffic happening here is other hosts sending logs to it, it sounds expected that it receives more than it transmits.

1
  • 1
    My impression of the question is that less than 2% of message data surviving the filtering and sent to the SIEM host was surprising. Which can't be answered with layer 3 counters, needs further analyses of how much is actually syslog traffic and how reliably it gets delivered then forwarded. Feb 20, 2021 at 15:05
0

Layer 3 counters do not prove what is happening at layer 7, the application. Possibly you have far more non logging traffic, like IP storage or software update. Or are filtering logs more aggressively than you thought.

Do packet capture on the log server for a few hours to get a sample. Find the ratio of protocols, such as with Wireshark's Statistics > Protocol Hierarchy. What percent is syslog?

Consider temporarily saving unfiltered log entries to do statistics on those. Would require a considerable sized fast storage. Log search engines like Graylog can ingest as many messages as you can send, at the cost of maintaining an Elasticsearch cluster.

Compare the raw data going through syslog to the SIEM. Get the real ratio of syslog data ingested to stored in the SIEM. Count up the number of unique host names or IP addresses and compare to your inventory.

If any hosts also store log messages locally, pick a few and query those files. Look for messages that should get to the SIEM, and confirm they do. (Local logs are not ideal when central logging exists, but might as well examine them.)

1
  • I can add to this: configure flow-tools, that is very useful to analyze traffic patterns. Feb 19, 2021 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.