5

I've tested my server¹ with a python port of mechanize - multi-mechanize. I ran several pretty simple tests - but I always get a drop from 10mbits to some kb of upload bandwith. And I've no idea why.


Whether I run 3,15 or 30 minutes makes no difference to the result. There is always a bandwith drop to almost zero between 110 and 120s, as you can see in the given analysis below. I've picked a short run, so it's easier to spot the drop.

A check of htop shows nothing special, cores are running between 2 and 7%.
memory usage is always around 1000mb (+-100) of 2048mb guaranteed memory.

When I check iftop there is nothing special but a upload drop from 10mbits to some kilobytes for ~10 seconds (110-120s)

All cronjobs / unneccessary tasks are disabled. All pages (front/random) are available. Each request is answered by a http response code 200. Apache and MySQL error logs are empty.

Since I'm an admin who's learning by doing I'm not sure if there are more relevant informations. Are loaded apache mods relevant? Hope I mentioned all important facts.

config.cfg

[global]
run_time = 180
rampup = 0
results_ts_interval = 10
progress_bar = on
console_logging = off
xml_report = off


[user_group-1]
threads = 10
script = frontpage.py

[user_group-2]
threads = 10
script = randompost.py

frontpage.py

import mechanize

class Transaction(object):
    def run(self):
        br = mechanize.Browser()
        br.set_handle_robots(False)

        resp = br.open('http://host.domain.tld/')
        resp.read()

        assert (resp.code == 200), 'Bad Response: HTTP %s' % resp.code
        assert ('Example Web Page' in resp.get_data())

randompost.py

actually the same as frontpage but with random pages

import mechanize
import random

pages = [
'...',
'...',
'...',
# ...
]

class Transaction(object):
    def run(self):
        br = mechanize.Browser()
        br.set_handle_robots(False)

        resp = br.open(random.choice(pages))
        resp.read()

        assert (resp.code == 200), 'Bad Response: HTTP %s' % resp.code
        assert ('Example Web Page' in resp.get_data())

elapsed time / response time (secs) elapsed time / response time (secs) elapsed time / tps




What tools / methods can I use to narrow down the cause of this trough?


Update

As @linuxdevops mentioned, I tried to download files with scp and ftp. My tests included a 10mb file and the folder of my website - means many files from 1-1xx kb. There was no abadonment of the transfer or a any noticable lag. I'm not sure if there are more professional ways to determine the consistency of a FTP / SCP transfer.

¹ vhost specs

  • 3 vcores a 1.5ghz
  • 2048 mb ram (guaranteed, no dynamic ram)
  • 100 mbit bandwidth
  • centos 6.5 32bit
  • apache 2.2.15
2
  • 1
    I'd try downloading/uploading files bypassing web server using different protocols like (s)ftp / scp to narrow down the problem (bypass dns resolution using ip address directly to simplify) Apr 30, 2014 at 18:00
  • @LinuxDevOps actually I never had any problems with ftp/scp - but that's a good idea, I'll try to do some detailed tests in this way.
    – Lucas
    Apr 30, 2014 at 18:15

1 Answer 1

1

A good place to start is with a tool like netperf. Google to find it

  • Start the netserver binary on your vhost
  • From your client run netperf: netperf -H <serverIP> -f M -l 240 -- -s 4194304

    • -f M (show output in MB/s)
    • -l (length in seconds)
    • -- (options follow the two dashes)
    • -s (socket size)

This is an easy way to find the right socket/buffer size. For example, the default socket size in Windows is only 8192. A copy using drag and drop will use this default size and you'll max out at around 22 MB/s. Once you increase that to 64k you'll start seeing your 100-120 MB/s. Most software these days allow you to change this or will hard code their tested sweet spot. So if using winscp, or filezilla or whatever utility for these file transfers then you'll want to check your TCP buffers in Linux, and your winsock buffers in Windows.

Linux Example: /etc/sysctl.conf

  • net.ipv4.tcp_rmem = 4194304 4194304 4194304
  • net.ipv4.tcp_wmem = 4194304 4194304 4194304

Windows: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters

  • DefaultReceiveWindow = 65536
  • DefaultSendWindow = 65536

Reboot

If you can run netperf for over your 120 seconds and don't see your trough, but then copy actual data to disk and still see it then you can move on to troubleshooting your disk. If you try various buffer/socket sizes and still see the decrease then my next step would be a packet capture.

On the vhost:

  1. tcpdump -i <interface> -vvv -nn -s0 port 12865 -w /desiredDir/troughTest.cap
  2. netserver
  3. From client: netperf -H <serverIP> -f M -l 300 -- -s 4194304

Let that run for a while then ctrl-c or kill it when you think you've got enough packets. Last, ctrl-c your tcpdump, transfer your /desiredDir/troughTest.cap file to your laptop/workstation, install wireshark if you haven't already, analyze packets

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .