The website that I administrate suddenly started running slowly today. I'm trying to figure out where the bottleneck is. I created a 14kb text file and this takes 5 seconds to open in the browser! I'm confident that the problem is not with my internet connection - I can watch youtube just fine for example.

If I access the 14kb file on the lan then the page takes 6ms to open. This makes me think that the problem is not apache.

However, the website also uses mysql, and when I open a page which displays data in a table it takes ages - like 10 minutes or more. On the lan this opens straight away. I check the full processlist while the page is loading over the internet and there are queries which are taking a long time:

mysql> show full processlist;
| Id   | User | Host      | db   | Command | Time | State | Info                  |
| xxxx | root | localhost | fms  | Sleep   |  307 |       | NULL                  | 
| yyyy | root | localhost | NULL | Query   |    0 | NULL  | show full processlist | 
2 rows in set (0.00 sec)

Can anyone help me diagnose this problem? I have full root access to the ubuntu machine running the webserver. The website is generated from apache and php.

more info

i have since been home and come back to work. strangely, the website works fine from my house! so here is the current status:

  • at home: works fine
  • at work on lan2 (different lan to the server): miserably slow
  • at work on lan1 (same lan as the server): works fine

it is possible that the problem is with the dns on work lan2. maybe the isp is caching or filtering the requests?

following the traceroute lead...

  • on work lan1 there is of course only one hop
  • on work lan2, traceroute using icmp echos (pings) did not complete in 60 hops (entries 9 through 60 are all * * *), but the traceroute using tcp syn packets completed in 12 hops
  • traceroute from my house also did not get through using icmp echos but it completed in 9 hops using tcp syn packets.

it seems like the problem is completely confined to lan2 - which indicates to me that there is either some configuration issue (eg a bad dns entry that has been cached for this website only, or else maybe some routing that is getting itself caught in a loop) or else some piece of hardware has died. all suggestions on debugging this issue are welcome.

further information from wireshark

i noticed that the pages are relatively quick to connect, but take ages to finish loading. i also ran wireshark while opening the page and found a lot (hundreds) of the following packets:

source           dest              protocol  info
website          work pc on lan2   TCP       [TCP Retransmission] [TCP segment of a reassembled PDU]
work pc on lan2  website           TCP       62783 > https [ACK] Seq=667 Ack=26435 Win=260 Len=0
website          work pc on lan2   TCP       [TCP Previous segment not captured] [TCP segment of a reassembled PDU]
work pc on lan2  website           TCP       [TCP Dup ACK 192#1] 62783 > https [ACK] Seq=667 Ack=26435 Win=260 Len=0 SLE=xxxxx SRE=xxxxx
website          work pc on lan2   TCP       [TCP segment of a reassembled PDU]

i'm wondering if the problem is with the ssl certificate cached on lan2? really grasping at straws now...i'm going to google some of the terms from the wireshark info and maybe that will lead somewhere.

  • 1
    The sleeping mysql process does not matter, it is just an open connection. I had a similar performance problem on a virtual machine, when another vm sucked up all available i/o, but since your site performs well on the lan, it sounds more like a network problem. – Ursula Aug 6 '13 at 11:32
  • Have you tried truning it off and on again? – Christopher Perrin Aug 6 '13 at 11:36
  • thanks guys. i have more info i will post now. and i will try a reboot :) – mulllhausen Aug 6 '13 at 11:43
  • no change after a reboot :( – mulllhausen Aug 6 '13 at 13:37

Since the traceroute is not being completed on LAN2 it could be a hardware fault. Do check for cable faults, router / switches whose service lights are blinking, replace them.

  • +1 - i would also check for a routing issue when your traceroute acts that weird. – Frederik Nielsen Aug 7 '13 at 0:49

i found a windows pc which was plugged in to both lans at work. the user mainly accesses the website discussed in the question, so i unplugged this pc from lan2 so that it would be able to communicate directly with the website on lan1. once i did this, all of the other pcs at work on lan2 started working fine for the website - connecting and loading is now back to the usual fast speeds. my guess is that this pc was routing half the website packets while the remainder went via the internet, and maybe they were not being reassembled back on the browser.

i'm not really sure if this is the case, but i am sure that the problem was fixed when i disconnected this pc from lan2.

obviously there was no way to guess the answer given the information i posted, but thanks to everyone who gave suggestions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.