The website that I administrate suddenly started running slowly today. I'm trying to figure out where the bottleneck is. I created a 14kb text file and this takes 5 seconds to open in the browser! I'm confident that the problem is not with my internet connection - I can watch youtube just fine for example.
If I access the 14kb file on the lan then the page takes 6ms to open. This makes me think that the problem is not apache.
However, the website also uses mysql, and when I open a page which displays data in a table it takes ages - like 10 minutes or more. On the lan this opens straight away. I check the full processlist while the page is loading over the internet and there are queries which are taking a long time:
mysql> show full processlist;
+------+------+-----------+------+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+------+-----------+------+---------+------+-------+-----------------------+
| xxxx | root | localhost | fms | Sleep | 307 | | NULL |
| yyyy | root | localhost | NULL | Query | 0 | NULL | show full processlist |
+------+------+-----------+------+---------+------+-------+-----------------------+
2 rows in set (0.00 sec)
Can anyone help me diagnose this problem? I have full root access to the ubuntu machine running the webserver. The website is generated from apache and php.
more info
i have since been home and come back to work. strangely, the website works fine from my house! so here is the current status:
- at home: works fine
- at work on lan2 (different lan to the server): miserably slow
- at work on lan1 (same lan as the server): works fine
it is possible that the problem is with the dns on work lan2. maybe the isp is caching or filtering the requests?
following the traceroute lead...
- on work lan1 there is of course only one hop
- on work lan2, traceroute using icmp echos (pings) did not complete in 60 hops (entries 9 through 60 are all * * *), but the traceroute using tcp syn packets completed in 12 hops
- traceroute from my house also did not get through using icmp echos but it completed in 9 hops using tcp syn packets.
it seems like the problem is completely confined to lan2 - which indicates to me that there is either some configuration issue (eg a bad dns entry that has been cached for this website only, or else maybe some routing that is getting itself caught in a loop) or else some piece of hardware has died. all suggestions on debugging this issue are welcome.
further information from wireshark
i noticed that the pages are relatively quick to connect, but take ages to finish loading. i also ran wireshark while opening the page and found a lot (hundreds) of the following packets:
source dest protocol info
website work pc on lan2 TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
work pc on lan2 website TCP 62783 > https [ACK] Seq=667 Ack=26435 Win=260 Len=0
website work pc on lan2 TCP [TCP Previous segment not captured] [TCP segment of a reassembled PDU]
work pc on lan2 website TCP [TCP Dup ACK 192#1] 62783 > https [ACK] Seq=667 Ack=26435 Win=260 Len=0 SLE=xxxxx SRE=xxxxx
website work pc on lan2 TCP [TCP segment of a reassembled PDU]
i'm wondering if the problem is with the ssl certificate cached on lan2? really grasping at straws now...i'm going to google some of the terms from the wireshark info and maybe that will lead somewhere.
