The tale of the NIC that quit;
I have chucked the end of dmesg output from a server into pastebin;
This server (PowerEdge 1850) has two NICs, eth0 and eth1. eth1 has a couple of VLANs defined on it, and those in turn are in different bridges, one bridge has multiple IPs. eth1 is the public facing interface. eth0 if for backend/management access.
The server went "off-line" in the sense that it stopped serving public requests and I received an alert. I connected via SSH into using the management IP on eth0, to find the server up, load is low, plenty of disk space, RAM and cpu cycles etc. All services were up and running, but the server wasn't serving any webpages.
That's when I checked dmesg and saw the above output. It seems that there was a problem with eth1 and it wouldn't send any packets, but it was receiving them. There are a few "Reset adapter" messages in the dmesg output, so I assume the server was "self repairing"?
[10716872.816012] e1000 0000:07:08.0: eth1: Reset adapter
tcpdump to see what was going on, (the output of which I have misplaced!). However, I could see that the default gateway that faces the public VLAN sub-interfaces on eth1 was ARP'ing out for the public IPs assigned to the server but it wasn't sending any response.
So this is likely why the public facing services weren't working. I restarted the interface with
sudo ifdown eth1 && sudo ifup eth1 which executed successfully, but didn't help.
I checked the arp table;
user@server:~$ arp -n Address HWtype HWaddress Flags Mask Iface 22.214.171.124 (incomplete) br12
Seeing this incomplete address I took a shot in the dark (not really expecting it to work) and added the MAC for the default gateway of 126.96.36.199 manually. It didn't work.
It had been several minutes of production downtime so I rebooted the server moments later, and everything was back to normal after it rebooted.
Below I have posted the /etc/network/interfaces contents, however I need some help understanding the pastebin entry I linked above. What is a possible cause for eth1 to quit it's day job, in the middle of the working day?
allow-hotplug eth0 allow-hotplug eth1 allow-hotplug eth1.1 allow-hotplug eth1.2 auto eth0 iface eth0 inet static address 10.0.1.25 netmask 255.255.255.0 auto eth1 iface eth1 inet manual auto eth1.2 iface eth1.2 inet manual vlan_raw_device eth1 auto br12 iface br12 inet static address 10.0.0.25 netmask 255.255.255.0 bridge_ports eth1.2 bridge_stp off auto eth1.1 iface eth1.1 inet manual vlan_raw_device eth1 auto br11 iface br11 inet static address 188.8.131.52 netmask 255.255.255.248 gateway 184.108.40.206 bridge_ports eth1.118 bridge_stp off auto br11:0 iface br11:0 inet static address 220.127.116.11 netmask 255.255.255.248 auto br11:1 iface br11:1 inet static address 18.104.22.168 netmask 255.255.255.248
For debug purposes;
user@server:~$ uname -a Linux server.site.com 3.4.10 #1 SMP Thu Sep 13 13:12:24 BST 2012 x86_64 GNU/Linux user@server:~$ cat /etc/issue Debian GNU/Linux 6.0 \n \l
The server has been up for 3 days and 17 hours now, no errors in dmesg/kern.log/message/syslog and it's running fine. This is the
lshw details for the NICs.