1

Let's see if I can explain this right:

  1. I have servers with two ethernet interfaces - one external and one internal facing.
  2. I have a DB (pg) on one server and accessible from other machines on the internal network.
  3. I have RoR apps on other servers and configured to use the DBs on said DB server.
  4. Mongrel Clusters ARE configured correctly - I can start the cluster and access the Rails app through the console.
  5. I am on ubuntu 9.10 server edition.
  6. IPs are assigned statically on the machine - there is no DHCP at work.

OK, so the issue is the init scripts to start Mongrel fire BEFORE the ethernet adapters appear to come up. The problem of course is that the DB is then not visible to the Rails App as the network is not yet active.

Relevant portion of the syslog boot events is posted below.

I have added the mongrel scripts "normally" to init by using /usr/sbin/update-rc.d mongrel_cluster defaults. I have moved the "boot order" to 99z_mongrel_cluster. PLEASE NOTE that in the syslog below that the mongrel cluster is executed right after CRON so it seems that I have it as late as possible WRT to init.d.

What looks like is happening is that the ethernet interfaces try to activate right at the end of the init cycle then take some time to activate. I'm a bit ignorant about some of this but it looks like we go from 7.537... on tcpdump to eth0 NIC up at 9.266... which is over 1.5 seconds to load?

This works "as it should" in my other environments which use DHCP to assign static IP addresses.

So, the questions are:

  1. is there a way to get the NICs to load sooner or more quickly (based on HDs coming up I'd guess "no" to the sooner question)?

  2. is there a way to make sure the NICs are active BEFORE moving on to other init tasks? Based on what I read of the LSB spec I have put $network in the init script for Required-Start for the mongrel cluster (again, have not had to do this in other test environments).

  3. is there something I have to do to configure the network to load more quickly? my /etc/hosts and /etc/resolv.conf look right.

  4. driver issue?

Much thanks. I've been scratching my head for a couple of days on this.

Jul 26 15:08:03 web-01 kernel: [    4.166954] usbhid: v2.6:USB HID core driver
Jul 26 15:08:03 web-01 kernel: [    4.321657] 0000:03:00.0: eth0: (PCI Express:2.5GB/s:Width x1) 00:30:48:9f:08:cc
Jul 26 15:08:03 web-01 kernel: [    4.321659] 0000:03:00.0: eth0: Intel(R) PRO/1000 Network Connection
Jul 26 15:08:03 web-01 kernel: [    4.321743] 0000:03:00.0: eth0: MAC: 3, PHY: 8, PBA No: 0101ff-0ff
Jul 26 15:08:03 web-01 kernel: [    4.321974] e1000e 0000:04:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Jul 26 15:08:03 web-01 kernel: [    4.322008] e1000e 0000:04:00.0: setting latency timer to 64
Jul 26 15:08:03 web-01 kernel: [    4.322264]   alloc irq_desc for 37 on node 0
Jul 26 15:08:03 web-01 kernel: [    4.322265]   alloc kstat_irqs on node 0
Jul 26 15:08:03 web-01 kernel: [    4.322270] e1000e 0000:04:00.0: irq 37 for MSI/MSI-X
Jul 26 15:08:03 web-01 kernel: [    4.322271]   alloc irq_desc for 38 on node 0
Jul 26 15:08:03 web-01 kernel: [    4.322272]   alloc kstat_irqs on node 0
Jul 26 15:08:03 web-01 kernel: [    4.322274] e1000e 0000:04:00.0: irq 38 for MSI/MSI-X
Jul 26 15:08:03 web-01 kernel: [    4.322275]   alloc irq_desc for 39 on node 0
Jul 26 15:08:03 web-01 kernel: [    4.322276]   alloc kstat_irqs on node 0
Jul 26 15:08:03 web-01 kernel: [    4.322278] e1000e 0000:04:00.0: irq 39 for MSI/MSI-X
Jul 26 15:08:03 web-01 kernel: [    4.397418] scsi6 : 3ware 9000 Storage Controller
Jul 26 15:08:03 web-01 kernel: [    4.397484] 3w-9xxx: scsi6: Found a 3ware 9000 Storage Controller at 0xfadff000, IRQ: 16.
Jul 26 15:08:03 web-01 kernel: [    4.481131] 0000:04:00.0: eth1: (PCI Express:2.5GB/s:Width x1) 00:30:48:9f:08:cd
Jul 26 15:08:03 web-01 kernel: [    4.481133] 0000:04:00.0: eth1: Intel(R) PRO/1000 Network Connection
Jul 26 15:08:03 web-01 kernel: [    4.481218] 0000:04:00.0: eth1: MAC: 3, PHY: 8, PBA No: 0101ff-0ff
Jul 26 15:08:03 web-01 kernel: [    4.757252] 3w-9xxx: scsi6: Firmware FE9X 4.10.00.007, BIOS BE9X 4.08.00.002, Ports: 2.
Jul 26 15:08:03 web-01 kernel: [    4.757743] scsi 6:0:0:0: Direct-Access     AMCC     9650SE-2LP DISK  4.10 PQ: 0 ANSI: 5
Jul 26 15:08:03 web-01 kernel: [    4.768713] sd 6:0:0:0: Attached scsi generic sg0 type 0
Jul 26 15:08:03 web-01 kernel: [    4.769035] sd 6:0:0:0: [sda] 585916416 512-byte logical blocks: (299 GB/279 GiB)
Jul 26 15:08:03 web-01 kernel: [    4.769880] sd 6:0:0:0: [sda] Write Protect is off
Jul 26 15:08:03 web-01 kernel: [    4.769885] sd 6:0:0:0: [sda] Mode Sense: 23 00 10 00
Jul 26 15:08:03 web-01 kernel: [    4.770609] sd 6:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jul 26 15:08:03 web-01 kernel: [    4.772342]  sda: sda1 sda2 < sda5 >
Jul 26 15:08:03 web-01 kernel: [    4.775558] sd 6:0:0:0: [sda] Attached SCSI disk
Jul 26 15:08:03 web-01 kernel: [    5.278745] PM: Starting manual resume from disk
Jul 26 15:08:03 web-01 kernel: [    5.278747] PM: Resume from partition 252:1
Jul 26 15:08:03 web-01 kernel: [    5.278748] PM: Checking hibernation image.
Jul 26 15:08:03 web-01 kernel: [    5.278922] PM: Resume from disk failed.
Jul 26 15:08:03 web-01 kernel: [    5.301586] kjournald starting.  Commit interval 5 seconds
Jul 26 15:08:03 web-01 kernel: [    5.301606] EXT3-fs: mounted filesystem with ordered data mode.
Jul 26 15:08:03 web-01 kernel: [    5.689456] type=1505 audit(1280182081.322:2): operation="profile_load" pid=480 name=/sbin/dhclient3
Jul 26 15:08:03 web-01 kernel: [    5.689838] type=1505 audit(1280182081.322:3): operation="profile_load" pid=480 name=/usr/lib/NetworkManager/nm-dhcp-client.action
Jul 26 15:08:03 web-01 kernel: [    5.690045] type=1505 audit(1280182081.322:4): operation="profile_load" pid=480 name=/usr/lib/connman/scripts/dhclient-script
Jul 26 15:08:03 web-01 kernel: [    5.703353] type=1505 audit(1280182081.332:5): operation="profile_load" pid=481 name=/usr/sbin/tcpdump
Jul 26 15:08:03 web-01 kernel: [    6.118815] Adding 1949688k swap on /dev/mapper/web--01-swap.  Priority:-1 extents:1 across:1949688k 
Jul 26 15:08:03 web-01 kernel: [    6.262044] EXT3 FS on dm-0, internal journal
Jul 26 15:08:03 web-01 kernel: [    6.298767] udev: starting version 147
Jul 26 15:08:03 web-01 kernel: [    6.798595] ADDRCONF(NETDEV_UP): eth1: link is not ready
Jul 26 15:08:03 web-01 kernel: [    6.803872] ip_tables: (C) 2000-2006 Netfilter Core Team
Jul 26 15:08:03 web-01 kernel: [    6.830538] ADDRCONF(NETDEV_UP): eth0: link is not ready
Jul 26 15:08:03 web-01 kernel: [    6.834499] lp: driver loaded but no devices found
Jul 26 15:08:03 web-01 kernel: [    7.285196] kjournald starting.  Commit interval 5 seconds
Jul 26 15:08:03 web-01 kernel: [    7.321785] EXT3 FS on dm-2, internal journal
Jul 26 15:08:03 web-01 kernel: [    7.321792] EXT3-fs: mounted filesystem with ordered data mode.
Jul 26 15:08:03 web-01 kernel: [    7.535727] type=1505 audit(1280182083.165:6): operation="profile_replace" pid=868 name=/sbin/dhclient3
Jul 26 15:08:03 web-01 kernel: [    7.536095] type=1505 audit(1280182083.165:7): operation="profile_replace" pid=868 name=/usr/lib/NetworkManager/nm-dhcp-client.action
Jul 26 15:08:03 web-01 kernel: [    7.536298] type=1505 audit(1280182083.165:8): operation="profile_replace" pid=868 name=/usr/lib/connman/scripts/dhclient-script
Jul 26 15:08:03 web-01 kernel: [    7.537283] type=1505 audit(1280182083.165:9): operation="profile_replace" pid=869 name=/usr/sbin/tcpdump
Jul 26 15:08:03 web-01 cron[950]: (CRON) INFO (pidfile fd = 3)
Jul 26 15:08:03 web-01 init: apport pre-start process (943) terminated with status 1
Jul 26 15:08:03 web-01 init: apport post-stop process (958) terminated with status 1
Jul 26 15:08:03 web-01 cron[961]: (CRON) STARTUP (fork ok)
Jul 26 15:08:03 web-01 cron[961]: (CRON) INFO (Running @reboot jobs)
Jul 26 15:08:03 web-01 logger: mongrel: starting mongrel cluster
Jul 26 15:08:04 web-01 kernel: [    9.266524] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Jul 26 15:08:04 web-01 kernel: [    9.266892] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jul 26 15:08:05 web-01 kernel: [    9.430149] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Jul 26 15:08:05 web-01 kernel: [    9.430505] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Jul 26 15:08:05 web-01 kernel: [    9.777969] e1000e: eth0 NIC Link is Down
Jul 26 15:08:07 web-01 kernel: [   12.056047] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Jul 26 15:08:15 web-01 kernel: [   19.831671] eth0: no IPv6 routers present
Jul 26 15:08:16 web-01 kernel: [   20.401469] eth1: no IPv6 routers present
Jul 26 15:08:38 web-01 ntpdate[783]: can't find host ntp.ubuntu.com
Jul 26 15:08:38 web-01 ntpdate[783]: no servers can be used, exiting
Jul 26 15:08:53 web-01 ntpdate[1130]: step time server 91.189.94.4 offset -0.293984 sec
Jul 26 15:17:01 web-01 CRON[1294]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Improve this question

3 Answers 3

Reset to default
2

Unless you have done something weird networking is started before any services are started so I am a bit surprised things are not working.

I am not sure what you have in your /etc/network/interfaces file, but you may want to replace any lines like allow-hotplug eth0 with auto eth0. The hotplug option means that the interface will not be brought up until there is a fully negotiated network link.

I would also check your switch configuration. You may need to enable portfast or whatever the equivalent is on your switches. Portfast allows the computer to skip some of the negotiation related to spanning tree, vlans, and etherchannel and bring up the link faster. But you should not use this feature on any switch ports that are used to connect to other switches.

Improve this answer
1
  • thanks for the response! yeah - should have specified that. the interfaces are defined as auto as you say. the only weird part of that file is the external NIC has DNS defined but internal does not although I cannot see that really being an issue. auto eth1 iface eth1 inet static address 10.0.42.2 netmask 255.255.255.0 network 10.0.42.0 broadcast 10.0.42.255
    – Jeff
    Jul 26, 2010 at 23:24
1

You need to make Mongrel startup dependent on networking being up. Add something like:

start on started networking

to your /etc/init/mongrel.conf (or similar). Look at other files in that directory for guidance or see man 5 init.

Improve this answer
0

Update - Dennis your recommendation pointed me in the right direction although Ubuntu / Debian doesn't make use of Upstart by default which is what "start on" appears to be a command of. Before installing that I wanted to see if I could get this working with SysVInit.

The LSB specifies this type of action with Required-Start. I had used $network originally which appears to only activate when the NIC comes online not necessarily when an IP Address is acquired or other network services become active. I had then moved to $all but rather than query services this simply appears to move the script execution to the end of all other init scripts but does not wait on services.

So, to solve, I added $named and $time to Required-Start. $named refers to name resolution and $time requires that system time has been set after querying a time server. I know from syslog that this happens after eth0 so that should be safe.

Thanks for the help - that definitely helped me out.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .