Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We gave access to a contractor to install a firewall and somehow while he was doing it he fracked something up. Everything went off-line about 24 hours ago and we are effectively out of business until I solve this and the person who messed up the thing is not returning calls.

I found a few errors. First, I'm not a server guy - I can look at log files and normally everything runs fine. All 'services' are running according to 1and1 server monitoring and mail is being delivered just fine. The whole thing was off-line until I (probably stupidly) updated the kernel from 6.2 to 6.3 this morning and I got everything back except the http access.

All the domains (~200 of them) are returning a 403 error and nothing is recorded in the access log.

On every restart I see this error in the messages log file:

init: Failed to spawn ttyS0 main process: unable to execute: No such file or directory

and a little later these:

kernel: WARNING: at kernel/sched.c:5914 thread_return+0x232/0x79d() (Not tainted)
kernel: Hardware name: X9SCL/X9SCM
kernel: Modules linked in: xt_iprange iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ext4 jbd2 serio_raw i2c_i801 i2c_core sg iTCO_wdt iTCO_vendor_support e1000e ext3 jbd mbcache raid1 sd_mod crc_t10dif ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
kernel: Pid: 367, comm: md3_raid1 Not tainted 2.6.32-220.2.1.el6.x86_64 #1
kernel: Call Trace:
kernel: [<ffffffff81069997>] ? warn_slowpath_common+0x87/0xc0
kernel: [<ffffffff810699ea>] ? warn_slowpath_null+0x1a/0x20
kernel: [<ffffffff814eccc5>] ? thread_return+0x232/0x79d
kernel: [<ffffffff8126a4d9>] ? cpumask_next_and+0x29/0x50
kernel: [<ffffffff813e9c05>] ? md_super_wait+0x55/0x90
kernel: [<ffffffff81090a10>] ? autoremove_wake_function+0x0/0x40
kernel: [<ffffffff813ebf46>] ? md_update_sb+0x206/0x3f0
kernel: [<ffffffff813ee922>] ? md_check_recovery+0x3f2/0x6d0
kernel: [<ffffffffa005b129>] ? raid1d+0x49/0x1050 [raid1]
kernel: [<ffffffff814ed985>] ? schedule_timeout+0x215/0x2e0
kernel: [<ffffffff814ef447>] ? _spin_unlock_irqrestore+0x17/0x20
kernel: [<ffffffff813eb336>] ? md_thread+0x116/0x150
kernel: [<ffffffff81090a10>] ? autoremove_wake_function+0x0/0x40
kernel: [<ffffffff813eb220>] ? md_thread+0x0/0x150
kernel: [<ffffffff810906a6>] ? kthread+0x96/0xa0
kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
kernel: [<ffffffff81090610>] ? kthread+0x0/0xa0
kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20

And something is wrong with the Named/BIND resulting in the same error for all domains:

zone DOMAINEXAMPLE.com/IN: loading from master file DOMAINEXAMPLE.com failed: file not found
zone DOMAINEXAMPLE.com/IN: not loaded due to errors.
_default/DOMAINEXAMPLE.com/IN: file not found

I'm pretty sure this is not enough information to solve the problem, but I'm willing to engage someone who can work this out for me.

Any help would be greatly appreciated.

share|improve this question
add comment

1 Answer

None of the errors you paste explain anything about 403 errors, those can be found in /var/log/httpd on standard centos systems.

The bind error means that someone messed up your bind config quite a bit. What are you using bind for? If it's just for local caching/resolving, temporarily using 8.8.8.8 as resolver works around that.

All in all, you've got quite a mess on your hands and without a bit more detail (such as apache logs) it'll be hard for us to provide more help. Maybe it's time to get a different contractor in, preferably someone who knows what he's doing.

share|improve this answer
    
+1 for the recommendation to call in another consultant. It sounds like there are multiple things messed up here, some of which may be related to the firewall work, some possibly not related. It's really going to take someone that knows what they're doing to get their proverbial hands on that server and fix things up. –  EEAA Dec 2 '12 at 16:04
    
Many thanks. I sort of figured it was messed up pretty bad. –  Craig Dec 3 '12 at 15:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.