3

We have a number of servers (application servers, web servers & ftp servers) running red hat 5 that are all virtual. We also have a similar setup that is Windows-based. Yesterday, our infrastructure team needed to shutdown the primary domain controller so they could move the physical server to a new rack. Their assumption was that as soon as the primary domain controller went down, the secondary domain controller would pick up. As soon as the primary domain controller powered down, the Linux-based app servers all slowed to a crawl, to the point that simply trying to log in via ssh took approximately 3 minutes.

Before we could finish troubleshooting the issue, the infrastructure team was able to bring the primary domain controller back on-line.

During the down-time of the primary domain controller, all windows-based servers appeared to be functioning normally.

Our first thought was that the Linux servers didn't have the secondary domain controller listed as a DNS server, but this is not the case. The red hat servers don't tie into any AD functionality, other than using it as a DNS server.

Any thoughts on what else we could check? We're not really Linux sys admins, so I'm not sure if we're missing something pretty basic.

Improve this question
2
  • Are you using LDAP for account management/authentication?
    – voretaq7
    Aug 2, 2011 at 20:05
  • 1
    We are not using LDAP for account management. We have a couple user accounts on each box that we set up manually.
    – Matt
    Aug 9, 2011 at 13:25

3 Answers 3

Reset to default
1

Depends on what you're using for authentication. Sounds like the fail-back mechanisms for whatever you're using were either taking too long, or working too slowly. If you were using LDAP for auth and had a single IP address listed in your configs for checking, then yes what you are seeing is entirely appropriate for that case. If you're using Winbind, it should be smart enough to fail over to another domain controller, but it still could take a while before reaching that decision.

I believe the "only one LDAP server can be listed in the LDAP-auth config" problem has been around for a while. One work-around for that is to make the DNS entry it points to be a round-robin DNS entry between multiple domain controllers. Another possibility, if you have the infrastructure for it, is to host the address on a load-balancer; we did this at my old job and it worked quite well.

Improve this answer
1

Do RHEL servers use DNS as DNS resolvers or they use it to connect to other services? Did you check the logs (e.g. /var/log/messages) on those servers what was going on?

To me it looks like some services on servers are very domain dependent and not resolving those domains resulted in some vigorous attempt to reconnect back to those domains.

You could maybe test this with temporarily suspending the domains that RHEL servers are using.

Improve this answer
1
  • Thanks Martin. I checked the system logs, but found no strange entries during the outage. At this point, our only real option is to bring down the primary controller again and troubleshoot while the issue is happening.
    – Matt
    Aug 5, 2011 at 13:05
1

Try turning off "GSSAPIAuthentication" in the /etc/ssh/sshd_config. I have had a similar problem that was solved this way. Some of the SSO GSSAPI features, I think try and do a reverse lookup that of course would fail if the DNS servers are offline.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .