I'm trying to Kerberize an Apache-server, and allow the created server principal to sign on to the Active Directory. I've followed one of the numerous tutorials available online, and it seems to work fine. I'm on the Linux side of the project, and Corporate IT is on the Windows side.

IT has provided me with a service account and a service principal for it. In this example, I'll refer to it as HTTP/mysite.mycorp.com@MYCORP.COM. They have provided me with a keytab file for said principal, which involves running a tool called ktpass.exe on the AD server.

I've verified that the KVNOs of the AD/KDC and the keytab file match. All is well.

There is a proper DNS A-record for the hostname, and a proper PTR record for the IP. Both servers are in time sync.

I'm able to request a ticket from the AD/KDC for the above mentioned service principal with the issued keytab file, like this:

kinit -k -t http.keytab HTTP/mysite.mycorp.com@MYCORP.COM

This works. I obtain a ticket, and I'm able to use this ticket for things like querying the AD/LDAP directory. The keytab also works great for running a Single Signon Apache site, which is partly the goal of this exercise.

Half an hour passes.

Attempts to log on with the above kinit command now fails with this message:

Client not found in Kerberos database

I'm unable to authenticate as the service principal, much as if the principal was deleted on the AD server.

Now it gets weird, at least for me:

By request, the AD administrator runs the ktpass.exe tool again, building a fresh keytab file for my service. The KVNO (Key Version Number) is incremented on the server, causing our Apache test server to stop validating Kerberos single signon. This is expected with my present configuration. What surprised all of us, was that now the kinit command worked again. We bought ourselves another half hour, and then it stopped working again.

Our IT department is at a loss here, and they're speculating that this is a problem with the AD server itself. I'm thinking it's configuration, but according to them, there are no half hour limits anywhere in their setup.

I've followed http://www.grolmsnet.de/kerbtut/ (see section 7) but the method seems to be the same in all the documentation I've found. I haven't found any reference to time limits on service principals.

EDIT: This seems to be a replication issue. Although no errors are reported in the replication process, the SPN value of the service account is changed (reverted?) from "HTTP/mysite.mycorp.com@MYCORP.COM" to "name-of-service-account@mycorp.com" after 30 minutes.

  • To clarify: after you grab the regenerated keytab, does kinit work again with this or with the old one? Does both SSO and kinit stop working at the same time? What is the output of klist command? I remember running into a similar issue 2 years ago, with Ubuntu 13.04 and Windows Server 2008R2 (in my case working time was more than ~ 30 min. IIRC), but refreshing the keytabs have done the job. – sam_pan_mariusz Aug 30 '15 at 6:11
  • Did you filtered/checked the security's eventlog on the DC to validate there. As I seen you have all log file from the web server, but less from the DC, exept the kinit's error. – yagmoth555 Sep 2 '15 at 20:45
  • Do not forget your bounty. Thank you in advance – Yves Martin Sep 5 '15 at 6:40

Thanks for all your input, guys. We got Microsoft on board, and they helped us debug the authentication process on the AD side. Everything worked as it was supposed to, but failed after thirty minutes.

While we were doing a remote debugging session, on of the participants noticed that the UPN/SPN of the service account was suddenly resat from HTTP/mysite.mycorp.com@MYCORP.COM to service-account@mycorp.com. After a LOT of digging around, including debugging AD replication, we found the culprit:

Someone had made a script that ran periodically (or probably by event, since it was exactly thirty minutes after running ktpass.exe), which updated the UPN/PSN to "ensure cloud connectivity". I do not have any supplemental information on the reasons for doing this.

The script was changed to allow existing UPN/SPN values ending in @mycorp.com, effectively solving the problem.

Tips for debugging issues like this:

  • Ensure all the participants in the authentication supports the same encryption types. Avoid DES - it's outdated and insecure.
  • Make sure to enable AES-128 and AES-256 encryption on the service account
  • Be aware that enabling DES on the service account means "use ONLY DES for this account", even if you enabled any of the AES encryptions. Do a search for UF_USE_DES_KEY_ONLY for details on this.
  • Make sure the UPN/SPN value is currect and matches the value in the issued keytab file (i.e. through an LDAP lookup)
  • Make sure the KVNO (Key Version Number) in the keytab file matches the on on the server
  • Inspect traffic between server and client (i.e. with tcpdump and/or WireShark)
  • Enable debugging of authentication on the AD side - inspect logs
  • Enable debugging of replication on the AD side - inspect logs

Again, thank you for your input.

| improve this answer | |

I also learned Kerberos beginning with Achim Grolms' mod_auth_kerb tutorial, really a good piece of documentation.

The keytab file only replaces the password authentication. Password is encoded in the file and these bytes are used in authentication challenge with KDC. Any password change on the service account will invalidate keytab authentication, and also increase the kvno number.

To confirm a service account SPN is available, I often authenticate with the service account password:

kinit HTTP/mysite.mycorp.com@MYCORP.COM

If failing, to confirm service account is not disabled, try basic authentication:

kinit account

If failing to authenticate, simply delete the account and create a new one with another login name to avoid troubles.

There is a high chance that another piece of software - for instance another system with a old generated keytab for the same SPN - tries to authenticate on this service account and disabled the account because of invalid password.

When setting Kerberos SSO, too fast operations may lead to inconsistencies in Active Directory. My general guideline when stuck in configuration process is to follow these steps:

  • delete "old" or "failing" service accounts for test and production systems
  • check with kvno the SPN you expect to configure no longer exists in realm
  • check with setspn -X there is no conflictual SPN on multiple accounts
  • create one service account per system, dedicated to a single fully qualified SPN, with a brand new login name
  • prevent password change and password expiration on the service account
  • let's wait a while for DC synchronisation
  • set password as ktpass option when generating keytab
  • check FQDN SPN and aliases with setspn -l account

Here is a set of commands to configure service account on DC:

ktpass -princ HTTP/mysite.mycorp.com@MYCORP.COM -mapuser mysiteAccount@MYCORP.COM
  -crypto RC4-HMAC-NT -ptype KRB5_NT_PRINCIPAL
  -pass long!$longp2ass3word -out c:\temp\http-mysite-mycorp-com.keytab
setspn -a HTTP/mysite mysiteAccount
setspn -l mysiteAccount

If operations are done too fast on different DCs between MMC and running ktpass for keytab generation in an administrative command line, then DCs synchronisation may lead to unexpected results like the one you describe. So let's wait a while between account creation and then ktpass and any additional setspn commands.

And commands to run on Linux to check everything works properly:

kinit mysiteAccount@MYCORP.COM
kinit HTTP/mysite.mycorp.com@MYCORP.COM
kinit -k -t http-mysite-mycorp-com.keytab HTTP/mysite.mycorp.com@MYCORP.COM
kvno HTTP/mysite.mycorp.com
kvno HTTP/mysite
| improve this answer | |
  • That's good advice, but the account seems to be disabled (although the AD doesn't indicate this). Running kinit without keytab parameters yield this result: kinit HTTP/mysite.mycorp.com kinit(v5): Client not found in Kerberos database while getting initial credentials – Saustrup Aug 31 '15 at 8:24
  • So you have to check if the service account stills has this FQDN SPN enlisted in its attributes with setspn -l account (or use a LDAP browser if you prefer that path...) – Yves Martin Aug 31 '15 at 16:20
  • If SPN has "disappeared", it means you may have generated keytab on a non-master DC, which received a sync event about this account from master DC, leading to SPN removal. – Yves Martin Aug 31 '15 at 16:24
  • You have to confirm your account is disabled by trying password authentication with kinit account. If it fails, it means another service tries to authenticate your service account with an invalid password or an invalid keytab. The best you have to do is to delete your account and create a new service account with a different login name to avoid such troubles – Yves Martin Sep 1 '15 at 21:19
  • 1
    @YvesMartin in Active Directory there is no such concept as a "master" except in the specific case of the FSMO role operations - none of which effect this circumstance. For almost all operations, AD DS Domain Controllers are multi-master peers. And for the sake of Kerberos, each is a KDC. – MDMarra Sep 2 '15 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.