5

We have been using PowerBroker Identity Services Open to authenticate active directory users from ubuntu hosts successfully for six months.

Recently AD authentication stopped working on several workstations after users performed an apt-get upgrade of 200+ packages at once. Authentication attempts give the errors, "invalid password," "user account has expired," or "is your account locked?"

I have not been able to link the issue to a specific package upgrade but workstations build from scratch with the same package versions do not experience the issue. I have tried reinstalling PBIS and validated all the config files but I'm missing something.... I'm at a loss and would love any advice anyone has. I'd rather not have to rebuild another box the next time this happens!

Authentication Attempts

I first verified that the the AD user account was enabled, was not locked and had not expired. Local user authentication works fine through lightdm and ssh.

  1. lightdm

    • valid credentials
      • error returned to user "Invalid password, please try again."
      • auth.log: nothing
      • syslog: nothing
    • incorrect password

      • error returned to user "Invalid password, please try again."
      • auth.log:

        lightdm: [lsass-pam] [module:pam_lsass]pam_sm_authenticate error [login:username][error code:40022]
        
      • syslog:

        lsass: [LwKrb5GetTgtImpl /builder/src-buildserver/Platform-8.0/src/linux/lwadvapi/threaded/krbtgt.c:276] KRB5 Error code: -1765328360 (Message: Preauthentication failed)
        lsass: [lsass] Failed to authenticate user (name = 'username') -> error = 40022, symbol = LW_ERROR_PASSWORD_MISMATCH, client pid = 17768
        
  2. ssh

    • valid credentials

      • ssh disconnect with "Connection closed by IP_ADDRESS."
      • auth.log:

        sshd[18237]: error: PAM: User account has expired for DOMAIN\\USER from HOSTNAME
        sshd[18237]: error: Received disconnect from IP_ADDRESS: 13: Unable to authenticate [preauth]
        
      • syslog: nothing
    • incorrect password

      • ssh disconnect with "Connection closed by IP_ADDRESS."
      • auth.log:

        sshd[18276]: [lsass-pam] [module:pam_lsass]pam_sm_authenticate error [login:domain\username][error code:40022]
        sshd[18272]: error: PAM: Authentication failure for domain\\username from hostname
        
      • syslog

        lsass: [LwKrb5GetTgtImpl /builder/src-buildserver/Platform-8.0/src/linux/lwadvapi/threaded/krbtgt.c:276] KRB5 Error code: -1765328360 (Message: Preauthentication failed)
        lsass: [lsass] Failed to authenticate user (name = 'domain\username') -> error = 40022, symbol = LW_ERROR_PASSWORD_MISMATCH, client pid = 18276
        
  3. Just trying some crazy local stuff (and no, the account is not locked in AD)

            root@hostname:~# su - domain\\username
            su: Authentication failure
            (Ignored)
            reenter password for pam_mount:
            DOMAIN\username@hostname:~$ sudo cat /etc/fstab
            [sudo] password for DOMAIN\username:
            sudo: account validation failure, is your account locked?
            DOMAIN\username@hostname:~$
    

Configuration

  • Ubuntu 14.04
  • PBIS Open 8.0.1.2029 (pbis-open-8.0.1.2029.linux.x86_64.deb.sh)
  • /opt/pbis/bin/config --dump

    AllowDeleteTo ""  
    AllowReadTo ""  
    AllowWriteTo ""
    MaxDiskUsage 104857600
    MaxEventLifespan 90
    MaxNumEvents 100000
    DomainSeparator "\\"
    SpaceReplacement "^"
    EnableEventlog false
    Providers "ActiveDirectory"
    DisplayMotd false
    PAMLogLevel "error"
    UserNotAllowedError "Access denied"
    AssumeDefaultDomain true
    CreateHomeDir true
    CreateK5Login true
    SyncSystemTime true
    TrimUserMembership true
    LdapSignAndSeal false
    LogADNetworkConnectionEvents true
    NssEnumerationEnabled true
    NssGroupMembersQueryCacheOnly true
    NssUserMembershipQueryCacheOnly false
    RefreshUserCredentials true
    CacheEntryExpiry 14400
    DomainManagerCheckDomainOnlineInterval 300
    DomainManagerUnknownDomainCacheTimeout 3600
    MachinePasswordLifespan 2592000
    MemoryCacheSizeCap 0
    HomeDirPrefix "/home"
    HomeDirTemplate "%H/%D/%U"
    RemoteHomeDirTemplate ""
    HomeDirUmask "022"
    LoginShellTemplate "/bin/bash"
    SkeletonDirs "/etc/skel"
    UserDomainPrefix "DOMAIN.COM"
    DomainManagerIgnoreAllTrusts false
    DomainManagerIncludeTrustsList
    DomainManagerExcludeTrustsList
    RequireMembershipOf "DOMAIN\\DOMAIN-GROUP"
    Local_AcceptNTLMv1 true
    Local_HomeDirTemplate "%H/local/%D/%U"
    Local_HomeDirUmask "022"
    Local_LoginShellTemplate "/bin/sh"
    Local_SkeletonDirs "/etc/skel"
    UserMonitorCheckInterval 1800
    LsassAutostart true
    EventlogAutostart true
    
  • /opt/pbis/bin/get-status

    LSA Server Status:
    
    Compiled daemon version: 8.0.1.2029
    Packaged product version: 8.0.2029.67662
    Uptime:        1 days 1 hours 4 minutes 26 seconds
    
    [Authentication provider: lsa-activedirectory-provider]
    
            Status:        Online
            Mode:          Un-provisioned
            Domain:        DOMAIN.COM
            Domain SID:    S-1-5-21-3537566271-1428921453-776812789
            Forest:        domain.com
            Site:          NYC
            Online check interval:  300 seconds
            [Trusted Domains: 1]
    
    
            [Domain: DOMAIN]
    
                    DNS Domain:       domain.com
                    Netbios name:     DOMAIN
                    Forest name:      domain.com
                    Trustee DNS name:
                    Client site name: NYC
                    Domain SID:       S-1-5-21-3537566271-1428921453-776812789
                    Domain GUID:      0b6b6d88-ea48-314a-8bad-a997a57bc1f4
                    Trust Flags:      [0x001d]
                                      [0x0001 - In forest]
                                      [0x0004 - Tree root]
                                      [0x0008 - Primary]
                                      [0x0010 - Native]
                    Trust type:       Up Level
                    Trust Attributes: [0x0000]
                    Trust Direction:  Primary Domain
                    Trust Mode:       In my forest Trust (MFT)
                    Domain flags:     [0x0001]
                                      [0x0001 - Primary]
    
                    [Domain Controller (DC) Information]
    
                            DC Name:              dc2.nyc.domain.com
                            DC Address:           10.x.x.50
                            DC Site:              NYC
                            DC Flags:             [0x0000f1fc]
                            DC Is PDC:            no
                            DC is time server:    yes
                            DC has writeable DS:  yes
                            DC is Global Catalog: yes
                            DC is running KDC:    yes
    
                    [Global Catalog (GC) Information]
    
                            GC Name:              dc1.nyc.domain.com
                            GC Address:           10.x.x.50
                            GC Site:              NYC
                            GC Flags:             [0x0000f3fd]
                            GC Is PDC:            yes
                            GC is time server:    yes
                            GC has writeable DS:  yes
                            GC is running KDC:    yes
    
  • /opt/pbis/bin/find-objects --user USERNAME

    User object [1 of 1] (S-1-5-21-3537566271-1428921453-776812789-1107)
    ============
    Enabled: yes
    Distinguished name: CN=USERNAME,OU=User,OU=User Accounts,DC=domain,DC=com
    SAM account name: username
    NetBIOS domain name: DOMAIN
    UPN: username@DOMAIN.COM
    Display Name: First Last
    Alias: <null>
    UNIX name: DOMAIN\username
    GECOS: First LAst
    Shell: /bin/bash
    Home directory: /home/DOMAIN/username
    Windows home directory: \\domain.com\dfs\NYC\Users\username
    Local windows home directory:
    UID: 1023411283
    Primary group SID: S-1-5-21-3537566271-1428921453-776812789-513
    Primary GID: 1023410689
    Password expired: no
    Password never expires: yes
    Change password on next logon: no
    User can change password: yes
    Account disabled: no
    Account expired: no
    Account locked: no    
    
  • /etc/pbis/pbis-krb5-ad.conf

    [libdefaults]
        default_tgs_enctypes = AES256-CTS AES128-CTS RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
        default_tkt_enctypes = AES256-CTS AES128-CTS RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
        preferred_enctypes = AES256-CTS AES128-CTS RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
        dns_lookup_kdc = true
        pkinit_kdc_hostname = <DNS>
        pkinit_anchors = DIR:/var/lib/pbis/trusted_certs
        pkinit_cert_match = &&<EKU>msScLogin<PRINCIPAL>
        pkinit_eku_checking = kpServerAuth
        pkinit_win2k_require_binding = false
        pkinit_identities = PKCS11:/opt/pbis/lib/libpkcs11.so
    
  • /etc/pam.d/common-session

    session [default=1]                     pam_permit.so
    session requisite                       pam_deny.so
    session required                        pam_permit.so
    session optional                        pam_umask.so
    session required                        pam_unix.so
    session optional                        pam_mount.so
    session [success=ok default=ignore]     pam_lsass.so
    session optional                        pam_systemd.so
    
  • /etc/pam.d/common-auth

    auth    [success=2 default=ignore]      pam_unix.so nullok_secure
    auth    [success=1 default=ignore]      pam_lsass.so try_first_pass
    auth    requisite                       pam_deny.so
    auth    required                        pam_permit.so
    auth    optional                        pam_cap.so
    auth    optional                        pam_mount.so
    
  • /opt/pbis/share/pbis.pam-auth-update

    Name: Likewise
    Default: yes
    Priority: 250
    Conflicts: winbind
    Auth-Type: Primary
    Auth:
            [success=end default=ignore]    pam_lsass.so try_first_pass
    Auth-Initial:
            [success=end default=ignore]    pam_lsass.so
    Account-Type: Primary
    Account:
            [success=ok new_authtok_reqd=ok default=ignore]         pam_lsass.so unknown_ok
            [success=end new_authtok_reqd=done default=ignore]      pam_lsass.so
    Session-Type: Additional
    Session:
            sufficient      pam_lsass.so
    Password-Type: Primary
    Password:
            [success=end default=ignore]    pam_lsass.so use_authtok try_first_pass
    Password-Initial:
            [success=end default=ignore]    pam_lsass.so
    
  • /usr/share/lightdm/lightdm.conf.d/50-ubuntu.conf

    [SeatDefaults]
    user-session=ubuntu
    greeter-show-manual-login=true
    
  • /usr/share/lightdm/lightdm.conf.d/50-unity-greeter.conf

    [SeatDefaults]
    allow-guest=false
    greeter-show-remote-login=false
    greeter-show-manual-login=true
    greeter-session=unity-greeter
    
3

The key line is this:

sshd[18237]: error: PAM: User account has expired for DOMAIN\\USER from HOSTNAME

This indicates that a PAM module believes the account has expired. I'd focus less on auth/session and more on account, which is the facility focused on account properties not related to authentication. Your first task is to identify the module causing the problem. Once you know that, it should be much easier to identify why the module thinks the user should be blocked.

Review the applicable account modules one by one, and try adding the debug flag to individual entries to expand the logging output if you need more hints. If truly stumped and it wouldn't violate the security of a critical environment, you can also try commenting the account lines one at a time until you identify your culprit.

As for what changed, more than likely your PAM config was modified when these packages were installed. Chances are that the users in question were in this state all along, but the database associated with the misbehaving account module was being bypassed. (skipped, commented, not present at all, etc.)

  • I've been banking my head against a wall for weeks on this -- thank you very, very much @Andrew B! I compared /etc/pam.d/common-account on a working machine with the non working machine and both pam_lsass.so entries were missing. For the record, here are the two lines that were missing from common-account on the system: account [success=ok new_authtok_reqd=ok default=ignore] pam_lsass.so unknown_ok AND account [success=1 new_authtok_reqd=done default=ignore] pam_lsass.so – Peter M Sep 25 '14 at 13:34
2

FYI: domainjoin-cli configure --enable pam will re-add these lines after an upgrade as well. PBIS Open 8.x and higher properly deliver a /usr/share/pam-configs/pbis configuration so that this shouldn't happen in the future.

Additionally, PBIS logs more specific errors to the daemon facility of syslog, so you can view them in ubuntu in /var/log/syslog rather than /var/log/secure.

1

Thanks again to @Andrew B for helping me find the solution.

For the record, here is a copy of the correct /etc/pam.d/common-account configuration that FIXES my issue (the two pam_lsass.so lines were missing from the systems that were not working):

 account [success=3 new_authtok_reqd=done default=ignore]        pam_unix.so
 account [success=ok new_authtok_reqd=ok default=ignore]         pam_lsass.so unknown_ok
 account [success=1 new_authtok_reqd=done default=ignore]        pam_lsass.so
 account requisite                       pam_deny.so
 account required                        pam_permit.so
  • Ooof. Yeah, that pam_unix.so line was bad news. success=3 means "skip over the next 3 lines if successful", and without those next two lines you're skipping past the end of the file. Hard to interpret what happens next without seeing your full account stack, but you'd definitely be skipping past 1 line in another file or the end of the stack entirely. – Andrew B Sep 25 '14 at 13:58
0

I experienced this problem on one of our Ubuntu 14.04.4 servers for one user. The user had an ø in the display name in AD. When I replaced the ø with o, he was able to log in. Why this was a problem on only one server, I haven't figuered out yet.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.