5

I am working on setting up authentication into an Acme Packet Net-Net 3820 (SBC) via RADIUS. The accounting side of things is working just fine with no issues. The authentication side of things is another matter. I can see from a packet capture that the access-request messages are in fact getting to the RADIUS server at which point the RADIUS server starts communicating with the domain controllers. I then see the chain of communication going back to the RADIUS and then finally back to the SBC. The problem is the response I get back is always an access-reject message with a reason code of 16 (Authentication failed due to a user credentials mismatch. Either the user name provided does not match an existing user account or the password was incorrect). This is confirmed by looking at the security event logs where I can see events 4625 and 6273. See the events below (Note: The names and IPs have been changed to protect the innocent):

Event ID: 6273


Network Policy Server denied access to a user.

Contact the Network Policy Server administrator for more information.

User:
    Security ID:            NULL SID
    Account Name:           real_username
    Account Domain:         real_domain
    Fully Qualified Account Name:   real_domain\real_username

Client Machine:
    Security ID:            NULL SID
    Account Name:           -
    Fully Qualified Account Name:   -
    OS-Version:         -
    Called Station Identifier:      -
    Calling Station Identifier:     -

NAS:
    NAS IPv4 Address:       10.0.0.10
    NAS IPv6 Address:       -
    NAS Identifier:         radius1.real_domain
    NAS Port-Type:          -
    NAS Port:           101451540

RADIUS Client:
    Client Friendly Name:       sbc1mgmt
    Client IP Address:          10.0.0.10

Authentication Details:
    Connection Request Policy Name: SBC Authentication
    Network Policy Name:        -
    Authentication Provider:        Windows
    Authentication Server:      RADIUS1.real_domain
    Authentication Type:        MS-CHAPv2
    EAP Type:           -
    Account Session Identifier:     -
    Logging Results:            Accounting information was written to the SQL data store and the local log file.
    Reason Code:            16
    Reason:             Authentication failed due to a user credentials mismatch. Either the user name provided does not map to an existing user account or the password was incorrect.

Event ID: 4625


An account failed to log on.

Subject:
    Security ID:        SYSTEM
    Account Name:       RADIUS1$
    Account Domain:     REAL_DOMAIN
    Logon ID:       0x3E7

Logon Type:         3

Account For Which Logon Failed:
    Security ID:        NULL SID
    Account Name:       real_username
    Account Domain:     REAL_DOMAIN

Failure Information:
    Failure Reason:     Unknown user name or bad password.
    Status:         0xC000006D
    Sub Status:     0xC000006A

Process Information:
    Caller Process ID:  0x2cc
    Caller Process Name:    C:\Windows\System32\svchost.exe

Network Information:
    Workstation Name:   
    Source Network Address: -
    Source Port:        -

Detailed Authentication Information:
    Logon Process:      IAS
    Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
    Transited Services: -
    Package Name (NTLM only):   -
    Key Length:     0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
    - Transited services indicate which intermediate services have participated in this logon request.
    - Package name indicates which sub-protocol was used among the NTLM protocols.
    - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

So at first glance it would seem that the issue is merely a case of an invalid username or mismatched password. This is further confirmed in the packet capture where I can see the MSCHAPv2 response has an error code of 691 (Access denied because username or password, or both, are not valid on the domain). The thing is I know I am using a valid username and I have tried many usernames including new ones I created just for troubleshooting. I don't know how many times I have reset the password in an attempt to ensure it is not a mismatch password. I have even made sure to use passwords that are fairly short and contain only letters to ensure there was no terminal encoding issues (we connect to the SBC via SSH clients). I have also done this same thing with the shared secret used during communication between the SBC and the RADIUS server. I have tried prefixing the username with the domain name at login (though I don't think that should be necessary). I have also tried using the full UPN of the user to login. I have tried several RADIUS testing clients (NTRadPing, RadiusTest, etc.), but they either don't support MSCHAPv2 or only support EAP-MSCHAPv2. I have even created my own client using PHP's PECL RADIUS module. Still it always seems to fail with the MSCHAPv2 authentication with an error code of 691. Does anyone have any ideas as to why I always get an invalid username or bad password response when I have done everything possible to ensure that is not the case?

Here are the specs for our RADIUS configuration:

  • Windows Server 2012 R2
  • SQL Server 2012 Back End Database for accounting.
  • The server has been authorized on the domain and is a member of the "RAS and IAS Servers" group. For which that group does have access to the accounts we are testing with.
  • The accounts we are testing with do have the "Control access through NPS Network Policy" option checked under their "Dial-in" property tab.
  • RADIUS clients configured to simply match on the IP address which you can see from the events above that it is applying the client friendly name.
  • Connection Request Policy: The "SBC Authenication" policy is being applied as seen above. The only condition is a regex expression that does successfully match the friendly name.
  • Network Policy: As seen in events above, none are getting applied. For troubleshooting purposes I have created a Network Policy that is set to "1" for the processing order and its only condition is a Day and Time Restriction currently set to any time, any day.
  • The authentication method is set to only MSCHAPv2 or MSCHAPv2 (User can change password after it has expired). I have tried adding this to just the Network Policy and I have also tried adding this to the Connection Request Policy and setting it to override the authentication method of the Network Policy.
  • We do have other RADIUS servers in our domain that use PEAP to authenticate wireless clients and they all work fine. However, we need this to work with MSCHAPv2 only (No EAP).
  • All other configurations are set to the defaults.

The only other things of note to consider is the fact that in the events above you can see that the Security ID is "NULL SID". Now I know this is common especially among failed logons but given that this issue is stating an invalid username or bad password, perhaps it matters in this case. Also, this server has been rebuilt using the same computer account in Active Directory. I do not know if it would have worked before the rebuild. Essentially we built this server and only got as far as authorizing the server to the domain and adding SQL when we decided to separate out the SQL role onto another server. Rather than uninstalling SQL we just rebuilt the machine. However, before reinstalling Windows I did do a reset on the computer account. I don't think this should matter but thought I would point it out if there is some weird quirk where reusing the same SID of a previously authorized NPS server would cause an issue.

All in all it is a fairly basic setup and hopefully I have provided enough information for someone to get an idea of what might be going on. Apologies if my understanding of this seems a bit basic, after all, when it comes to RADIUS servers I guess you could say I'm the new guy here.

Edit 1:

In an attempt to further troubleshoot this issue I have tried bringing up additional servers for testing. Here are the additional tests I have performed.

Multiple Domains

I have now tried this in 3 different isolated domains. Both our test and production domains as well as my private home domain which has very little in the way of customizations aside from the modifications made for Exchange and ConfigMgr. All have the same results described above.

VPN Service

Using Windows Server 2012 R2 we brought up a separate server to run a standard VPN setup. The intent was to see if we could use RADIUS authentication with the VPN and if that worked we would know the issue is with the SBCs. However, before we could even configure it to use RADIUS we just attempted to make sure it worked with standard Windows Authentication on the local VPN server. Interestingly, it too fails with the same events getting logged as the RADIUS servers. The client machine being a Windows 8.1 workstation. Again I point out that we have working RADIUS servers used specifically for our wireless environment. The only difference between those RADIUS servers and the ones I am having problems with is that the working wireless servers are using PEAP instead of MSCHAPv2.

FreeRADIUS

Now I'm no Linux guru but I believe I have it up and running. I am able to use ntlm_auth to authenticate users when logged on to the console. However, when the radiusd service tries to use ntlm_auth to do essentially the same thing it fails and returns the same message I've been getting with the Windows server (E=691). I have the radiusd service running in debug mode so I can see more of what is going on. I can post the debug info I am getting if requested. The lines I am seeing of particular interest however is as follows:

(1) ERROR: mschap : Program returned code (1) and output 'Logon failure (0xc000006d)'
(1) mschap : External script failed.
(1) ERROR: mschap : External script says: Logon Failure (0xc000006d)
(1) ERROR: mschap : MS-CHAP2-Response is incorrect

The thing to note here is that while we are essentially still getting a "wrong password" message, the actual status code (0xc000006d) is slightly different than what I was getting on the Windows Servers which was (0xc000006a). From this document you can see what these codes mean: NTSTATUS values . The good thing about this FreeRADIUS server is that I can see all of the challenge responses when it is in debug mode. So if I can wrap my head around how a MSCHAPv2 response is computed I can compare it to see if this is simply a miscomputed challenge response. Update Was just noticing that the 6a code is just the sub-status code for the 6d code. So nothing different from the Windows Servers, I still wonder if there is a computation error with the challenge responses though.

Currently, I am working on bringing up a Windows Server 2008 R2 instance of a RADIUS server to see if that helps at all. However, I would be surprised if something with the service broke between W2K8 R2 and W2K12 R2 without anyone noticing until now. If this doesn't work I may have to open a case with Microsoft. Update: Same results with W2K8 R2.

Update 2

I opened a case with Microsoft and they worked on it for a couple of weeks. The first week I spent my time just trying to get them to understand this has nothing to with wireless and that the device we are trying to connect to does not support authentication against a RADIUS server using any form of EAP. Once they finally understood that we are trying to setup the authentication method as just MSCHAPv2 only, his initial reaction was simply "you can't do that". He said, that no matter what you always need some form of EAP or PEAP setup in the top box (even for PAP and other "Less secure methods" in that authentication method window). This seemed highly unlikely to me so I asked for some documentation stating this for which he then change the subject and never provided any documentation. It became clear I was getting nowhere with Microsoft when they told me that the issue seems to be with the username and password. So it took them 2 weeks to come to that conclusion when the subject line of my premiere ticket had the E691 error code in it which means mismatched username and password and despite the very lengthy paragraph of me explaining that I have done everything possible to ensure it is not a bad username and password.

Unfortunately my project lead does not want to waste anymore premiere support hours on this so we have closed the case. We will likely just deal with the SBC local shared login and setup some other way to enforce accountability (there is only 4 of us that will have access).

I will point out that before I was told to drop the project I did get the FreeRADIUS server to work using MSCHAPv2 only but was only able to do this using accounts local to the FreeRADIUS server. That is plain text passwords stored in a FreeRADIUS config file somewhere. So obviously not a solution but it at least shows that the SBC is correctly communicating to a RADIUS server via MSCHAPv2. This and the other things I have mentioned above lead me to believe that the issue lies between the RADIUS server and the Domain Controllers. While NTLM authentication works fine on both the Windows RADIUS and FreeRADIUS servers while logged into the servers locally (Can login to the Windows RADIUS via the test account and can get successful authentication on the FreeRADIUS server when using ntlm_auth command with just a username and password), neither RADIUS server seems to authenticate via users when coming in as a RADIUS access request (FreeRADIUS uses the same ntlm_auth command I use when logged in locally but instead of providing a username and password to the command it provides a username and challenge response).

So I will end this thread here but I will keep an eye on this and it will notify me if anyone posts something. If someone posts a solution or has comments I will respond.

  • 1
    I'm impressed w/ your troubleshooting so far and in your description of the issue and the environment. I don't know that I can get you an answer, but I wanted you to know that you're definitely going about things in the right way. Bravo! – Evan Anderson Jun 26 '14 at 18:59
  • Can you set the Acme device to run PAP? (It would appear that the "authentication-methods" command under "security/authentication/radius-servers" will let you do that.) That'll put the password into an encoded plaintext on the wire, where you can at least see if the Acme is doing something dumb to the password (munging it up somehow). – Evan Anderson Jun 26 '14 at 19:12
  • @EvanAnderson Right now our SBCs are still have a few unrelated components that are being configured by the vendor consultant. So at the moment we temporarily only have read access to the configuration of the SBCs and the vendor consultant is out on vacation until next week. When he gets back we can have him try that and I will let you know what I see in the packet capture. I can say that with the ability to read the config I have checked multiple times and saw that the password was correct but yes, would be good to test whether it is being passed to the server as seen in the config. – New Guy Jun 26 '14 at 19:56
  • The statement about a password "in the config" confuses me. The configuration will contain the RADIUS secret, but the password we're talking about would be the password supplied by a user attempting to authenticate to the SBC device. – Evan Anderson Jun 26 '14 at 19:59
  • My comment was poorly worded....I will do a packet capture to get the username/password combo from the test account attempting to authenticate. What I was referring to was that I had read somewhere that you can get the MSCHAPv2 E=691 message when the issue is actually a bad or malformed shared secret. I was just adding that it looked good in the config. – New Guy Jun 26 '14 at 22:39
2

Solution

Wouldn't you know it, about a month after we had abandoned this approach to using a RADIUS server to control access to the SBC we find a possible solution. Of course at this point we are too busy with other projects to go back and try this solution out so I can't say 100% if this would fix the problem but once I explain you will probably agree this is the answer.

One of my colleagues was at a Microsoft conference having various discussions when it dawned on him that MSCHAPv2 relies on NTLM to generate the password challenges and responses. Now plain old MSCHAP and MSCHAPv2 (i.e. not EAP-MSCHAPv2 or PEAP) when used in Windows RAS services will use NTLMv1 by default.

As many of of you have already started to catch on, we, like many administrators, have disabled NTLMv1 on our DCs and as such the DCs will only accept NTLMv2 requests. This explains why the failure I continued to get was a "bad password" error. The password being sent to the DCs was in NTLMv1 format and was getting ignored.

Once we realized this, I was able to do some more research and I found the following article:

http://support.microsoft.com/kb/2811487

This article describes the same behavior I was experiencing including the E=691 error code I have mentioned. This article also provides a workaround to force RAS services to use NTMLv2 when building a MSCHAPv2 response. Funny how easy it is to find these articles after you know precisely what the issue is.

Again, I have yet to verify this is the issue as I haven't had time to rebuild the RADIUS servers that I have already decommissioned but I do plan on trying this out sometime if I get a chance. I just wanted to post this possible solution in case someone else stumbles across this issue. If anyone can confirm this before I do please let me know.

Edit - Confirmed

We were asked to take on a bigger role with these SBCs and as such we came back to this project and brought up a Windows RADIUS server again. This time we applied the registry key described in the link above. After taking a packet capture of the communication between the RADIUS server and the SBCs I can in fact now see "Access Accept" messages getting fired back at the SBCs. So I can now confirm at least in our scenario that the issue we were having is as described above.

TL;DR

NTLMv1 was disabled on the DCs. Setting a registry key to force the RADIUS server to use NTLMv2 fixed the issue.

0

From my experience MS NPS tells you if RADIUS pre-shared key doesn't match. Something like this:

Event 14: A RADIUS message was received from RADIUS client x.x.x.x with an invalid authenticator. This is typically caused by mismatched shared secrets. Verify the configuration of the shared secret for the RADIUS client in the Network Policy Server snap-in and the configuration of the network access server.

You'll get it in the event log.

That fact you see Access-Reject in your captured traffic also means that pre-shared key matches on both sides otherwise NPS just wouldn't reply and you would see no further RADIUS packets after Access-request.

What is also important in the logs you provided is:

Sub Status:     0xC000006A

This means that "user name is correct but the password is wrong". It's pretty explicit on my view.

I know that MS-CHAPv2 doesn't require to store password using reversible encryption but just to check this guess could you set this checkbox for the user account you use, re-set its password after that and try it again.

  • I would have to agree with you, I was a bit skeptical about the shared secret being the issue as well. My comment above was based on forum posts from people who swear that is what fixed their issue. I only included it in this post as I am starting to grasp at straws here to get this working. – New Guy Jun 30 '14 at 13:26
  • I did try setting the reversible password option for the test account and then reset the password on it. Got the same E=691 error message. I also tried MSCHAPv1 and CHAP via my php "client" and it failed with the same error. That said, I don't know for certain I implemented my php client properly so that may not mean anything. Your sub status comment should help. I verified this by attempting to login with a username I know doesn't exist and the sub status code changed to 0xC0000064. So at least I know it is the password it is failing on. Now to figure out why. – New Guy Jun 30 '14 at 14:28
  • I should clarify, the above test with CHAP returned a "Reason Code: 19". Essentially stating that it could not find a reversible password for the account despite the fact that I had enabled it on the account, waited for replication, and then reset the password to make sure there was a reversible password stored with the account. – New Guy Jun 30 '14 at 14:36
  • Your last note is interesting. You enabled reversible encryption and re-set password but NPS says it can not find password encrypted with reversible encryption. It appears that your NPS uses wrong/out of sync AD server as backend. – Max Doronin Jun 30 '14 at 23:56
  • I find that odd as well. It is definitely using the correct DCs, confirmed this in the packet captures. I did wait a good while after setting the option to reset the password. I supposed I could try again and just force AD replication to ensure it has updated everything. Though I do wonder if there is perhaps a default domain policy of some kind that prevents this insecure method of authentication, I haven't found any to that affect myself but I could be missing something. – New Guy Jul 1 '14 at 15:30
0

FYI - After upgrading a working RADIUS + MSCHAP system, I was getting the same error.

"\000E=691 R=1"

Turns out there was a samba change of some sort that threw off the rights to the winbind socket. Adding freerad to the winbindd_priv group fixed the issue.

/etc/group: winbindd_priv:x:110:freerad

  • Yes, that was a common problem I came across when doing research on the issue. However, in my case all the proper permissions were setup correctly and in addition to that the main issue was trying to get this to work with a Microsoft RADIUS server. That said, we recently discovered what may actually be the problem in our case. I haven't had a time to verify it yet but once I do I had planned on coming back here to post the answer. – New Guy Oct 30 '14 at 13:37
0

We had the same problem with authentication from our NPS servers after adding new 2012R2 domain controllers to an existing 2008R2 environment. The new 2012R2 domain controllers had NTLMv1 disabled where the 2008R2 domain controllers had it enabled.

The NPS servers (running 2008R2) where randomly denying access for users.

The following event was logged on the NPS servers: Event ID 6273 (Security log) Network policy server denied access to a user. Reason code: 16 Reason: Authentication failed due to a user credentials mismatch. Either the user name provided does not map to an existing user account or the password was incorrect.

When the NPS servers connected to the 2008R2 dc's everything worked like a charm. But on the 2012R2 dc's access was denied. All because NTLMv1 was disabled on the new 2012R2 domain controllers.

I can confirm that the solution in the MS KB article (http://support.microsoft.com/kb/2811487) works perfectly!.

Thank you very much for this excellent article!!! It saved me a huge amount of time in solving this issue ; - )

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.