Hide client certificate dialog in browser, while still allowing client certificates

Question

I have an nginx instance serving a web application over HTTPS. The web application serves both browsers and native mobile apps. The native mobile apps use client certificates as an additional layer of authentication, while the web browser uses only username/password. The browser and mobile apps use different paths, but the same TCP endpoint (IP/port).

The problem is that the web browser pops up a dialog prompting the user to specify a client certificate. It is perfectly fine to just cancel this dialog, as it is configured to be optional in nginx and validation of the certificate is done on the application side. However, this is very confusing to users, and I need to figure out a way of not showing the dialog.

My requirements:

• It would be very hard to have different endpoints (need to be port 443 on both, and allocating a new IP will be difficult). Please do not suggest changing endpoints. I am well aware of this solution and this is plan B if everything else is absolutely impossible, but this will involve lots and lots and lots of IT headache. It is easily worth a couple of days of development to avoid changing endpoints.
• Other than changing endpoint, I have full control over the client code (i.e. mobile app), as well as the server deployment. I could change protocols as necessary, but it needs to support HTTPS in a "normal" browser.
• If it is difficult to have different client side certificate settings per path (which this old thread seems to indicate), I could resort to selection based on user agents instead. This is no security issue, as the application verifies that a certificate was used for the "sensitive" paths.
• If Nginx cannot do it, I would be happy to switch to another reverse proxy that can handle it. (Traefik, Apache, ...?)

Answer 1

EDIT: As pointed out by Krumelur, the TLS protocol allows session re-negotiation to be initiated by the web server, where the client can ask for client certificate if required. Therefore the original answer below isn't valid for the issue.

---

The certificate authentication happens during TLS client-server handshake, and HTTP only runs once the handshake has been performed.

The user-agent string is part of the HTTP request the client sends over the TLS tunnel. It is not known to the server before the TLS tunnel has been established and client has sent its first HTTP request.

So, the only option is to work with TLS. Now, looking at RFC 5246 chapter 7.4, we can see the specification of the Hello messages, which initiate the TLS handshake.

Client Hello message contains:

• current time
• 28 random bytes
• Cipher suite
• A list of compression methods supported
• Possible TLS extensions

The server sends then back its own Hello message, and Server certificate immediately after the Hello message. Then, it sends a Server Key Exchange Message, and then Server Certificate Request if configured.

Now we can see that the only information client sends does not include any information on the client type. The only extra information given there is the TLS extension part, and the list of current TLS extensions is at https://www.iana.org/assignments/tls-extensiontype-values/tls-extensiontype-values.xhtml.

Looking at the list, there is no client type information even in TLS extensions. Even if there was such an extension, all mobile or all desktop clients must support that extension before the system would work correctly.

Therefore the conclusion is that you cannot accomplish your goal of using client certificates on mobile and not on desktop, because the server cannot know when to require a client certificate.

Your only option is to use SNI as suggested by @Alexey Ten