Our client has a server and our web apps run on port 443 and 2000. Our client has two groups of IP addresses. One group is for the computers used in the company building. These computers can browse both web apps. Another group of IP addresses is for users connecting remotely via VPN connection. Those users can browse the web app served on port 443 but cannot browse the one on port 2000. They get a timeout error on their browsers.
We know that this is not a software issue because users in the company building can use both apps with no issue. They also allowed traffic to both ports from both IP address groups. When someone connects to the web apps by using the computers inside the building, firewall software shows all requests are served successfully. However, when they try to connect to port 2000 over VPN connection, the request is sent to the server but users cannot get a response. Firewall software shows "Incomplete" text. Notice that this problem happens only if port 2000 is requested over VPN connection. If port 443 is requested over VPN, there is no problem.
We setup a TCP echo server listening on port 2000 on the server and tried connecting to the port 2000 via telnet command over VPN connection. We see that it can connect. However, text sent over telnet is not echoed. If we try the same thing from a computer inside the company building, we can connect to the server over telnet and also text sent over telnet is echoed.
We also ran tcpdump command on the server to listen TCP packages on port 2000. When we connect to the server via telnet, we can see TCP activity from both IP groups. However, when we send a text over telnet, only the ones sent from computers inside the company building are visible in tcpdump output. Messages sent from VPN users' computers do not create a TCP activity on tcpdump.
We spent lots of hours and couldn't figure out the issue. Any help is appreciated.