0

From past 2 days, I'm struggling with a weird issue, found no answers related to my problem, earlier everything was working fine, but suddenly it stopped.

Issue exists only on my client system, from any other system everything works fine. I'm trying to use a proxy say 201.17.10.12:11111 for curl request and it fails with "Connection timed out". wget, ssh also fails from client server. nmap & ping works.

I checked with netstat -nputw after curl request and it shows connection in SYN_SENT state, also analyzed with tcpdump, results are

$ tcpdump host 201.17.10.12 -w packets &

$ curl -x http://201.17.10.12:11111 http://www.example.com
curl: (7) Failed to connect to 201.17.10.12 port 11111: Connection timed out

$ tcpdump -nn -r packets | less

22:04:00.556502 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2550685 ecr 0,nop,wscale 7], length 0
22:04:01.556560 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2550936 ecr 0,nop,wscale 7], length 0
22:04:03.560552 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2551437 ecr 0,nop,wscale 7], length 0
22:04:07.572559 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2552440 ecr 0,nop,wscale 7], length 0
22:04:15.588592 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2554444 ecr 0,nop,wscale 7], length 0
22:04:31.604579 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2558448 ecr 0,nop,wscale 7], length 0
22:05:03.668547 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2566464 ecr 0,nop,wscale 7], length 0
~

But now if check with nmap, it shows a response

$ nmap -sS -Pn -p 11111 -T4 -vv --reason -oN nmap.results 201.17.10.12

Starting Nmap 6.40 ( http://nmap.org ) at 2017-05-18 22:03 IST
Initiating Parallel DNS resolution of 1 host. at 22:03
Completed Parallel DNS resolution of 1 host. at 22:03, 0.41s elapsed
Initiating SYN Stealth Scan at 22:03
Scanning 201.17.10.12 (201.17.10.12) [1 port]
Discovered open port 11111/tcp on 201.17.10.12
Completed SYN Stealth Scan at 22:03, 0.21s elapsed (1 total ports)
Nmap scan report for 201.17.10.12 (201.17.10.12)
Host is up, received user-set (0.16s latency).
Scanned at 2017-05-18 22:03:23 IST for 0s
PORT      STATE SERVICE REASON
11111/tcp open  unknown syn-ack

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.68 seconds
           Raw packets sent: 1 (44B) | Rcvd: 1 (44B)

Firewall & apparmor is not running on both client and proxy server (Ubuntu 14.04). Please help since I'm unable to think now, if nmap/ping are showing that everything is fine then why it is failing to connect when using as proxy and what could i do to resolve this issue?

EDIT 1 curl with -vv shows:

$ curl -vv  -x http://201.17.10.12:11111 http://www.google.com
* Rebuilt URL to: http://www.google.com/
* Hostname was NOT found in DNS cache
*   Trying 201.17.10.12...
* connect to 201.17.10.12 port 11111 failed: Connection timed out
* Failed to connect to 201.17.10.12 port 11111: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to 201.17.10.12 port 11111: Connection timed out

EDIT 2:

I just noticed this is happening with any program i use on client, ssh/wget/curl. Only nmap and ping works. If i try to connect to server to any non-open ports, it immediately exits with message

$ ssh root@201.17.10.12 -p 32
ssh: connect to host 201.17.10.12 port 32: Connection refused

but if try to connect to any open/listening ports it waits till "Connection timed out" Checking port 32 with nmap shows State:Closed, Reason:Reset. and checking open port 22 with nmap shows State:Open, Reason:syn-ack. Means nmap is getting syn-ack response properly, whether other programs are either not receiving response or not able to read response and waits till timeout occurs.

Output of iptables -L -v -n at server

$ iptables -L -v -n

Chain INPUT (policy ACCEPT 2334K packets, 1925M bytes)
 pkts bytes target     prot opt in     out     source               destination 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination 

Chain OUTPUT (policy ACCEPT 2605K packets, 778M bytes)
 pkts bytes target     prot opt in     out     source               destination 

Output of iptables -L -v -n at client

$ iptables -L -v -n
Chain INPUT (policy ACCEPT 7219K packets, 1845M bytes)
 pkts bytes target     prot opt in     out     source               destination 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination 

Chain OUTPUT (policy ACCEPT 8616K packets, 1288M bytes)
 pkts bytes target     prot opt in     out     source               destination 
  • What do the proxy logs say? – Aaron May 18 '17 at 17:49
  • request never reaches out of requesting server when curl proxy is used. Hence, no logs in proxy server. – agaggi May 18 '17 at 17:52
  • You may have to strace -ff -o /dev/shm/curl_trace.log curl ... to see why nmap and curl get receiving different network behavior. I assume you have already used -vv with curl to validate what it is doing. – Aaron May 18 '17 at 20:39
  • I've updated the question with output of curl -vv, please have a look – agaggi May 19 '17 at 13:38
  • That's an odd one. I would expect either iptables owner module or something like apparmor or selinux to be in effect, but you said apparmor is disabled, so I honestly have no idea. I would be digging through strace at this point. – Aaron May 19 '17 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.