1

I'm having this problem where when I start up nginx it starts sending/receiving traffic at a rate of 1 GET request every second from nginx on port 80 to /ping, but I can't track down what is causing it. Any ideas on how to track down what process is causing this traffic?

Log samples: /tmp/nginx.error.log

2011/03/31 21:30:33 [error] 6587#0: *7213 connect() to unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 10.252.27.111, server: myhost.com, request: "GET /ping HTTP/1.1", upstream: "http://unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock:/ping", host: "newcaprica11"
2011/03/31 21:30:33 [error] 6587#0: *7215 connect() to unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 10.204.203.236, server: myhost.com, request: "GET /ping HTTP/1.1", upstream: "http://unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock:/ping", host: "newcaprica11"
2011/03/31 21:30:34 [error] 6587#0: *7217 connect() to unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 10.252.27.111, server: myhost.com, request: "GET /ping HTTP/1.1", upstream: "http://unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock:/ping", host: "newcaprica11"
2011/03/31 21:30:34 [error] 6587#0: *7219 connect() to unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 10.204.203.236, server: myhost.com, request: "GET /ping HTTP/1.1", upstream: "http://unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock:/ping", host: "newcaprica11"

Nginx access Log: nginx.vhost.access.log

10.252.27.111 - - [31/Mar/2011:21:33:23 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"
10.204.203.236 - - [31/Mar/2011:21:33:23 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"
10.252.27.111 - - [31/Mar/2011:21:33:24 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"
10.204.203.236 - - [31/Mar/2011:21:33:24 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"
10.252.27.111 - - [31/Mar/2011:21:33:25 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"

I replaced our real hostname with myhost.com, but the strange thing is that in the error.log there is that host host: "newcaprica11"

More details:

nmap -P0 10.204.203.236

Starting Nmap 5.21 ( http://nmap.org ) at 2011-03-31 21:38 UTC
Nmap scan report for ip-10-204-203-236.ec2.internal (10.204.203.236)
Host is up (0.00042s latency).
Not shown: 994 filtered ports
PORT     STATE  SERVICE
80/tcp   open   http
6667/tcp closed irc
8001/tcp closed unknown
8080/tcp closed http-proxy
9000/tcp closed cslistener
9090/tcp closed zeus-admin

So they are running some sort of webservice on 80

Additionally a telnet reveals:

telnet 10.204.203.236 80
Trying 10.204.203.236...
Connected to 10.204.203.236.
Escape character is '^]'.
GET / HTTP/1.0 

HTTP/1.1 302 Found
Server: Apache/2.2.9 (Debian) Phusion_Passenger/2.2.15
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.15
X-Runtime: 310
Cache-Control: no-cache
Location: http://newcaprica9/neighborhoods
Status: 302
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
Content-Length: 98
Date: Thu, 31 Mar 2011 21:39:16 GMT
X-Varnish: 2154834965
Age: 0
Via: 1.1 varnish
Connection: close

<html><body>You are being <a href="http://newcaprica9/neighborhoods">redirected</a>.</body></html>Connection closed by foreign host.

So clearly its some Rails or Rack app running something shitty.

Improve this question
2
  • How are you deciding that this is a connection FROM nginix, and not a connection TO nginix? This sounds like the kind of thing a server monitor would do. Can you show a log example or something?
    – DerfK
    Mar 31, 2011 at 21:28
  • Thanks @DerfK, I posted some lines from my logs, thanks for your help.
    – JP Silvashy
    Mar 31, 2011 at 21:36

1 Answer 1

Reset to default
2

From the log files, it looks a lot like the servers 10.252.27.111 and 10.204.203.236 are requesting the /ping resource. Nginx is complaining because /ping is configured within Nginx to go and connect to a local socket unicorn.sock and pull data from there to fulfill the request. It's getting a connection refused from the socket so is throwing the 502 error.

In this context, I believe upstream is the backend that nginx is connecting to in order to get the response for the request.

I'm not sure if you've anonymised those IP addresses, or if they really are RFC1918 addresses. Either way, they're the ones doing the requesting.

I think.

Improve this answer
1
  • yup it seems you're right, and since they we're requests from a 10. network (which was AWS in this case), I just iptabled those bitches, because it seems like some weak internal DoS or some type of monitoring thing someone is running.
    – JP Silvashy
    Mar 31, 2011 at 21:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .