1

I have a Debian wheezy server running a few web applications, a MongoDB database and a Redis server behind a NGinx server. Only the NGinx server is public facing and the other services are reverse proxied behind it. This setup has been working perfectly until two days ago where there was a temporary power outage in the datacenter where my server is located. After rebooting and doing regular post-crash maintenance (deleting lock files, repairing DB, etc.) I noticed that NGinx was timing out on every service it proxies. Here are the steps I have taken to try resolve the problem:

  1. Check Logs
    I have checked the logs for every service and everything is clean with no errors (other that NGinx reporting the upstream connection time-out).

  2. Check services are running
    All the processes for the WSGI application, MongoDB, etc. are running and I have also checked netstat:

    # netstat -ntple
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
    tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      0          21730537    1469/nginx      
    tcp        0      0 0.0.0.0:2525            0.0.0.0:*               LISTEN      1000       21730714    1511/python     
    tcp        0      0 0.0.0.0:9090            0.0.0.0:*               LISTEN      1000       21730931    1627/python     
    tcp        0      0 0.0.0.0:2022            0.0.0.0:*               LISTEN      0          21730651    1553/sshd       
    tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      1000       21730885    1624/python     
    tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      104        21730531    1376/mongod     
    tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      105        21730621    1532/redis-server *
    tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      1000       21730731    1500/python     
    tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      0          21730536    1469/nginx      
    tcp6       0      0 :::2022                 :::*                    LISTEN      0          21730654    1553/sshd       
    tcp6       0      0 :::6379                 :::*                    LISTEN      105        21730619    1532/redis-server *
    
  3. Check loopback interface and ping 127.0.0.1
    The loopback interface is properly set up in /etc/network/interfaces and ifconfig reports it up and running. I can also ping 127.0.0.1 and localhost without problem.

  4. Disable firewall
    Disabling the firewall did not change the situation. The connection is still timing out.

  5. Try to connect via telnet
    I tried to telnet to one of the services and that is where I noticed an odd pattern:

    # telnet 127.0.0.1 6379
    Trying 127.0.0.1...
    telnet: Unable to connect to remote host: Connection timed out
    
    # telnet ::1 6379
    Trying ::1...
    Connected to ::1.
    Escape character is '^]'.
    

When I try to connect to a service (Redis in that example) via IPv4 it times out, but if I try to connect via IPv6 it connects instantly. Is there some file related to IPv4 connectivity that could cause this type of behavior? Is there a way to fix this without having to reimage the server?

Update

After reading SYN's answer I have tried to connect to the same service (see above) but using my server's public IP instead (but still from inside the server) and it connects instantly. My understanding is that it works because it listens to 0.0.0.0 which accepts connections on any interface. But connecting from 127.0.0.1 still does not work and neither does connecting to a service that listens on 127.0.0.1 specifically. My conclusion then would be that there is indeed a problem with my loopback interface (on IPv4). Here is the output from ifconfig :

# ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:7984 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7984 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:711801 (695.1 KiB)  TX bytes:711801 (695.1 KiB)

venet0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:127.0.0.2  P-t-P:127.0.0.2  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
          RX packets:35812 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47530 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2568793 (2.4 MiB)  TX bytes:34332070 (32.7 MiB)

venet0:0  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:*public ip*  P-t-P:*public ip*  Bcast:*public ip*  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1

Is there something from there that would explain the malfunction of the loopback interface? Is there another log or config file that I have overlooked that could explain or potentially fix the problems I'm having with this interface?

Update 2

A quick update to add that my server is a VPS under OpenVZ. From my (continuing) Google searches I have found that OpenVZ does networking a bit differently than other platforms so I am including that info here to potentially steer us in the right direction. From what I have seen though, nobody who's had a problem remotely similar to mine seem to have found the solution... (e.g. this post from Unix & Linux StackExchange).

  • OpenVZ networking is a little weird. Consider using localhost instead of 127.0.0.1, or better yet, moving to a different type of VPS. – Michael Hampton Nov 11 '16 at 3:52
1

I would bet you can connect redis on your IPv4. Unless redis listens on 127.0.0.1:6379, you can't connect (nor telnet) to localhost.

Not familiar enough with IPv6 to explain why it would work though.

Then again, I doubt nginx proxies traffic to redis. Can you show us your which virtualhost(s) is/are enabled? Is it normal your python processes listen on 0.0.0.0? If so, you should probably enable back whatever firewall rules you've disabled.


Update, reading OP's updates:

Nice to see you've found something. Meanwhile, my first remark regarding connecting to localhost was just plain wrong, apologies.

  • My understanding is that if a server listens on 0.0.0.0 it is accessible on all interfaces isn't it? Then I should be able to connect to it on 127.0.0.1 and I have been for months until the recent situation I describe in my question. Then again, MongoDB listens on 127.0.0.1 (port 27017) and I cannot connect to it either right now. – J-F Desrochers Nov 10 '16 at 4:32
  • You are right, NGinx does not proxies traffic specifically to my redis server. It proxies to a Python WSGI application (the python processes you see) which then accesses the redis database locally. I left that part out of the question to avoid confusion. – J-F Desrochers Nov 10 '16 at 4:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.