I'm trying to setup udp load balancer with IP and port transparent proxy. My environment runs on docker. We are using mesos + marathon to manage containers. For UDP load balancing I use Nginx. App and Nginx runs in docker containers. I have few requirements: 1. One client should always connect to the same app node during his session. 2. App behind nginx should assume that it is communicating to the client directly.

I used this article for my setup: https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/

So nginx is running with root user. Containers are on the same network. I've included a bunch of modules for nginx like stream and nginx-sticky-module-ng and etc.

Nginx Stream conf: `

upstream app-server {
  server some-app:5684;
  server some-app:5684;

server {     
    listen 5684 udp;
    proxy_pass app-server:5684;
    proxy_bind $remote_addr:$remote_port transparent;
    proxy_responses 1;
    proxy_timeout 1s; 

On Each upstream server I set default gw to Nginx IP.

On nginx I set following iptables rules.

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p udp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p udp -s --sport 5684 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 0 #I assume this should be 0,because there could be different ports
ip rule add fwmark 1 lookup 100
ip route add local dev lo table 100

But client can't reach the server it seems. It looks like packets just got stuck on the Nginx. Am I missing some configuration or something wrong with iptables rules?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.