Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My scenario

enter image description here

  • 2 physical machines, each with 2 network interfaces (eth0, eth1).
  • they act as virtual machine host
  • for one machine alone, the networking works with no problems
  • first i had two seperated networks and each machine has a own virtual dhcp server running
  • then i tried to merge the seperated networks into one
  • firewall: shorewall (connection policy for lan = allow on both machines)
  • dhcp server: dnsmasq
  • both machines can connect to the internet

i would like to connect the two machines over a crossovercable and want them to share one network, so i have only one dhcp server with one network and each server in this lan can connect to each other. does this make sense or are two seperate dhcp server and networks the better way?

Problem

below i added some configs and i did some connection tests.

in short:

  • machine1 + machine2 can reach ips on machine1
  • machine1 + machine2 cannnot reach ips on machine2
  • machine1 + machine2 can reach ips on machine2
  • external dnat (for example ssh) does work for machine1 (port 5678 -> 10.62.63.20:22)
  • external dnat (for example ssh) does not work for machine2 (port 5678 -> 10.62.63.30:22)

if i ssh connect to machine1 port 5678 the connection to 10.62.62.20 works and i only see shorewall log entries from this connection on machine1. but if i connect to machine2 port 5678 the connection does not work and i can see a martian log on machine1

Nov 29 15:26:57 machine1 kernel: [ 7495.749894] martian source **ssh.client.ip.addr** from **yyy.yyy.yyy.yyy**, on dev br1

Machine Overview

machine1 virtual machines:

  • dhcp 10.62.63.2
  • web1 10.62.63.20

machine2 virtual machines:

  • web2 10.62.63.30

Config Files

machine1 /etc/shorewall/rules

***snip***
DNAT:debug  net  lan:10.62.63.20:22  tcp  5678  -  xxx.xxx.xxx.xxx
***snip***

machine2 /etc/shorewall/rules

***snip***
DNAT:debug  net  lan:10.62.63.30:22  tcp  5678  -  yyy.yyy.yyy.yyy
***snip***

machine1 /etc/networking/interfaces

# Loopback device:
auto lo
iface lo inet loopback

# device: eth0
#allow-hotplug eth0 
auto eth0
iface eth0 inet manual


# device: eth1
#allow-hotplug eth1
auto eth1
iface eth1 inet manual 

auto br0
iface br0 inet static
  address   xxx.xxx.xxx.xxx
  broadcast xxx.xxx.xxx.xxx
  netmask   xxx.xxx.xxx.xxx
  gateway   xxx.xxx.xxx.xxx
  bridge_ports eth0
  bridge_fd 0
  bridge_hello 2
  bridge_maxage 12
  bridge_maxwait 0
  bridge_stp off


auto br1
iface br1 inet static
  address 10.62.63.1
  broadcast 10.62.63.255
  netmask 255.255.255.0
  bridge_ports eth1
  bridge_fd 0
  bridge_hello 2
  bridge_maxage 12
  bridge_maxwait 0
  bridge_stp off

machine2 /etc/networking/interfaces

# Loopback device:
auto lo
iface lo inet loopback

# device: eth0
#allow-hotplug eth0 
auto eth0
iface eth0 inet manual


# device: eth1
#allow-hotplug eth1
auto eth1
iface eth1 inet manual 

auto br0
iface br0 inet static
  address   yyy.yyy.yyy.yyy
  broadcast yyy.yyy.yyy.yyy
  netmask   yyy.yyy.yyy.yyy
  gateway   yyy.yyy.yyy.yyy
  bridge_ports eth0
  bridge_fd 0
  bridge_hello 2
  bridge_maxage 12
  bridge_maxwait 0
  bridge_stp off


auto br1
iface br1 inet static
  address 10.62.63.3
  broadcast 10.62.63.255
  netmask 255.255.255.0
  bridge_ports eth1
  bridge_fd 0
  bridge_hello 2
  bridge_maxage 12
  bridge_maxwait 0
  bridge_stp off

Tests

machine1 (10.62.63.1)

routes:

ip route show
yyy.yyy.yyy.yyy/yy dev br0  proto kernel  scope link  src yyy.yyy.yyy.yyy
10.62.63.0/24 dev br1  proto kernel  scope link  src 10.62.63.1
default via yyy.yyy.yyy.yyy dev br0
  • ping 10.62.63.3 to br1 ip (remote): ok
  • ping 10.62.63.1 to br1 ip (local): ok
  • ping 10.62.63.2 to dns (local): ok
  • ping 10.62.63.20 to web01 (local): ok
  • ping 10.62.63.30 to web02 (remote): ok
  • ssh 10.62.63.20 to web01 (local): ok
  • ssh 10.62.63.30 to web02 (remote): ok

machine2 (10.62.63.3)

routes:

ip route show
yyy.yyy.yyy.yyy/yy dev br0  proto kernel  scope link  src yyy.yyy.yyy.yyy
10.62.63.0/24 dev br1  proto kernel  scope link  src 10.62.63.3
default via yyy.yyy.yyy.yyy dev br0
  • ping 10.62.63.3 to br1 ip (local): ok
  • ping 10.62.63.1 to br1 ip (remote): ok
  • ping 10.62.63.2 to dns (remote): ok
  • ping 10.62.63.20 to web01 (remote): ok
  • ping 10.62.63.30 to web02 (local): ok
  • ssh 10.62.63.20 to web01 (remote): ok
  • ssh 10.62.63.30 to web02 (local): ok
share|improve this question
    
ip neigh ls as well as ip route get <ip> for the nonworking IP addresses? –  Michael Renner Nov 28 '12 at 20:35
    
as well as tcpdump -p -i br1 -n -s0 for the nonworking packets and upload the dumps to cloudshark.org –  Michael Renner Nov 28 '12 at 20:36

1 Answer 1

up vote 0 down vote accepted

the problem was the outbound packets, not the inbound.

the dhcp server supplied the same gateway for both machines (10.62.63.1), so dnat inbound packets on machine2 had no problem to reach their location (web on machine2), but then the returning packet was sent to the gateway of machine1 (10.62.63.1) and not from where it was comming (10.62.63.3).

so it lands on machine1 as martion packet.

solution was to add tagging on the dns (dnsmasq) so the virtual machines on the different hosts get different gateways:

/etc/dnsmasq.conf

*** snip ***

dhcp-host=set:machine1,ff:ff:ff:ff:ff:ff,web01,10.62.63.20
dhcp-host=set:machine2,ee:ee:ee:ee:ee:ee,web02,10.62.63.30

dhcp-option=tag:machine1,option:router,10.62.63.1
dhcp-option=tag:machine2,option:router,10.62.63.3

*** snip ***
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.