5

I decided to play with veth : create a veth pair and send a ping from one end to another.

$ ip link add type veth
$ ip addr add 192.168.99.1 dev veth4
$ ip addr add 192.168.99.2 dev veth5
$ ip link dev veth4 set up
$ ip link dev veth5 set up

Let's check.

$ ip a

18: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 6a:dc:02:5b:f0:f3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.1/24 scope global veth4
       valid_lft forever preferred_lft forever
19: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 22:ec:d5:e8:7c:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.2/24 scope global veth5
       valid_lft forever preferred_lft forever

Everything seems to be ok. Now try to ping.

$ ping -I veth4 192.168.99.2
PING 192.168.99.2 (192.168.99.2) from 192.168.99.1 veth4: 56(84) bytes of data.
From 192.168.99.1 icmp_seq=1 Destination Host Unreachable
From 192.168.99.1 icmp_seq=2 Destination Host Unreachable
From 192.168.99.1 icmp_seq=3 Destination Host Unreachable


$ sudo tshark -i veth5
Capturing on 'veth5'
l  1   0.000000 6a:dc:02:5b:f0:f3 -> Broadcast    ARP 42 Who has 192.168.99.2?  Tell 192.168.99.1
1   2   1.003206 6a:dc:02:5b:f0:f3 -> Broadcast    ARP 42 Who has 192.168.99.2?  Tell 192.168.99.1

So veth5 recevies arp requests but doesn't bother to answer. What's the matter?

5 Answers 5

8

This is a very complex, very old question, involving several different areas, specifically VETH, Network Namespaces, ARP, route tables, and NAT...

I am adding the answer to this problem not just for OP, but for others out there, hopefully saving someone else from the grey hairs this gave me...

Suffice to say, after much research and testing -- veth pairs can exactly be used this way, as they are literally two seperate virtual interfaces. However, because they are virtual -- you will encounter some really odd problems with regards to ARP tables/entries. After several weeks of tackling this exact scenario (necessary for work), I have figured out exactly how to do this, and in the process have learned way more about ARP, NAT, and routing than I wanted to.

In order to accomplish exactly what you want, type the following (in my case, Ubuntu 16.04). Please note that once you are in the test namespace, you have to exit bash to leave, as everything (route tables, iptables, etc) from that point on is different than on the host machine. I usually open up two terminals, one that stays in the host -- the other that stays in the guest namespace.

Preface

# Tell the system we want to support IPv4 forwarding  
add/verify that the following is in `/etc/sysctl.conf`

`net.ipv4.ip_forward = 1`

# if it wasn't there, reparse syscrtl + restart networking
sysctl -p;
/etc/init.d/networking restart;

HOST SETUP

# create a veth pair
ip link add name vHOST type veth peer name vGUEST

# choose a private MAC address and private IP address
ifconfig vHOST hw ether 02:1d:8d:dd:0c:61
ifconfig vHOST 10.11.0.1/24 up

# have to setup routes   FROM HOST -> GUEST
ip route add 10.111.0.0/24 via 10.11.0.1 dev vHOST  # gateway

# have to explicitly assign what the MAC is for the vGUEST in the vHOST interface ARP table
arp -i vHOST -s 10.111.0.1 02:1d:8d:dd:0c:60 # vGUEST ip + mac address

# We must tell vHOST-vGUEST "tunnel", vHOST side that it's not a REAL bridge, so any ARP requests must be answered by vHOST
echo 1 > /proc/sys/net/ipv4/conf/vHOST/proxy_arp


# setup forwarding + NAT (so packets can come back)
iptables -A FORWARD -s 10.111.0.0/24 -o vHOST -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.111.0.1 -j SNAT --to 192.168.42.124

GUEST SETUP

# create the "test" namespace, so we can verfiy settings
ip netns add test

# add vGUEST "interface" to the "test" namespace
ip link set vGUEST netns test

# enter the "test" namespace with a bash shell
ip netns exec test bash

# choose a private MAC address and private IP address
ifconfig vGUEST hw ether 02:1d:8d:dd:0c:60
ifconfig vGUEST 10.111.0.1/24 up

# have to setup routes   FROM GUEST -> HOST
ip route add default via 10.111.0.1 dev vGUEST   # gateway

# have to explicitly assign what the MAC is for the vHOST in the vGUEST interface ARP table
arp -i vGUEST -s 10.11.0.1 02:1d:8d:dd:0c:61 # vHOST ip + mac address

now, GUEST/HOST can ping eachother, and vGUEST can ping outside the machine

Of paticular note, multimac can also do the exact same thing as veth, in a slightly different way, as multimac supports a many-to-one relationship, where veth is a one-to-one relationship.

In order to support this on without namespaces, you have to use policy routing (which is a bit more complex than network namespaces)

2
  • Unfortunately this renders veth interfaces unusable for me... Inserting arp entries manually is a hack that is not feasible for my use case. Isn't there any better solution for this?
    – Mike76
    Commented Nov 12, 2018 at 17:52
  • I think that I might work around this limitation by creating a dummy interface and setting up a bridge between one of the veth interfaces and the dummy interface. I will try whether configuring a different ip address on the bridged dummy interface does the job.
    – Mike76
    Commented Nov 12, 2018 at 18:48
4

Veth pairs aren't used this way. A veth pair is literally a device pipe, one end of the pipes packets come out the other end.

The simplest synonym I can offer is imagining one half of the pair is a ethernet device, whilst the other end is a switch port the device is plugged into. You shouldn't treat the pair as being two separate devices, but one device which has two 'ends' to push/pull from.

5
  • Well, I can use it this way for example if put one end in different namespace, I wonder why I could not do it without namespaces.
    – nshy
    Commented Mar 29, 2014 at 5:39
  • The same reason you cant do ping -I eth0 <eth1IP> when both are on the same network? When you dont specify the device (and it works) it actually uses lo. Commented Mar 29, 2014 at 7:04
  • Mmm, still don't understand. Hypothetical eth0 and eth1 are not connected. But veth0 and veth1 are. And when I put say veth1 in namespace and from host ping -I veth0 veth1IP - it works.
    – nshy
    Commented Mar 29, 2014 at 12:08
  • veth0 and veth1 are not connected, rather they are two ends to the same device, so it makes little sense to assign two addresses to it. My point about eth0 and eth1 is that if a test case without using a veth pair doesn't work, why would you expect a veth pair to work? Commented Mar 29, 2014 at 12:44
  • 1
    I have exactly the same misunderstanding as the OP. Why does it work with namespaces but not without? What is the difference? I do think of the veth as a pipe (well, as a "cable"), which can be used to connect two devices (say a host and a switch), so each device owns an end of the pipe, and does reply to ARP requests! (And in this scenario, it is common to have a different IP address assigned to each end of the pipe).
    – Norswap
    Commented Nov 12, 2014 at 17:37
1

You can do it without namespaces. Be sure to set accept_local to 1. In case you have duplicate ip addresses across multiple VMs, you can try this approach blog

ip link add name br0 type bridge
ip link add veth10 type veth peer name veth11
ip link add veth20 type veth peer name veth21
ip link set veth10 master br0
ip link set veth20 master br0
ip link set veth10 up
ip link set veth20 up
ip link set veth11 up
ip link set veth21 up
ip link set br0 up
ip addr add 192.168.1.1/24 dev veth11
ip addr add 192.168.1.2/24 dev veth21
echo 1 > /proc/sys/net/ipv4/conf/veth11/accept_local
echo 1 > /proc/sys/net/ipv4/conf/veth21/accept_local
# test veth11 -> veth21 connectivity 
ping 192.168.1.2 -I veth11
# test veth21 -> veth11 connectivity
ping 192.168.1.1 -I veth21

From kernel doc

accept_local - BOOLEAN
    Accept packets with local source addresses. In combination with
    suitable routing, this can be used to direct packets between two
    local interfaces over the wire and have them accepted properly.
    default FALSE

NOTE: iproute2 configs are not persistent. Be sure to persist configs with nmcli or network scripts to preserve across reboots

0

try set these:

echo 1 > /proc/sys/net/ipv4/conf/all/accept_local
echo 1 > /proc/sys/net/ipv4/conf/default/accept_local
echo 1 > /proc/sys/net/ipv4/conf/veth4/accept_local
echo 1 > /proc/sys/net/ipv4/conf/veth5/accept_local
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/veth4/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/veth5/rp_filter
0

For veth interfaces you need to bit change ip rule: no arp, no ping answer etc.

# ip rule
0:  from all lookup local
1000:   from all lookup [l3mdev-table]
32766:  from all lookup main
32767:  from all lookup default

After move lookup local after l3mdev veth works correctly

# ip rule
1000:   from all lookup [l3mdev-table]
32000:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

You can switch off ARP by ip link set arp off dev <veth> but with lead lookup local, remote veth in the same netns does not response own trafic anyway...

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.