4

We're moving a bunch of services, say from 1.2.3.4 to 5.6.7.8.

In order to test that the new services are correctly configured, we'd like to redirect (to the new host) all traffic destined for the original host that originates from our testing workstations.

Of course, such redirection could be implemented on routers within the network itself—but for stability reasons we've decided to implement it on each workstation directly (which are all OS X 10.10 Yosemite, so use pre-v4.7 OpenBSD pf).

I've added to /etc/pf.anchors/com.apple:

rdr-anchor "910.TestServiceMove/*"
anchor "910.TestServiceMove/*"
load anchor "910.TestServiceMove" from "/etc/pf.anchors/910.TestServiceMove"

And created /etc/pf.anchors/910.TestServiceMove:

rdr pass log on lo0 from any to 1.2.3.4 -> 5.6.7.8
pass out log route-to lo0 from any to 1.2.3.4 keep state

When the rules are loaded, both appear to work correctly:

$ sudo tcpdump -v -n -e -ttt -i pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes
00:00:00.000000 rule 0.910.TestServiceMove.0/0(match): pass out on en1: (tos 0x0, ttl 64, id 40691, offset 0, flags [DF], proto TCP (6), length 64)
    9.9.9.9.58029 > 1.2.3.4.22: Flags [S], cksum 0x291a (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063366865 ecr 0,sackOK,eol], length 0
00:00:00.000047 rule 0/0(match): rdr in on lo0: (tos 0x0, ttl 64, id 40691, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 896a (->b4da)!)
    9.9.9.9.58029 > 5.6.7.8.22: Flags [S], cksum 0xb284 (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063366865 ecr 0,sackOK,eol], length 0

But the TCP handshake does not complete (SYN-ACKs are ignored, and SYN is repeatedly sent until the connection times out):

$ sudo tcpdump -v -n -e -ttt host 5.6.7.8
tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Packet Tap), capture size 65535 bytes
00:00:00.000000 e8:80:2e:e7:67:bc > 84:80:2d:35:e5:43, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 40691, offset 0, flags [DF], proto TCP (6), length 64)
    9.9.9.9.58029 > 5.6.7.8.22: Flags [S], cksum 0xb284 (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063366865 ecr 0,sackOK,eol], length 0
00:00:00.015524 84:80:2d:35:e5:43 > e8:80:2e:e7:67:bc, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    5.6.7.8.22 > 9.9.9.9.58029: Flags [S.], cksum 0x7ce4 (correct), seq 1901846890, ack 3399416414, win 14480, options [mss 1460,sackOK,TS val 523934721 ecr 2063366865,nop,wscale 7], length 0
00:00:00.986946 e8:80:2e:e7:67:bc > 84:80:2d:35:e5:43, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 25319, offset 0, flags [DF], proto TCP (6), length 64)
    9.9.9.9.58029 > 5.6.7.8.22: Flags [S], cksum 0xae9c (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063367865 ecr 0,sackOK,eol], length 0
00:00:00.014938 84:80:2d:35:e5:43 > e8:80:2e:e7:67:bc, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    5.6.7.8.22 > 9.9.9.9.58029: Flags [S.], cksum 0x78fa (correct), seq 1901846890, ack 3399416414, win 14480, options [mss 1460,sackOK,TS val 523935723 ecr 2063366865,nop,wscale 7], length 0
00:00:00.397794 84:80:2d:35:e5:43 > e8:80:2e:e7:67:bc, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    5.6.7.8.22 > 9.9.9.9.58029: Flags [S.], cksum 0x776c (correct), seq 1901846890, ack 3399416414, win 14480, options [mss 1460,sackOK,TS val 523936121 ecr 2063366865,nop,wscale 7], length 0
00:00:00.588237 e8:80:2e:e7:67:bc > 84:80:2d:35:e5:43, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 50201, offset 0, flags [DF], proto TCP (6), length 64)
    9.9.9.9.58029 > 5.6.7.8.22: Flags [S], cksum 0xaab4 (correct), seq 3399416413, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 2063368865 ecr 0,sackOK,eol], length 0

I guess that the TCP stack is discarding SYN-ACKs that have originated from a host other than that to which the SYN was sent. But shouldn't redirection rules rewrite traffic in both directions—indeed, shouldn't keep state be ensuring that the connection is tracked for this purpose?

Improve this question
15
  • @Downvoter: Care to comment? Have I not shown research effort? Is the question unclear? I'd happily improve the question if you could help me understand what's wrong with it!
    – eggyal
    Mar 22, 2015 at 12:20
  • 1
    Why not simply change the DNS records or edit your host file? That's easier and does the same.
    – megamorf
    Mar 22, 2015 at 12:37
  • @megamorf: It only "does the same" where name lookups are involved. Some of the applications that we are testing unfortunately do not use hostnames, but rather use the IP address directly.
    – eggyal
    Mar 22, 2015 at 12:49
  • @megamorf: In addition, for the applications that do use name lookups, we'd need to ensure that wildcard labels in over 2000 domains (most of which are outside our control) all resolve to the new host: sure, we could create an internally split DNS to accomplish that (though could not use /etc/hosts because of the wildcards), but would need to limit its viewing to the testing workstations. With so many domains and so much configuration involved—plus the applications that use hard IP addresses—we have determined that packet redirection is the most appropriate option.
    – eggyal
    Mar 22, 2015 at 12:50
  • It may be a dumb question, but I can not make any sense of it ... Why are you routing it through loopback? Wouldn't rdr pass log from any to 1.2.3.4 -> 5.6.7.8 work as well and without the pass?
    – Fox
    Mar 25, 2015 at 23:00

3 Answers 3

Reset to default
1

If the services you are accessing are all accessed by name then I'd just create a host entry with in the hosts file on your dev machines.

For example, if your machine was called myserver and resolved to 1.2.3.4 Then you can now create a host file entry /etc/hosts with

myserver 5.6.7.8

Anything that now tries to reach myserver will end up hitting the new machine.

Alternatively, just create a new isolated network with the new server at the old address and plug your dev machine into it to test.

The problem with a straight redirection or route rule is that the new server doesn't answer under the name of the old address. So you'd have to use NAT.

Improve this answer
2
  • I've already commented that hosts is not a suitable solution for two reasons: (1) some applications do not use name lookups; and (2) others rely on wildcard DNS labels. An isolated network with the new machine at the old address would be a plausible solution, but my question was how to get the redirection working. You say that I need to use NAT because "the new server doesn't answer under the name of the old address" but my reading of the pf manual is that rdr rules should handle exactly this situation: else what is the point in them?
    – eggyal
    Mar 26, 2015 at 18:51
  • I might have misunderstood the OSX rdr app... anyway, I'd just use a separate network. The amount of time you've spent working this out you'd have tested it by now.
    – hookenz
    Mar 26, 2015 at 18:56
0

Wouldn't it just be easier to throw the machines behind a simple loadbalancer or NAT? That's generally more of an industry-standard practice, anyway, and would allow you to further scale the application over time.

Improve this answer
5
  • Which machines? The test workstations? They're distributed across 28 offices globally; where do you propose this "simple loadbalancer" be placed? In any event, I'm interested in getting packet redirection working. There's nothing, conceptually, about it that shouldn't work—and the pf documentation, to my reading, makes it abundantly clear that this configuration is supported. So all I really need is some help getting it to work, rather than countless suggestions of alternatives (that happen not to be very useful in our situation anyway: hence why we chose this approach in the first place!).
    – eggyal
    Mar 26, 2015 at 22:45
  • Then, I'd say your question wasn't terribly clear, and should be rephrased. My apologies, but I read you're moving from one IP, to one IP. So, in that general case, it's generally easier to use use a third-party NAT (which, in my opinion, is a bit easier that pf rewrite rules - though generally requires a bastion-type host)
    – RVT
    Mar 26, 2015 at 23:03
  • You may well be right that my question is unclear—I'm certainly not getting from it the response I anticipated. We are "moving from one IP to one IP", but at this time we are simply trying to test that the services on the new IP are correctly configured. I will have a think about how best to rephrase my question to make this clear.
    – eggyal
    Mar 26, 2015 at 23:09
  • Use a fake network. You'd be done by now.
    – hookenz
    Mar 26, 2015 at 23:40
  • LVS was my first thought. That could be done in a Linux-VM and does not require additional hardware.
    – Nils
    Mar 27, 2015 at 14:35
-1

Can you just offer the "old" address from the "new" server ala sub-interface, or is the new address in a totally different subnet?

If you can't attach the new IP to the new server, then you haven't really moved the address. To route traffic across the old address to the new server, you need to NAT the address in the old subnet. Traffic would land on the "old" address and then be source NAT'd to a new address on the same server/appliance and sent to the "new" IP. Return traffic would then pass back to the SNAT address, which would un-NAT the traffic and return it to the original requester. This is really not a good long term solution, especially if the "new" address is across a WAN link. You've put off an inevitable failure and added latency to the mix as well.

A better long term solution would be to consider: 1. use DNS instead of IP 2. virtualize either the IP address (NAT'ing device like a firewall or ipmasq box), 3. or the server itself (load-balancer)

Improve this answer
5
  • We're not trying to move the service right away. The question is only how to test that the new server is configured appropriately, so it is undesirable at this time either to move the IP address or to indiscriminately redirect traffic from the old address (both those solutions would, obviously, affect the incumbent service). Afraid I'm going to downvote, as I don't see how these suggestions are useful. Sorry.
    – eggyal
    Mar 22, 2015 at 15:03
  • you didn't answer my question
    – Rick Buford
    Mar 25, 2015 at 22:09
  • It's on a totally different subnet.
    – eggyal
    Mar 25, 2015 at 22:11
  • Then I have, actually, answered your question. To properly route the traffic to an IP, you will have to NAT the traffic from the "old" subnet to the "new" host.
    – Rick Buford
    Mar 25, 2015 at 22:14
  • Configuration at the "old" (i.e. currently live, public-facing) service is not an option, not least because it risks affecting ongoing live traffic. We must configure the test platform to test the new service without affecting the live service. Your proposal does not assist with that, whereas the approach outlined in my question would achieve exactly what we want—if only I could understand how to fix the configuration. The pf documentation is quite clear that this is possible, I just need some help getting it to work.
    – eggyal
    Mar 25, 2015 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.