2

How do you configure the interfaces in the OS without using ec2-net-utils?

I need to configure around 18 public ips to be accessible on a single EC2 instance. Each public IP needs to be bound to a single internal IP on the instance. I have 2 ENIs configured with the necessary Elastic IPs and these are attached to the instance but I am stuck at getting the OS recognise the ENIs.

I have been following this tutorial but can't work out how to extend it to configure multiple IPs per interface. Also there seems to be some debate according to this question about what the correct way to do this is.

If anyone would be able to point me in the right direction I would be really grateful. I have nearly wasted 2 days on this!

Here are the details:

  • OS: Centos 7
  • ENI1 : Secondary Private IPs = [10.0.1.97 ...10.0.1.106]
  • ENI2 : Secondary Private IPs = [10.0.1.218 ... 10.0.1.226]
  • VPC gateway 10.0.1.1
  • If this doesn't get you somewhere useful, please tell us where you're stuck, more specifically... what you've tried, and in what way it fails. You might also examine the source code of ec2-net-utils, to give you some idea of what mechanism that code uses. – Michael - sqlbot Mar 8 '16 at 15:18
  • Are you trying to automate the config, or just make it work? /sbin/ifconfig eth0:1 10.0.0.97/24 up should, for example, be sufficient to bring up the next interface... then eth0:2 eth0:3 and so on. There's (afaik) no DHCP support for the secondary addresses. – Michael - sqlbot Mar 8 '16 at 15:35
1

Since the previous answer is clutter, I will put another one with workflow, just by using AWS CLI (you can write fine tune automation script using AWS SDK) (http://docs.aws.amazon.com/cli/latest/userguide/installing.html)

  1. Create ENI with private IP using. (aws ec2 create-network-interface) Write down ENI id
  2. Allocate EIP for VPC. (aws ec2 allocate-address --domain vpc) write down EIP-id
  3. Link EIP to ENI-id, point to correct private IP (aws ec2 associate-address)
  4. Create or launch EC2 instance, attach to the ENI. (aws ec2 attach-network-interface)

Once you put everything script in place, recreate the EC2 instance with the proper ENI is just matter of minutes.

(Updated answer below) In Linux, to assign multiple IP address to an interface, the correct assignment to the interface is to add additional ip address to the physical interface. For ubuntu , etc, it is something like eth0:0 , eth0:1, for 1st interface, eth1:0, eth1:1 for subsequent interface.

And this is slightly different for Centos, ie.

enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether ....
inet 192.168.1.150/24 brd 192.168.1.255 scope global enp0s3
inet 192.168.1.151/24 brd 192.168.1.255 scope global secondary enp0s3
inet 192.168.1.152/24 brd 192.168.1.255 scope global secondary enp0s3

So the correct documentation should be this one. http://www.unixmen.com/linux-basics-assign-multiple-ip-addresses-single-network-interface-card-centos-7/

In short, Centos will automatically create one network-interface file for each interface. Just go /etc/sysconfig/network-scripts/ and check each file name as ifcfg-eth* (don't ask me why the above link show enp0) . The tricky part is whether your Centos enforce to use NetworkManager, and you must configure as required by the centos documentation wiki.centos.org/FAQ/CentOS7

So you should see least 4 interface file for your m3.xlarge , e.g. /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 /etc/sysconfig/network-scripts/ifcfg-eth2 /etc/sysconfig/network-scripts/ifcfg-eth3

Since the main interface should be running, it will give you the hint of the running instance IP address. So if you open /etc/sysconfig/network-scripts/ifcfg-eth0, if the instance configure as IP 10.0.1.10, you should see this

IPADDR0="10.0.1.10"

To add more IP address, just do as unixmen link say, i.e.

IPADDR1="10.0.1.97"
IPADDR2="10.0.1.98"
IPADDR3="10.0.1.99"

Then open ifcfg-eth1, ifcfg-eth2, ifcfg-eth3 and do repeat the task. After done that, use "systemctl restart network" to restart. (backup all config file so you just override the file in the future)

Next , you play with the routing part. Just print out your VPC route table, Subnet. Now inside /etc/sysconfig/network-scripts/, you deal with route-ethX , i.e. route-eth0, route-eth1, route-eth2,route-eth3. That's mean, you must know your own network to configure this part (which should be shown in your VPC layout). Since you mentioned only one VPC gateway 10.0.1.1, so for all route-* file, it should be something like this

# I just assume your put all your 10.0.1.x in the CIDR /24 segments
#
# file route-eth0
# Assume your first ENI IP address is  10.0.1.10 
# format : default via gateway-ip dev dev-name table route-table-number 
default via 10.0.1.1 dev eth0 table 0

# format : network cidr  dev dev-name src ENI-intrace-IP route-table-number  
10.0.1.0/24 dev eth0 src 10.0.1.10 table 0

# file route-eth1
# Assume your 2nd ENI IP address is  10.0.1.11
default via 10.0.1.1 dev eth1 table 1
10.0.1.0/24 dev eth1 src 10.0.1.15 table 1

# file route-eth2
# Assume your 2nd ENI IP address is  10.0.1.12
default via 10.0.1.1 dev eth1 table 2
10.0.1.0/24 dev eth2 src 10.0.1.12 table 2

Then you follow the document you mentioned instruction, in /etc/sysconfig/network-scripts, create a rule-ethX , i.e. rule-eth0, rule-eth1 1. Increment the table number to match route-ethX 2. Change the IP to the assigned internal network address of the ENI.

# file rule-eth0 ,but as the doc suggest, you should skip this file.
# format : from  ENI_IP/CIDR table <table_number> 
from 10.0.1.10/32 table 0

#file rule-eth1
from 10.0.1.11/32 table 1

You should play the Centos network setup with your Local Vmware/virtualbox virtual network adapter. Then you don't need to worry a sudden

  • all this does is attach the ENI to the instance, there is still the problem of configuring the OS to use the secondary IP address(s). See the docs at docs.aws.amazon.com/AWSEC2/latest/UserGuide/MultipleIP.html, specifically the section on "Configuring the Operating System on Your Instance to Recognize the Secondary Private IP Address" – Russell Mar 9 '16 at 2:28
  • Oh, now I understand the issues. Seems I owe an apology to @Michael - sqlbot. So it appear to be bad documentation from those solution . :-D – mootmoot Mar 9 '16 at 19:15
  • The internetstaff document say "Multiple EC2 Network Interfaces " , it didn't say multiple ENI with multiple private address. I assume you get stuck there. – mootmoot Mar 9 '16 at 20:50
  • Thanks for improving the answer @mootmoot. This pretty much gets the job done. The final thing I have left is that, although I can communicate with the machine on all the public addresses fine, the machine cannot resolve external addresses itself, for example ping 8.8.8.8 outputs "Network is Unreachable" this is solved if I type "route add default gw 10.0.1.1 eth0" but I cannot make this change permanent. I have tried adding "GATEWAY=1.0.1.1" in /etc/sysconfig/network and in /etc/sysconfig/network-scripts/ifcfg-eth0 but nothing helps. Any ideas? – Russell Mar 10 '16 at 7:07
  • Since internetstaff suggest use the default route, that's why I suggest to skip route-eth0. Maybe you should add route-eth0 , with the entry "from <First_ENI_IP>/32 table 0" (replace that <> stuff with your first ENI IP), then restart the network. Because it change the default route. As important reminder : always try to this out in your local VM. Unless you are able to ssh the instance via another ENI IP, then you still have a backup connection if route-eth0 screw up. – mootmoot Mar 10 '16 at 8:57
0

The answer by @mootmoot got me most of the way on this one but, just for completeness, here are the steps that eventually got me all the way to a working configuration. I'm sure there are better ways to do this but in case someone else is stuck ...

I made the following modifications to the files in /etc/sysconfig/network-scripts

I left the interface defined in ifcfg-eth0 alone and added aliases as follows:

ifcfg-eth0:0

DEVICE=eth0:0
BOOTPROTO="static"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
PEERDNS="yes"
IPV6INIT="no"
DEFROUTE="no"


IPADDR0="10.0.1.XXX"
PREFIX0="24"

Where 10.0.1.XXX is the first secondary address on the ENI (the primary address remains configured by dhcp as specified in the default ifcfg-eth0 file)

continue adding ifcfg-eth0:X files until all the ips in the ENI are specified.

Then define the second device in the file ifcfg-eth1 using the same template as for the eth0 aliases above and the primary IP address of the ENI.

Then add aliases for eth1 by defining files called ifcfg-eth1:0 etc as above.

define the following routes files:

route-eth0

default via 10.0.1.1 dev eth0  table 1

route-eth1

default via 10.0.1.1 dev eth1  table 2

where 10.0.1.1 is the address of the VPC gateway

I then needed to define rules for all IPs in eth2 and all secondary ips in eth1 as follows:

rule-eth0

from 10.0.1.106 lookup 1
from 10.0.1.105 lookup 1
... etc

rule-eth1

from 10.0.1.41 lookup 1
from 10.0.1.226 lookup 1
... etc   

I also added the line

GATEWAYDEV=eth0

to /etc/sysconfig/networking

Hope that helps someone in a similar situation, thanks for all the advice.

-1

First, I will ALWAYS check AWS EC2 Network interface limit. (ref : http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI)

For example , if you create c1.medium instance, you cannot use more than 2 ENI and exceed 6 IP addresses per interface. If you try to create 20 virtual network interface in the OS, it will fail.

I am not sure the reason that you want to bound 1 Public IP to 1 internal IP, unless you intend to move to IPv6, put ELB in front your EC2 instance to wrap the IPv6 to IPv4. (Because EC2 doesn't support IPv6 )

But looking at the way of AWS VPC setup, you cannot think like using Centos in a Physical server box, which you can make something flexible like eth0:0, eth0:1, ...,eth0:20,

Unfortunately, this is not the case of AWS ENI and network interface. It seems AWS assign unique MAC for individual "private ip addresses". So limited private IP is tied to an interface. As stated above, you can only c1.medium create 2 network interface, each with 6 private IP max.
(ref : http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/MultipleIP.html)

You need to think as in AWS VPC ways, not the flexible way of physical machine may give. Like some extreme case :

  • VPC : The CIDR and instance IP address must be related. E.g. 10.1.0.0/19. The connection relies on the AWS VPC route tables.
  • Physical Server and network : You can use multiple CIDR to all the machine. E.g. a Server can have 1 network interface with IP adderss from two and more CIDR , e.g. 10.1.0.0/19, 172.16.0.0/19 . eth0:1 10.1.0.15 eht0:2 172.16.7.15. You can setup/add/delete your own route.

It seems you need to redesign your network infra according to AWS VPC ways.

You may ask, what if you think about Elastic IP? Well, EIP limit for VPC are 5. And do remember this, Public IP address are scarce. Perhaps splitting the application usage with multiple-domains is a better idea.

(Update) Associate Public IP address to Private IP. (Try to use AWS cli or aws sdk script so you can recreate the instance with a script)

  1. Create EC2 instance , assign multiples private IP to each Network interface
  2. Go to VPC - Elastic IP , allocate Elastic IP
  3. Go to the newly allocated elastic IP, select associate, there is option to attach to specific EIP and Private IP address.

Since aws set EIP limit to 5 for each VPC, you need to check whether the soft limit has been raise. To test it out, just try allocating 18 EIP, and point to individual Private IP address.

You cannot assign EIP inside Centos instance. EIP options is found under EC2 and/or VPC console.

  • Yes I am aware of the limit of the number of addresses per ENI, I am using m3.xlarge so this is not the problem. Splitting to multiple machines is not an option either. This is a staging server set up to replicate one of our physical servers. I know this can be done. It was done before but unfortunately I terminated the previous instance without realizing how complicated it would be to recreate the network set-up. – Russell Mar 8 '16 at 14:54
  • even m3.xlarge only support 4 ENI , 15 Private address per interface. Again, You may request "exceptional request" from AWS; or the "magic" actually is done through ELB or something else. Unless you are the one who create the network interface, otherwise, you need to refer to the person who "make things works before", or any documentation that explain how it is done. – mootmoot Mar 8 '16 at 15:02
  • 1
    I have 2 ENIs , 10 private interfaces per interface so this is fully supported. In fact if we were using an Amazon Linux AMI you can configure all this with ec2-net-utils (Unfortunately this is not an option). This is fully possible with Centos of course as well, just need to configure manually. Telling me to look at "any documentation" does not really help either. That is why I'm asking on Server Fault. If you don't have an answer then please remove your post. – Russell Mar 8 '16 at 15:11
  • It is an issue of EIP limit towards your Private IP address. If the EIP limit is not raise (5 per VPC) , you cannot make a 1 to 1 assignment from EIP towards the private IP. I just suspect the "working instance" that you terminated, actually achieve the goal using some other method. Looking at the Centos AMI, I have no problem putting Private IP address until it hit the limit. Next , assigning EIP to specific ENI and specific Private IP address, no issue either. But then, I use up my ENI. – mootmoot Mar 8 '16 at 15:26
  • @mootmoot EIP is always mapped 1:1 towards a single private IP. Some of the questions raised in the body of this answer might have been more appropriate as comments requesting clarification. We know that OP has a compatible supported VPC/ENI configuration. The question is, simply stated, appears to be "how do you configure the interfaces in the OS without using ec2-net-utils." This answer strays far from there. -1 I'm afraid. – Michael - sqlbot Mar 8 '16 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.