3

I am trying to make use of an AWS VPC Endpoint to access an S3 resource with no luck. Could you help?

Details:

I have an EC2 Instance inside a Security Group. This is inside a VPC and subnet with an internet endpoint and a routing table entry to it for 0.0.0.0/0. However, I don't want this specific EC2 instance to be able to talk to the internet but I do want it to be able to talk to an S3 Bucket located at https://s3-eu-west-1.amazonaws.com/XXXXX/YYYYYY.sh.

In response, I have overloaded the default allow Egress rule on the securityGroup so outbound connections are denied by adding an egress rule to port 22. The amazon documentation tells me that local (private) AWS addresses will still be allowed. With this in mind, I added a VPC endpoint into my VPC and updated my routing table with pl-6da54004 (com.amazonaws.eu-west-1.s3) :: vpce-9f47a1f6.

With this done, I cant access my S3 resource from within my EC2 instance. I try and wget https://s3-eu-west-1.amazonaws.com/XXXXX/YYYYYY.sh with no luck.

Do you have any idea what is wrong? My thoughts are:

  • my ec2 instance seems to resolve the S3 domain to 54.231.130.244, but this is not the same for every instance. Do i need to add a securityGroupEgress rule or a route for an S3 IP range? if so, what is the range?
  • Perhaps I should be accessing my S3 resources via a different domain name that instead resolved to an internal IP address?

Thank you, Toby

5

First, a bit of background. The DNS resolver for VPC instances is a virtual component that is built in to the infrastructure. It's immune to the outbound security group rules... but the resolution of the hostnames for S3 endpoints doesn't change when you provision an S3 endpoint for your VPC.

What a VPC endpoint for S3 does is a couple of different things. Understanding what those things are is key to understanding whether it will do what you need. tl;dr: it will, in this case.

First, you notice they are configured in the route tables as "prefix lists." A VPC endpoint takes a set of predefined IPv4 network prefixes, and hijacks the routes to those prefixes for every route table that includes the respective prefix list so that your traffic to any of those networks will traverse the VPC endpoint instead of the Internet Gateway and any intermediate NAT instance.

In essence, this opens a new path out from your VPC to the AWS service's IP address ranges... but where those IP addresses take you is not, initially, and he same place as they would take you without the VPC endpoint in place.

The first place you hit looks just like S3 but it isn't identical to the Internet-facing S3, because it knows about your VPC endpoint's policies, so that you can control which buckets and actions are accessible. These do not override the other policies, they augment them.

An endpoint policy does not override or replace IAM user policies or S3 bucket policies. It is a separate policy for controlling access from the endpoint to the specified service. However, all types of policies — IAM user policies, endpoint policies, S3 policies, and Amazon S3 ACL policies (if any) — must grant the necessary permissions for access to Amazon S3 to succeed.

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html#vpc-endpoints-access

Note that if you do not restrict bucket access with an appropriate policy, and instead enable full access, the instances will be able to access any bucket in the S3 region if the bucket's policies allow it, including public buckets.

Now, the tricky part. If your instance's security group doesn't allow access outbound to S3 because the default "allow" rule has been removed, you can allow the instance to access S3 via the VPC endpoint, with a specially-crafted security group rule:

Add a new outbound rule to the security group. For the "type," choose HTTPS. For the destination, choose "Custom IP."

The documentation is not consistent with what I see in the console:

The Destination list displays the prefix list IDs and names for the available AWS services.

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html#vpc-endpoints-security

Well... no, it doesn't. Not for me, least, not as of this writing.

The solution is to choose "Custom IP" and then, instead of an IP address block or security group ID, type the prefix list id for your VPC endpoint, in the form of pl-xxxxxxxx in the box for the IP address. You can find this in the VPC console, by looking at the destinations in one of the subnets associated with the VPC endpoint.

  • 1
    Thank you very much for that detailed reply Michael. I am doing all this within a cloud formation template. I havent been unable to test your suggestion yet, but presumably I will be able to put an egress rule as a reference to the VPC endpoint? eg { "Ref" : "S3Enpoint"}, where S3Endpoint is of type AWS::EC2::VPCEndpoint – Toby Wilkins Oct 5 '15 at 15:52
  • 1
    I've confirmed with AWS support that it's not possible to do this all at once in CloudFormation. Unfortunately there is no way to get the prefix list ID via CloudFormation which is what I wanted to do as the return value from the AWS::EC2::VPCEndpoint is the resource ID of the endpoint itself and not the prefix list ID. To work around this I'm using a conditional in my VPC stack that only creates the egress rule if it is passed in. So I run it once to setup the VPC with the endpoint, then manually run it again with the new prefix list parameter. – Soliah Nov 3 '15 at 23:25
  • 2
    Prefix lists are per service and region only, they exist independently of the VPC endpoint and your account: you can get them with e.g. aws ec2 describe-prefix-lists. The id is safe to embed - eg com.amazonaws.us-west-1.s3 is always pl-6ba54002. You can use cloudformation maps to avoid passing it as a parameter. – Simon Buchan Apr 28 '17 at 0:01
  • @SimonBuchan, thanks! You appear to be entirely correct. DescribePrefixLists looks like it may have debuted in the 2015-04-15 version of the EC2 API... so I guess at the time I wrote this, I had not seen this and was still doing things the hard way. I'll need to revisit this answer to bring it up to date. – Michael - sqlbot Apr 28 '17 at 0:34
  • Also note if you're using cloudformation you can just add a AWS::EC2::VPCEndpoint resource and point it at your VPC and the route tables to update, and set ServiceName set to com.amazonaws.us-west-1.s3 (for example) - no need to mess with prefix lists or egress rules at all. – Simon Buchan May 8 '17 at 5:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.