30

I added a VPC endpoint to my VPC using CloudFormation, and allowed s3 usage. The routes are visible in the AWS console, but not in the local routing tables of the EC2 instances:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.29.4.129    0.0.0.0         UG    0      0        0 eth0
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0
172.29.4.128    0.0.0.0         255.255.255.128 U     0      0        0 eth0

How do I verify that the EC2 instances in the VPC actually uses the VPC endpoint for S3, and not the available internet connection?

2
  • 1
    Just try accessing your s3 bucket from the instance Dec 28, 2016 at 10:52
  • 3
    @ArjunPrasad By accessing, I can confirm that S3 is available, not that it is available over the VPC endpoint
    – M. Glatki
    Dec 28, 2016 at 10:55

7 Answers 7

22

I guess the straightfwd way is to actually probe those routes.

You can traceroute to s3 and see if the NAT Gateway's internal IP is anywhere in the output (eg. the first hop).

First, check the NAT Gateway internal IPs in the console.

Example output with the endpoint set - no gateway IP shown. This is what you want to see.

$ traceroute -n -T -p 443 s3.amazonaws.com                                
traceroute to s3.amazonaws.com (52.216.204.93), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  52.216.204.93  0.662 ms  0.668 ms  0.637 ms

Example output of a different destination, going via NAT (see the first hop)

$ traceroute -n -T -p 443 serverfault.com
traceroute to serverfault.com (151.101.129.69), 30 hops max, 60 byte packets
 1  172.20.10.188  0.206 ms  0.147 ms  0.145 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  100.65.13.49  0.956 ms 100.65.13.113  1.253 ms *
 8  52.93.28.209  1.083 ms 52.93.28.231  1.213 ms 52.93.28.235  1.151 ms
 9  100.100.4.38  1.770 ms 100.100.4.46  2.089 ms 100.100.4.36  1.723 ms
10  103.244.50.242  1.136 ms 100.100.4.44  1.702 ms  2.738 ms
11  151.101.129.69  1.013 ms 103.244.50.244  1.745 ms 151.101.129.69  1.142 ms
5
  • I am not sure if this is accurate, per the docs here traffic to the endpoints do not traverse nat or internet gateways docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints.html Nov 20, 2020 at 20:56
  • 4
    @NoahSparks That's right, and it's in line with the answer. The answer is basically: if you don't see a VPC IP as first hop then all good, it's going via endpoint; but if you see one, then you're NAT-ed and the endpoint is probably misconfigured. That's what the docs indicate.
    – Valer
    Nov 26, 2020 at 11:41
  • Ah sorry, I guess I misread this initially and thought you were saying the example with the nat gateway in the route was the desired behavior. It's clear now though, thanks. Nov 28, 2020 at 13:53
  • You might have to write bucket region in the URL like that: traceroute -n -p 443 s3.us-west-2.amazonaws.com Dec 3, 2021 at 12:24
  • I have setup with a S3 Gateway and NAT gateway in my route table. I see the NAT gateway IP as the first hop for s3.amazonaws.com. But when I try the bucket policy with "deny accept when from the VPC", as per other answer, I can view the bucket. That seems to suggest that it is using the gateway for S3 access. I've verified by removing the gateway, keeping the bucket policy and the access is then denied. Then I tried traceroute in MY REGION: sudo traceroute -n -T -p 443 s3.eu-west-2.amazonaws.com which didn't show the NAT gateway.You do need to use s3 with the correct region (unless us-east)
    – Stagg
    Feb 1, 2022 at 14:04
15

I have found a method to verify the VPC endpoint usage.

  1. Log in to an AWS EC2 instance in the VPC
  2. Configure the aws cli client
  3. run aws ec2 describe-prefix-lists; for Windows PowerShell, Get-EC2PrefixList

The result should contain the the VPC endpoints prefix list ID in the attribute PrefixListId.

For additional verification, you can apply the following policy to an S3 bucket:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::mybucket"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:sourceVpc": [
            "vpc-121212"
          ]
        }
      }
    }
  ]
}

with your vpc ID instead of vpc-121212. You should then only be able to access the S3 bucket from the given VPC

2
11

You can turn on S3 logging and check if the files are being accessed from your private IP rather than public. If your logging shows private IPs are accessing the buckets you've configured it correctly. Goodluck!

7

I would recommend to launch ec2 instance (with IAM role allowed to list s3 buckets) in subnet without internet access.

Basically only 2 active rules in route table (your VPC subnet range and s3 endpoint).

Connect to instance and run command:

aws s3 ls /**

It should fail with timeout because boto by default will create request to global s3 url (s3.amazonaws.com).

export AWS_DEFAULT_REGION=us-east-1** ## your region here
aws s3 ls /**

should list your buckets in us-east-1 region (vpc router will route your request to s3.us-east-1.amazonaws.com).

1
  • This would be a good idea, except for the fact that you won't be able to connect to your instance once you remove the other routes. I suppose you could leave something running to test this while you're not connected, though!
    – Tim Malone
    Sep 26, 2022 at 8:09
4

Elaborating on @m-glatki solution, add a policy on the bucket that restricts S3 access to a particular VPC Endpoint:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::mybucket",
            "Condition": {
                "StringNotEquals": {
                    "aws:SourceVpce": "vpce-01ab2c3d4"
                }
            }
        }
    ]
}

You will only be able to list bucket contents from a process that uses the VPC endpoint. Otherwise, you will receive a message:

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

To get the VPC Endpoint ID, use this command: aws ec2 describe-vpc-endpoints

See this link

1
2

@m-glatki's answer (which for some reason is the accepted one) is factually incorrect.

First of all, you have to explicitly enable an ec2 VPC interface to even be able to perform the aws ec2 describe-prefix-lists call, otherwise you will get a timeout.

Secondly, even if you can call that api, it won't tell you whether you are routing your traffic through that endpoint. It just provides details about a specific prefix list (PL) in the current region.

What you have to do is associate an S3 VPC endpoint to the subnet's route table and make sure your EC2 instance or service's security group allows egress connectivity to via that endpoint (you should be fine with the default "allow all" egress rule). This will route S3 traffic via the endpoint, even if you have a NAT gateway attached to it.

You can verify that your traffic is not routed through the NAT by checking its associated cloudwatch logs (See BytesOutToDestination, BytesOutToSource, BytesInFromDestination, and BytesInFromSource metrics)

Also check S3 bucket logs as @michael correctly pointed out.

1

Your instance forwards packets destined to S3 to the local gateway, and from there the VPC 'router' forwards them to the S3 endpoint. No client configuration or knowledge is required.

You could configure the S3 endpoint with a very restrictive set of ACLs such that it denies all requests and observe your client receive the failure as well.

2
  • I suppose with ACL you mean SecurityGroups? I will try to add a limiting egress rule to the instance.
    – M. Glatki
    Dec 28, 2016 at 19:13
  • 1
    @M.Glatki by ACL, I believe he means endpoint policy. Dec 28, 2016 at 19:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .