6

I have two systems, one running Ubuntu 11.04 and one running 10.04 LTS. The same configuration on both machine yields different results. The 10.04LTS machine is unable to find my AWS credential files.

On 11.04

export | grep EC2 

declare -x EC2_CERT="/home/jdw/.ec2/cert-PJU5MW2PCN24SKPOIVFTWCMBUXWWZO7U.pem" 
declare -x EC2_HOME="/opt/ec2-ami-tools-1.3-66634" 
declare -x EC2_PRIVATE_KEY="/home/jdw/.ec2/pk-PJU5MW2PCN24SKPOIVFTWCMBUXWWZO7U.pem"

ls -al /home/jdw/.ec2/ 
total 16 
drwxr-xr-x 2 jdw jdw 4096 2011-08-18 13:31 . 
drwxr-xr-x 58 jdw jdw 4096 2011-09-23 10:24 ..
-rw-r--r-- 1 jdw jdw 916 2011-08-18 13:30 cert-PJU5MW2PCN24SKPOIVFTWCMBUXWWZO7U.pem
-rw-r--r-- 1 jdw jdw 928 2011-08-18 13:30 pk-PJU5MW2PCN24SKPOIVFTWCMBUXWWZO7U.pem

/usr/bin/ec2-consistent-snapshot --description "Autosnapshot root" vol-283d5246 --region us-west-1 snap-0c679a62

On 10.04 LTS

export | grep EC2 

declare -x EC2_CERT="/root/.ec2/cert-PJU5MW2PCN24SKPOIVFTWCMBUXWWZO7U.pem" 
declare -x EC2_HOME="/mnt/ec2-api-tools-1.4.4.2" 
declare -x EC2_PRIVATE_KEY="/root/.ec2/pk-PJU5MW2PCN24SKPOIVFTWCMBUXWWZO7U.pem"

ls -al /root/.ec2 
total 16 
drwxr-xr-x 2 root root 4096 2011-09-23 10:50 . 
drwx------ 8 root root 4096 2011-09-23 10:50 ..
-rw-r--r-- 1 root root 916 2011-09-23 10:11 cert-PJU5MW2PCN24SKPOIVFTWCMBUXWWZO7U.pem
-rw-r--r-- 1 root root 928 2011-09-23 10:47 pk-PJU5MW2PCN24SKPOIVFTWCMBUXWWZO7U.pem

/usr/bin/ec2-consistent-snapshot --description "Autosnapshot root" vol-283d5246 --region us-west-1 ec2-consistent-snapshot: ERROR: Can't find AWS access key or secret access key at /usr/bin/ec2-consistent-snapshot line 97.

I assume I am missing some dependency on the 10.04 box but have not had any luck finding what it might me. The only dependency I have run across is the PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2 which is present on both machines.

Further information.

It seems that despite having my environment variables set correctly, ec2-consistent-snapshot is looking for my key in a file named .awssecret

open("/root/.awssecret", O_RDONLY) = -1 ENOENT (No such file or directory) write(2, "ec2-consistent-snapshot: ERROR: "..., 124) = 124 exit_group(2) = ? ~

And it appears that providing the credentials from the command line truncates the credential names:

execve("/usr/bin/ec2-consistent-snapshot", ["/usr/bin/ec2-consistent-snapshot", "--aws-access-key-id-file", "/root/.ec2/cert-PJU5MW2PCN44GRAV"..., "--aws-secret-access-key-file", "/root/.ec2/pk-PJU5MW2PCN44GRAVIV"..., "--description", "Autosnapshot SMC root ", "vol-283d5446", "--region", "us-west-1"], [/* 21 vars */]) = 0

I have attempted using shorter names and the name .awssecret for the pem files and this removes the 'cannot find' condition, but it results in a 'cannot validate the credentials' error. I therefore feel that resolving the main error will quite likely resolve the secondary one.

I'm positive this is some missing library or something on my 10.04 system.

3 Answers 3

5

I see what the confusion is. It's very common on AWS and I didn't even notice the first time I read your description. You're passing the wrong credential objects in the command line options.

The ec2-consistent-snapshot program needs the "Access Key ID" and "Secret Access Key". These are completely different values from the "Certificate" and "Private Key", even though they are used to accomplish the same purpose.

Here's an article I wrote which tries to explain many of the different credentials and where to find them:

Understanding Access Credentials for AWS/EC2
http://alestic.com/2009/11/ec2-credentials

2

I am the author/maintainer of ec2-consistent-snapshot (with thanks to other contributors of patches) and will be the first to admit that it tries too hard to find your AWS credentials by searching in too many places. Unfortunately, there is little consistency in where Amazon tools and other third party tools tell you to put your credentials. My goal was to avoid adding to this problem, so I figured it would be good to look in several of the most common places and use the first credentials found.

This works well for most scenarios, but it can cause problems if you have multiple, different AWS credentials stored in your file system, environment variables, and environment variables pointing to the file system.

The software makes a couple poor choices about prioritizing which values should override other values. For example, it looks like you are specifying --aws-access-key-id-file on the command line, but this will be overridden by the envariable $AWS_ACCESS_KEY_ID if it is set. Command line options should override environment or file system settings, but in this case it's getting confused because it's a specific value specified in the environment vs. a file path specified in the command line and it chooses specific values over values stored in a file somewhere.

Turn on the --debug command line option and look for debug output with the information like:

ec2-consistent-snapshot: Using AWS access key: ...

If this is the right access key id, then you might want to edit the program and add in a debug line directly afterwards that outputs the secret access key. If it is not the right access key id, then you'll want to look through the different places where the software looks for AWS credentials (see the manpage) and find where it is getting that information.

Note: In your debug line, I don't think the credentials are actually getting truncated in the software. I believe that is just the debugger saying that the actual values passed were longer than it cares to show.

4
  • Thanks for the help. I have followed through with the steps and agree that there is some confusion in how the keys are selected. So, to avoid that confusion, I am using the command-line options with --debug to see what happens. When I use the command-line options to specify the key and cert file, the correct key is selected, however I still get the error: AWS was not able to validate the provided access credentials When I do not specify the credentials on the command-line, I see that an incorrect key is being selected, but I am avoiding that scenario for this test case. Any more ideas?
    – jdw
    Sep 26, 2011 at 13:03
  • Did you try modifying the code to also output the secret access key id on debug? Sep 26, 2011 at 16:11
  • I did not because with --debug it is writing it out anyhow. This is what I get when I run it (with the access key sufficiently munged): pastebin.com/bkV5q181
    – jdw
    Sep 26, 2011 at 16:18
  • I don't know if you sufficiently munged that private key for security purposes. I recommend you rotate to use new AWS cert/pk. Sep 26, 2011 at 16:27
0

This is resolved. Thank you to @eric for walking me through the credentials.

The whole issue of environment variables vice command-line variables was a red-herring for me. It turns out that the working 11.04 box was working because, at some point, I had put my AWS credentials into an .awssecret files in my home directory. The fact that I also had a variety of environment variables set was irrelevant.

Once I set my .awssecret file to the same values on the 10.04 box, all was well.

It turns out that with a properly formatted .awssecret file, there is no need at for the following environment variables:

EC2-CERT, EC2_PRIVATE_KEY, AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY set at all.

2
  • Perfect. I was writing this as you wrote that. Thanks!
    – jdw
    Sep 26, 2011 at 16:33
  • Glad to hear it. Unfortunately, I really don't see any way to reduce confusion given the way that AWS has set up so many different credentials and ways to specify them. If I changed ec2-consistent-snapshot to have a single way to pass in the credentials it would break half the people using it today. Sep 26, 2011 at 17:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .