5

I am new to linux and AWS.

I ran sudo chmod 2770 / command on my ec2 instance and after that

I was getting Permission denied on everything I was doing(even using ls or cd)

So I exited my connection(using cygwin) and tried to re-connect but now I get

Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

I tried setting chmod 400 my.pem , chmod 600 my.pem, chmod 777 my.pem but nothing worked.
I am trying to connect using ssh -i my.pem ec2-user@xx.xx.xx.x which was working fine earlier.
What is the solution?

  • I have installed so many things on my instance. I don't want to loose it. Is there any way to get it back ? – iAmLearning Feb 19 at 7:09
  • 5
    You should be using EBS snapshots to ensure you can recover your instance to a point in time that it's working. The advice from MLu below is what I'd have said too. Don't experiment on servers that have important data. I did something similar on my EC2 instance when I was learning, I had to restore to a snapshot because it was far too difficult to fix. – Tim Feb 19 at 7:41
14

Solution is to start a new instance and never do what you did again. It would be too complicated to try to properly recover all the permissions that you reset to 2770.

If you have any valuable files on the broken instance you can stop it, mount its root volume to the new instance and copy the files from there.


Update: as @GeraldSchneider points out you may be lucky if you didn't recursively change all the permissions everywhere. You'll have to start a new instance and use it to fix the root permissions back to 0755. Follow for example the instructions here: Changed AWS EC2 firewall rule and locked out of ssh (instead of Fix the firewall do sudo chmod 755 /mnt or wherever you mount the other disk).

Hope that helps :)

  • 4
    My first thought was to rebuild the instance as well, but then I realized that the OP only changed the permission on / itself, not the directories and files below. It should be pretty easy to fix this, but I'm not familiar with EC2 and don't know how to mount the volume on a different instance. – Gerald Schneider Feb 19 at 7:21
  • Is rebuilding possible ? I am just learning ec2 so I don't care about permissions/security. I just need it working again if possible. – iAmLearning Feb 19 at 7:38
  • @GeraldSchneider good point, updated the answer. – MLu Feb 19 at 7:39
  • @iAmLearning now's as good a time as any to at least understand the Unix filesystem permissions model. – 0xdd Feb 19 at 19:24
3

What you did was make every file in the filesystem a 2770 permission.

-rwxrws--- 1 username  agroup  2 Feb 19 23:07 thefilename

Thats a sticky bit in the group column, which means all files inside a directory are owned by agroup.


I've never bjorked an AWS image quite that badly. But I've seen a few probs that kill them.

FIRST Revert to your last snapshot, before hosing the file modes.

You don't do periodic snapshots?

SECOND Look at your backups. Is it going to be more or less work to rebuild the box vs restore your data from backups?

What? You don't have backups either?

Then the last ditch standard recovery method would be something like:

  1. Create a new instance from a current AMI, ideally the same distro as your broken machine. It can be something small like a t3.nano
  2. Detach the volumes from your broken machine, and attach them to the new instance as sdf, g, h... and so on
  3. Log into your new instance as root and for each of your broken instance's disks run

    fsck /dev/xvdX
    mkdir /sdX
    mount /dev/xvdX /sdX
    cd /sdX
    ls -l
    
  4. At this point you need to decide whether its worth using chmod over and over to fix your problem, or whether you copy the data to your new instance and set it up over again.

  5. So manually change into each directory, and chmod each file to what it should be. Keep two windows open and compare the live host's files with the broken disks mounted. Make sure you're changing the RIGHT files - check often!!!

  6. When you've done the lot, shut down the temp machine, detach the disks in the EC2 web gui and then reattach them to the old machine, in the same mountpoints from which they came. NOTE the root drive is attached as sda1 not sda but all other volumes are attached as sdb through z.


Either way, you should set up automated snapshots or backups, or both!

To prevent yourself doing this exact same thing again, alias chmod to

 chmod --preserve-root

But this won't protect any other directory.

Also don't use sudo in front of commands just by habit.

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.