6

I've broken my sudoers file on a heavy loaded server with a syntax error. And unfortunately I've lost my root password. The server is critical and I'd like to avoid rebooting. So, no sudo, no root, no reboot, no kdesudo or gksu. The server is a virtual machine running on kvm.

How would you fix this?

Improve this question
9
  • (1) If you’re talking about a VM, you should say so in the body of the question and use appropriate tag(s). (2) Are you just saying that you can’t do a clean shutdown, because you can’t become root, or are you saying that this is a critical, high-availability server, and you have a requirement to keep it running (i.e., not shut it down)?
    – Scott
    Oct 10, 2013 at 21:11
  • (1) Fixed the question. (2) The server is critical and heavy-loaded, so I'd not like to stop it.
    – raacer
    Oct 10, 2013 at 21:17
  • 2
    If this server is so critical, it surely has backups and/or HA. Failover to other node and/or restore the sudoers file from a backup.
    – dawud
    Oct 10, 2013 at 21:51
  • 3
    And that's why you use visudo. It does syntax checking.
    – Magellan
    Oct 10, 2013 at 21:56
  • 1
    Are ssh keys set up and can you log in as root via ssh?
    – Michael Martinez
    Oct 10, 2013 at 22:10

7 Answers 7

Reset to default
4

This is not a general solution, but I'm posting this as an inspiration for people in the same situation.

I didn't have access to solutions mentioned so far but I was very lucky to have access to a cron task that ran as root. I used that cron task to run another script that fixed /etc/sudoers and from now on I will use only visudo.

So if you are in this situation cron tasks can be helpful. No downtime/server restart and no risk to mess up data with dd.

Improve this answer
2

Just an unverified idea:

On a host system you have access to the image of the guest partition, right ? You can't easily mount that image since it's in use. But you don't need to mount the partition to edit a single file.

  1. Find the sudoers file sector in the partition - either use the text search for some phrase that you know exists in your sudoers file and not frequently appears at other files, like # This file MUST be edited with the 'visudo' command as root. Probably you may use for that some GUI hex editor, alternatively I believe grep can do the job here too.

    Or maybe you can discover the sector where sudoers file resides from within the guest OS...

  2. Use hex editor to edit file directly on disk, or use dd - copy sector to text file, edit it and copy it back to the original sector.

Difficulties: 1. maybe the file is cached inside the guest OS, probably you will find a way to flush caches. 2. With text search you may find several files looking like your sudoers, like deleted versions of it, either you find-out which is the right one or fix all of them.

Improve this answer
5
  • 1
    Wow, directly editing the virtual disk? Sure it might be possible, but the chance for catastrophic failure seems very high.
    – Zoredache
    Oct 10, 2013 at 22:46
  • 1
    If you do the things right, chance for failure is zero. All operations are read - search text etc., once you find the sector - dd if=partition bs=512 count=1 skip=NNN of=textfile. Edit the text file (that is a regular text file on the host OS now, not on a live partition). so far all is safe. Verify the file size is exactly 512 bytes (one sector), and verify that by saving it you don't introduce another syntax error at the sector boundary. Risky operation is single and atomic - dd if=textfile bs=512 count=1 seek=NNN of=partition.
    – Sandman4
    Oct 10, 2013 at 22:58
  • Thanks! I came to the same solution :) And this worked! See my report amongs another answers. The idea is same, but I've added some deatils on how I did this.
    – raacer
    Oct 10, 2013 at 23:24
  • I'm in doubt. Which answer I should accept now? Maybe they should be merged somehow? Or having identical answers is ok on stackexchange?
    – raacer
    Oct 10, 2013 at 23:30
  • Your should accept answer which best answers your questtion. Your answer is more detailed, accept it.
    – Sandman4
    Oct 11, 2013 at 3:56
2

I have experienced a similar sudoers problem on my Synology NAS device.

My solution was to put the following command into the "Task Scheduler"

chmod 0755 /etc/sudoers

Which I set to run as a root.

Although my problem was permissions related, I bet a properly configured command line series of output redirections would clear out the file and replace the contents with a basic shell that would get you back up and running.

Something like:

echo "Defaults syslog=authpriv" > /etc/sudoers

echo "%administrators ALL=(ALL) ALL" >> /etc/sudoers

echo "Cmnd_Alias SHELL = /bin/ash, /bin/sh, /bin/bash" >> /etc/sudoers

echo "Cmnd_Alias SU = /usr/bin/su" >> /etc/sudoers

echo "%wheel ALL=(ALL) NOPASSWD: ALL, !SHELL, !SU" >> /etc/sudoers

Improve this answer
1

Boot your pc from a live distro, like Ubuntu; mount the partition containing the root directory. Let us assume that partition is /dev/sda2, you can mount it with

sudo mount /dev/sda2 /mnt

Now you do a chroot to the /mnt directory,

sudo chroot /mnt

Now you are effectively root on your hard drive. You can change root and users' passwords by means of:

passwd
passwd username

Now that you have reset passwords, fixing the sudoers file is very simple:

pkexec visudo

It will ask for your password. Strictly speaking, this requires authorization with PolicyKit. You may fin more information here. Altogether, it is easier than you think.

Improve this answer
5
  • 1
    I can boot in recovery mode or even mount the guest partition under host system. But I'd not like to stop the server.
    – raacer
    Oct 10, 2013 at 21:22
  • 1
    If you boot in recovery mode you are stopping the server. But at any rate: you find instructions here, debuntu.org/…. to change root password in recovery mode. The inst ructions are valid for all linux systems, not just Debian
    – MariusMatutiae
    Oct 10, 2013 at 21:43
  • He said he doesn't want to reboot the system.
    – Michael Martinez
    Oct 10, 2013 at 22:11
  • Read 3 comments above: "I can boot in recovery mode".
    – MariusMatutiae
    Oct 10, 2013 at 22:20
  • Yes, I can, it's simple to reboot it. But I do not want! :)
    – raacer
    Oct 10, 2013 at 23:15
1

Fortunately our server is virtual server, and I can access the host system. I've fixed the problem by editing the raw disk data. You also may try this way, but be very careful. The method is dangerous. You may corrupt another files, partition or even whole hard drive. So do it for your risk, and ensure you have recent backups.

  1. Install hexedit on host system.
  2. Run sync on guest.
  3. Open the guest hd device with hexedit.
  4. Find the corrupted string and note down it's position (something like 0x29221D2B8 at the bottom of screen).
  5. Convert the number to decimal using some calc or util (I've used kcalc).

  6. Try to read the corrupted text with dd and ensure the numbers are correct. skip=position/ibs.

    sudo dd if=/dev/vg0/vm-100-disk-1 count=1 bs=10 ibs=10 skip=1104162476

  7. Overwrite the corrupted text somehow to fix the error. In my case I just replaced first symbol with '#' to comment out broken things. Use of/obs/seek parameters instaed of if/ibs/skip with same numbers. Be careful with newlines and eof.

    echo "#ncludedir" | sudo dd of=/dev/vg0/vm-100-disk-1 count=1 bs=10 obs=10 seek=1104162476

  8. You can read again with dd and check if you did what you need.

  9. Return to the vritual machine terminal and try to use sudo now. Maybe you'll need to empty the disk read cache somehow. Have a good luck on this step! :)

Improve this answer
4
  • For clarity I'd use bs=1 count=10 seek=position. ibs and obs are redundand if you use bs=, you can omit them.
    – Sandman4
    Oct 11, 2013 at 4:24
  • While I would never condone this sort of operation, I'm impressed at the gumption and execution nevertheless!
    – Joshua Miller
    Oct 11, 2013 at 4:29
  • @Sandman4 I have not found any note about this in man dd, so even now I'm not sure bs is enough :) Could you please point me into some line in docs?
    – raacer
    Oct 11, 2013 at 15:42
  • @raacer hmm, I dunno. The fact is I never used ibs nor obs,I didn't even knew they exist.... `**ibs=BYTES** read up to BYTES bytes at a time (default: 512) obs=BYTES write BYTES bytes at a time (default: 512) bs=BYTES read and write up to BYTES bytes at a time
    – Sandman4
    Oct 12, 2013 at 17:12
1

It's a old question, so i don't know. But now i had solved mounting the vmdk image of my vm with guestmount

sudo guestmount -a ubuntu-focal-20.04-cloudimg.vmdk -i --rw /mnt/vmdk

Then i had changed the file with visudo

/mnt/vmdk/etc# visudo sudoers
Improve this answer
0

Look for the date of the last update, and then search a root exploit since that date.

Improve this answer
2
  • Nice idea. What is the probabiliy for success?
    – raacer
    Oct 10, 2013 at 21:19
  • Only tried it once, successfully.
    – bbaassssiiee
    Dec 31, 2015 at 14:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .