Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've broken my sudoers file on a heavy loaded server with a syntax error. And unfortunately I've lost my root password. The server is critical and I'd like to avoid rebooting. So, no sudo, no root, no reboot, no kdesudo or gksu. The server is a virtual machine running on kvm.

How would you fix this?

share|improve this question

migrated from superuser.com Oct 10 '13 at 21:45

This question came from our site for computer enthusiasts and power users.

(1) If you’re talking about a VM, you should say so in the body of the question and use appropriate tag(s). (2) Are you just saying that you can’t do a clean shutdown, because you can’t become root, or are you saying that this is a critical, high-availability server, and you have a requirement to keep it running (i.e., not shut it down)? –  Scott Oct 10 '13 at 21:11
(1) Fixed the question. (2) The server is critical and heavy-loaded, so I'd not like to stop it. –  raacer Oct 10 '13 at 21:17
If this server is so critical, it surely has backups and/or HA. Failover to other node and/or restore the sudoers file from a backup. –  dawud Oct 10 '13 at 21:51
And that's why you use visudo. It does syntax checking. –  Magellan Oct 10 '13 at 21:56
Are ssh keys set up and can you log in as root via ssh? –  Michael Martinez Oct 10 '13 at 22:10

4 Answers 4

Look for the date of the last update, and then search a root exploit since that date.

share|improve this answer
Nice idea. What is the probabiliy for success? –  raacer Oct 10 '13 at 21:19

Just an unverified idea:

On a host system you have access to the image of the guest partition, right ? You can't easily mount that image since it's in use. But you don't need to mount the partition to edit a single file.

  1. Find the sudoers file sector in the partition - either use the text search for some phrase that you know exists in your sudoers file and not frequently appears at other files, like # This file MUST be edited with the 'visudo' command as root. Probably you may use for that some GUI hex editor, alternatively I believe grep can do the job here too.

    Or maybe you can discover the sector where sudoers file resides from within the guest OS...

  2. Use hex editor to edit file directly on disk, or use dd - copy sector to text file, edit it and copy it back to the original sector.

Difficulties: 1. maybe the file is cached inside the guest OS, probably you will find a way to flush caches. 2. With text search you may find several files looking like your sudoers, like deleted versions of it, either you find-out which is the right one or fix all of them.

share|improve this answer
Wow, directly editing the virtual disk? Sure it might be possible, but the chance for catastrophic failure seems very high. –  Zoredache Oct 10 '13 at 22:46
If you do the things right, chance for failure is zero. All operations are read - search text etc., once you find the sector - dd if=partition bs=512 count=1 skip=NNN of=textfile. Edit the text file (that is a regular text file on the host OS now, not on a live partition). so far all is safe. Verify the file size is exactly 512 bytes (one sector), and verify that by saving it you don't introduce another syntax error at the sector boundary. Risky operation is single and atomic - dd if=textfile bs=512 count=1 seek=NNN of=partition. –  Sandman4 Oct 10 '13 at 22:58
Thanks! I came to the same solution :) And this worked! See my report amongs another answers. The idea is same, but I've added some deatils on how I did this. –  raacer Oct 10 '13 at 23:24
I'm in doubt. Which answer I should accept now? Maybe they should be merged somehow? Or having identical answers is ok on stackexchange? –  raacer Oct 10 '13 at 23:30
Your should accept answer which best answers your questtion. Your answer is more detailed, accept it. –  Sandman4 Oct 11 '13 at 3:56
up vote 2 down vote accepted

Fortunately our server is virtual server, and I can access the host system. I've fixed the problem by editing the raw disk data. You also may try this way, but be very careful. The method is dangerous. You may corrupt another files, partition or even whole hard drive. So do it for your risk, and ensure you have recent backups.

  1. Install hexedit on host system.
  2. Run sync on guest.
  3. Open the guest hd device with hexedit.
  4. Find the corrupted string and note down it's position (something like 0x29221D2B8 at the bottom of screen).
  5. Convert the number to decimal using some calc or util (I've used kcalc).
  6. Try to read the corrupted text with dd and ensure the numbers are correct. skip=position/ibs.

    sudo dd if=/dev/vg0/vm-100-disk-1 count=1 bs=10 ibs=10 skip=1104162476

  7. Overwrite the corrupted text somehow to fix the error. In my case I just replaced first symbol with '#' to comment out broken things. Use of/obs/seek parameters instaed of if/ibs/skip with same numbers. Be careful with newlines and eof.

    echo "#ncludedir" | sudo dd of=/dev/vg0/vm-100-disk-1 count=1 bs=10 obs=10 seek=1104162476

  8. You can read again with dd and check if you did what you need.

  9. Return to the vritual machine terminal and try to use sudo now. Maybe you'll need to empty the disk read cache somehow. Have a good luck on this step! :)

share|improve this answer
For clarity I'd use bs=1 count=10 seek=position. ibs and obs are redundand if you use bs=, you can omit them. –  Sandman4 Oct 11 '13 at 4:24
While I would never condone this sort of operation, I'm impressed at the gumption and execution nevertheless! –  Joshua Miller Oct 11 '13 at 4:29
@Sandman4 I have not found any note about this in man dd, so even now I'm not sure bs is enough :) Could you please point me into some line in docs? –  raacer Oct 11 '13 at 15:42
@raacer hmm, I dunno. The fact is I never used ibs nor obs,I didn't even knew they exist.... `**ibs=BYTES** read up to BYTES bytes at a time (default: 512) obs=BYTES write BYTES bytes at a time (default: 512) bs=BYTES read and write up to BYTES bytes at a time –  Sandman4 Oct 12 '13 at 17:12

Boot your pc from a live distro, like Ubuntu; mount the partition containing the root directory. Let us assume that partition is /dev/sda2, you can mount it with

sudo mount /dev/sda2 /mnt

Now you do a chroot to the /mnt directory,

sudo chroot /mnt 

Now you are effectively root on your hard drive. You can change root and users' passwords by means of:

passwd username

Now that you have reset passwords, fixing the sudoers file is very simple:

pkexec visudo

It will ask for your password. Strictly speaking, this requires authorization with PolicyKit. You may fin more information here. Altogether, it is easier than you think.

share|improve this answer
I can boot in recovery mode or even mount the guest partition under host system. But I'd not like to stop the server. –  raacer Oct 10 '13 at 21:22
If you boot in recovery mode you are stopping the server. But at any rate: you find instructions here, debuntu.org/…. to change root password in recovery mode. The inst ructions are valid for all linux systems, not just Debian –  MariusMatutiae Oct 10 '13 at 21:43
He said he doesn't want to reboot the system. –  Michael Martinez Oct 10 '13 at 22:11
Read 3 comments above: "I can boot in recovery mode". –  MariusMatutiae Oct 10 '13 at 22:20
Yes, I can, it's simple to reboot it. But I do not want! :) –  raacer Oct 10 '13 at 23:15

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.