2

Please Note: The problem I describe below seems to have "gone away" on its own. But that's not very comforting, so I'm hoping for a solution, or at least an explanation, should it recur.

I was just preparing a Slicehost virtual server for a LAMP role, using the Ubuntu Lucid server (CLI-only). I created an account for myself (yukondude) but was doing most of the installations as root. At some point in the process, the mysql-server package (aptitude install -Pr mysql-server) failed with this error:

Unable to set password for the MySQL "root" user

An error occurred while setting the password for the MySQL administrative user.
This may have happened because the account already has a password, or because of
a communication problem with the MySQL server.

This was the first time mysql-server had been installed, so the problem couldn't have been a previous password. The installation script then dumped these lines to the terminal:

100903 19:57:05 [Note] Plugin 'FEDERATED' is disabled.
/usr/sbin/mysqld: Can't create/write to file '/tmp/iblv4tJ0' (Errcode: 13)
100903 19:57:05  InnoDB: Error: unable to create temporary file; errno: 13
100903 19:57:05 [ERROR] Plugin 'InnoDB' init function returned error.
100903 19:57:05 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
ERROR: 1146  Table 'mysql.user' doesn't exist
100903 19:57:05 [ERROR] Aborting

It hung at this point, so I manually killed the installation processes.

I then discovered that the Apache2 server was refusing to serve my test index.html file, instead producing a 403 Forbidden message, even though the permissions for /var/www/index.html were wide open and the Apache config hadn't changed. I was also unable to log in through SSH to my yukondude account, and the command su - yukondude produced a cd /home/yukondude Permission Denied error (I'm afraid I don't have a copy of the exact error message). The permissions and ownership for /home/ and /home/yukondude/ were perfectly normal. Looking back at the mysql-server errors, I wondered if the access to the file in /tmp/ was related.

What could've caused this bizarre permissions problem? I've never seen anything like it before. I wondered whether PAM might be involved, but I don't know that system as well as I should. I'm not sure whether the mysql-server install was really the culprit, or just a victim of the permissions problem that cropped up at that moment.

I tried a complete re-install from a fresh image with the exact same result. My third and final attempt did work (the difference was that I updated my package versions before any installations), so I appear to be in the clear this morning. But I'm nervous that the same problem might someday re-occur. I tried Googling for help, but haven't figured out a good search phrase. Any suggestions?

Update: I'm going to start a bounty. As I pointed out, this problem isn't affecting me anymore, but I sure would like to have an inkling of what happened. I'd also appreciate comments from anyone who's experienced something similar in case those circumstances provide clues.

5
  • Do you have any ls -l outputs?
    – Rob Olmos
    Sep 4, 2010 at 18:21
  • Sorry, no. I foolishly didn't keep a copy of the directory permissions, but I checked them over and over and over and even chmod-ed the /home and /home/yukondude directories to 777 to see if it made any difference -- nope.
    – yukondude
    Sep 4, 2010 at 18:25
  • did you (perhaps accidentally) turn on SELinux or AppArmor? Sep 4, 2010 at 22:24
  • @Gilles: no, not that I know of. I'll look into those though and see if they provide any hints on what happened. Thanks for the tip.
    – yukondude
    Sep 4, 2010 at 22:35
  • @Gilles: turns out you can't actually run SELinux or AppArmor on a Slicehost virtual server without special help from the admins, so I don't think I could've done it accidentally.
    – yukondude
    Sep 7, 2010 at 17:49

6 Answers 6

2

Try chmod +t /tmp and see if that fixes it.

3
  • Good idea. I didn't notice if the sticky bit was set or not. But would that prevent non-root account CLI logins? I wouldn't have thought that /tmp would be needed for something that simple. And would it also explain why Apache suddenly stopped serving pages with a 403 Forbidden?
    – yukondude
    Sep 14, 2010 at 17:25
  • Well, that fits better then my "answer". Especially because of the way you could 'trigger' the failure.
    – Karst Lok
    Sep 14, 2010 at 19:17
  • @yukondude: were you logging in via ssh? Perhaps it could not create some sockets in /tmp, as ssh usually does. But your mileage may vary. If you can try reproducing the bug to be certain, might be useful for future SF users
    – lorenzog
    Sep 15, 2010 at 12:19
1

One pretty common thing that could have happened is that you ran chmod or chown with the wrong parameters.

A quick search of serverfault for recover chmod returns lots of questions describing similar symptoms where the person ran the wrong chmod command.

3
  • I wondered about that myself, and looked through .bash_history just in case, but didn't find anything. All of the ownerships and permissions of the /home, /tmp, and /var/www directories appeared correct and yet the login, mysql-server install, and apache processes all seemed to fail simultaneously with permission-related errors. Even chmod-ing directory permissions to 777 didn't fix anything, so I figure the problem must've gone even deeper. It's a puzzler.
    – yukondude
    Sep 4, 2010 at 21:28
  • Have you tried running the mount command to see if the filesystem has been remounted as read-only? That happens under some conditions like filesystem corruption or HD errors.
    – Zoredache
    Sep 5, 2010 at 0:41
  • I had checked the mount as well, but it was read-write. In fact, I had been installing other packages as root during this time without noticing the problem. It was only the non-root accounts that had the problem (which wouldn't have been the case for a ro filesystem).
    – yukondude
    Sep 5, 2010 at 15:35
1

Could be your Xen host having problems with a disk or disk-access.

Running an install job can only be done as root, so you had to su or sudo for it. When you tried to logon with another account, you where not denied access to the system, but denied access to your home folder. It's safe to say that rules out any problems with PAM.

We once had a problem with similar symptoms. Our Xen Guests tried to write something to disk, wich failed, ultimately resulting in kernel panic on the guest. Before that, the guests behaved much like your server. Turned out a disk with guest-images in the Xen host was failing, which led to guests i/o operations timing out. In some cases the Xen host accepts your disk-writes when it actually has not commited them yet, when the write times out on the host, your server-image will be corrupt.

1
  • I have a feeling you might be right, but I kinda hope not because that means I would have no control over it happening again, and no way to fix it other than through a support call. I'll double-check with Slicehost to see if anything noteworthy happened at their end. Strange that I could start over with a fresh image and then cause the problem again (seemingly during the mysql-server install).
    – yukondude
    Sep 9, 2010 at 21:49
0

Three ideas:

  • Slicehost's underlying system suffered an issue like a kernel forced remount to read-only
  • Your slice or Slicehost's system ran out of disk space
  • A messed up compilation script (never run them as root) or installer
2
  • The problem may well be related to Slicehost's Xen setup, but it was strange that it seemed to occur both times during the MySQL installation. I checked and nothing was mounted RO after the problem occurred, and there was still plenty of disc space. I was still merrily installing stuff as root the first time it happened. I do suspect that you're right about a kernel issue -- but a reboot (many, actually) didn't fix the problem.
    – yukondude
    Sep 8, 2010 at 4:35
  • @yukondude The only other thing I can think of was a pretty messed up script. Like a compilation script (which should never be run as root) or some other installer.
    – Rob Olmos
    Sep 9, 2010 at 22:33
0

Is it posible you have two user with the same UID? Look /etc/passwd

4
  • No, that wasn't it, although now I'm curious just what would happen if two accounts shared an UID. This problem seemed to have shut down all non-root account simultaneously. I wouldn't think that would happen if just two of them had the same UID.
    – yukondude
    Sep 8, 2010 at 15:29
  • @yukondude: Nothing good ... nothing good at all
    – Zypher
    Sep 8, 2010 at 21:02
  • I had this problem once, two users with the same UID. Some files showed one user and another files showed the other. I fixed it before anything went wrong.
    – Maxfer
    Sep 14, 2010 at 7:11
  • Two users with the same ID should just work. I do that all the time with the users root and toor. (Two different admins, both ID 0). And files are checked vs IDs even though the shell helpfully translates to the the first name found in /etc/passwd.
    – Hennes
    Feb 12, 2013 at 14:26
-2

just do this:

sudo chmod 777 /tmp/
1
  • (a) this doesn't answer the question (they're asking "What could cause...", not "What do I do to fix..") ; (b) This is INCORRECT (or at least incomplete) -- You are neglecting the fact that the sticky bit needs to be set on /tmp
    – voretaq7
    Feb 12, 2013 at 16:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .