Operation not permitted when starting Unicorn

Question

I've created an nginx/unicorn/capistrato setup on Ubuntu (Amazon EC2) by following mostly this guide. I guess everything is set up like it should but when I start Unicorn I get (a LOT of) this error in the log:

E, [2012-09-08T08:57:20.658092 #12356] ERROR -- : Operation not permitted (Errno::EPERM)
/home/deployer/apps/bridgekalenderen.no/shared/bundle/ruby/1.9.1/gems/unicorn-4.3.1/lib/unicorn/worker.rb:82:in `initgroups'

I see it's related to the user's permissions but I just can't figure out what I've left out. The server starts up nicely if I start it with sudo (or, rvmsudo, really).

The user has sudo capabilities, I have chmod'ed the app several times so the file permissions there should be ok. The unicorn socket in /tmp is owned by the deployer user, so that shouldn't be the problem either.

Does anybody have a clue where to look?

UPDATE:

After some digging I found out that it boils down to a call to Process.initgroups which throws EPERM. I've verified this in irb. I can't figure out what causes the error. The user can read /etc/group.

Answer 1

If I understand you well, you are trying to start a service without sudo under normal user. This will not work and will give your errors like the ones you received Operation not permitted. This is true in most (if not all) cases because any service will require one or more of the following to do its job:

1. Binding/listening on ports < 1024.
2. Reading file(s) not readable by non-root.
3. Writing file(s) not writeable by non-root.
4. and possibly more (I can not recall).

Answer 2

I finally figured it out. The problem was that the deployer user's primary group was wrong. It should be 'staff' but was 'deployer' instead. This means that unicorn tries to hand over the ownership of new worker processes to the group it's supposed to use, but only root can do that.

Just in case anybody else needs to know, I changed the primary group by editing /etc/passwd like this:

deployer:x:1002:50:,,,:/home/deployer:/bin/bash

'50' is the gid of 'staff'. It was 1002 to begin with. To get the gid of the 'staff' group, do:

cat /etc/group | grep staff

It will say something along the lines of:

staff:x:50:<comma separated list of users in this group>

The gid is the number after 'x'.

Answer 3

I've encountered the same issue. What ended up working for me was setting the following in my unicorn config in the after_fork do |server, worker| block: (it's from the github unicorn config)

  begin
    uid, gid = Process.euid, Process.egid
    user, group = 'deployer', 'staff'
    target_uid = Etc.getpwnam(user).uid
    target_gid = Etc.getgrnam(group).gid
    worker.tmp.chown(target_uid, target_gid)
    if uid != target_uid || gid != target_gid
      Process.initgroups(user, target_gid)
      Process::GID.change_privilege(target_gid)
      Process::UID.change_privilege(target_uid)
    end
  rescue => e
    if RAILS_ENV == 'development'
      STDERR.puts "couldn't change user, oh well"
    else
      raise e
    end
  end

Answer 4

I lost a man day recently to this. The context was trying to run Unicorn on a user named apps and the Errno::EPERM error just kept biting us. Here's the stack trace:

E, [2018-04-12T20:45:07.277588 #2048] ERROR -- : Operation not permitted (Errno::EPERM)
/apps/cas-seas3/shared/bundle/ruby/2.2.0/gems/unicorn-5.4.0/lib/unicorn/worker.rb:143:in `initgroups'
/apps/cas-seas3/shared/bundle/ruby/2.2.0/gems/unicorn-5.4.0/lib/unicorn/worker.rb:143:in `user'
/apps/cas-seas3/shared/bundle/ruby/2.2.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:657:in `init_worker_process'
/apps/cas-seas3/shared/bundle/ruby/2.2.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:682:in `worker_loop'
/apps/cas-seas3/shared/bundle/ruby/2.2.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/apps/cas-seas3/shared/bundle/ruby/2.2.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:563:in `maintain_worker_count'
/apps/cas-seas3/shared/bundle/ruby/2.2.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:293:in `join'
/apps/cas-seas3/shared/bundle/ruby/2.2.0/gems/unicorn-5.4.0/bin/unicorn:126:in `<top (required)>'
/apps/cas-seas3/shared/bundle/ruby/2.2.0/bin/unicorn:23:in `load'
/apps/cas-seas3/shared/bundle/ruby/2.2.0/bin/unicorn:23:in `<top (required)>'

Instrumenting the code base with a print statement around line 143 gave us the following debug info:

user = apps
group=apps
uid=1040
gid=110
whoami=apps
Process.egid = 1001

That discrepancy between the gid and Process.egid tripped this code block:

if gid && Process.egid != gid
  Process.initgroups(user, gid)
  Process::GID.change_privilege(gid)
end

and that's what caused the failure.

I finally traced it down to a difference between the GID / EGID and the issue was that the apps user belonged to a different primary group (admin). When I shifted the apps user to the apps group as primary then it worked. This was done by editing the /etc/passwd file which defined the box's user accounts.

Another way to test for this is to use sg to manually run unicorn with the correct group using the sg command like this:

bundle exec sg apps -c unicorn -c /apps/cas-seas3/current/config/unicorn.rb -E deployment -D

If you find that works then this is a very good sign to look into your user / group settings.

Unicorn seems to be very sensitive to user / group settings to check /etc/passwd and /etc/group and also use this ps line to check the difference on the executing Unicorn process for differences in the UID and GID:

ps -eo uid,gid,egid,args | grep unicorn

Note: The unicorn after hook above did not work for me. Nothing worked except for this.