top is only showing current user processes

Question

Recently got a dedicated server running CentOS 6.7, we have ran updates and noticed that top is only showing processes for the current user.

[myuser@server2 ~]$ top -b -n1 
top - 20:19:20 up 1 day, 10:09,  3 users,  load average: 0.80, 0.50, 0.41
Tasks:  11 total,   1 running,  10 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32880988k total, 26893324k used,  5987664k free,   140872k buffers
Swap:  1046520k total,        0k used,  1046520k free, 19532120k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND               
 1648 myuser  20   0 98.8m 1020  688 S  0.0  0.0   0:00.00 man                   
 1651 myuser  20   0  103m 1184 1016 S  0.0  0.0   0:00.00 sh                    
 1652 myuser  20   0  103m  684  500 S  0.0  0.0   0:00.00 sh                    
 1656 myuser  20   0  103m  912  752 S  0.0  0.0   0:00.07 less                  
 3363 myuser  20   0  100m 1708  700 S  0.0  0.0   0:00.04 sshd                  
 3364 myuser  20   0  105m 1916 1524 S  0.0  0.0   0:00.00 bash                  
 8337 myuser  20   0 14940 1096  880 R  0.0  0.0   0:00.00 top                   
30429 myuser  20   0  100m 1696  696 S  0.0  0.0   0:00.16 sshd                  
30430 myuser  20   0  105m 1924 1536 S  0.0  0.0   0:00.01 bash                  
31132 myuser  20   0  100m 1692  692 S  0.0  0.0   0:00.05 sshd                  
31133 myuser  20   0  105m 1928 1536 S  0.0  0.0   0:00.04 bash            

However when ran with sudo i get the normal output I would expect

[myuser@server2 ~]$ sudo top -bn1
top - 20:22:08 up 1 day, 10:12,  3 users,  load average: 0.36, 0.40, 0.39
Tasks: 166 total,   1 running, 165 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32880988k total, 26898188k used,  5982800k free,   141196k buffers
Swap:  1046520k total,        0k used,  1046520k free, 19532188k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND               
32705 otherusr  20   0 21.9g 5.8g  15m S 39.8 18.3  28:47.76 java                  
    1 root      20   0 19280 1524 1232 S  0.0  0.0   0:00.76 init                  
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd              
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.40 ksoftirqd/0           
    5 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 kworker/0:0H          
    6 root      20   0     0    0    0 S  0.0  0.0   0:03.20 kworker/u16:0  
~~~~~omitted~~~~~

I have tried running top not in batch mode and using the "u" & "U" and leaving it blank and pressing [enter], with no luck.

I'm pretty sure I'm running actual top, launching with absolute path had no impact.

$ which top
/usr/bin/top
$ file /usr/bin/top
/usr/bin/top: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
$ alias
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias vi='vim'
alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'

No toprc is present in /etc/

$ ls -a /etc | grep -c toprc
0

proc is mounted as follows

$ sudo cat /proc/mounts | grep proc
none /proc proc rw,relatime 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
$ sudo cat /etc/fstab | grep proc
proc        /proc   proc    defaults        0   0

More details on Procs

[myuser@server2 ~]$ sudo -u otherusr ps -A
  PID TTY          TIME CMD
21921 pts/0    00:00:00 ps
32703 ?        00:00:00 screen
32705 pts/3    01:08:01 java
[myuser@server2 ~]$ sudo ls -l /proc/ | grep 32705
dr-xr-x---  7 otherusr root     0 Sep 23 19:27 32705
[myuser@server2 ~]$ sudo ls -l /proc/32705/ | grep stat
-r--------  1 otherusr root 0 Sep 23 21:54 mountstats
-r--r--r--  1 otherusr root 0 Sep 23 19:27 stat
-r--r--r--  1 otherusr root 0 Sep 23 19:27 statm
-r--r--r--  1 otherusr root 0 Sep 23 19:27 status
[myuser@server2 ~]$ ls /proc/ | egrep "[0-9]{1,9}";
17363
17364
26124
26125
3363
3364
[myuser@server2 ~]$ sudo -u otheruser ls /proc/ | egrep "[0-9]{1,9}"
26132
32703
32705
[myuser@server2 ~]$ umask
0002
[root@server2 ~]# umask
0022

Any Ideas?

Answer 1

One possibility is /proc mounted with either hidepid=1 or hidepid=2. This mount option was added in latter Linux kernels and back ported sometime around CentOS 5.9 and 6.3.

Mount options
   The proc filesystem supports the following mount options:

   hidepid=n (since Linux 3.3)
          This option controls who can access the information in
          /proc/[pid] directories.  The argument, n, is one of the
          following values:

          0   Everybody may access all /proc/[pid] directories.  This is
              the traditional behavior, and the default if this mount
              option is not specified.

          1   Users may not access files and subdirectories inside any
              /proc/[pid] directories but their own (the /proc/[pid]
              directories themselves remain visible).  Sensitive files
              such as /proc/[pid]/cmdline and /proc/[pid]/status are now
              protected against other users.  This makes it impossible
              to learn whether any user is running a specific program
              (so long as the program doesn't otherwise reveal itself by
              its behavior).

          2   As for mode 1, but in addition the /proc/[pid] directories
              belonging to other users become invisible.  This means
              that /proc/[pid] entries can no longer be used to discover
              the PIDs on the system.  This doesn't hide the fact that a
              process with a specific PID value exists (it can be
              learned by other means, for example, by "kill -0 $PID"),
              but it hides a process's UID and GID, which could
              otherwise be learned by employing stat(2) on a /proc/[pid]
              directory.  This greatly complicates an attacker's task of
              gathering information about running processes (e.g.,
              discovering whether some daemon is running with elevated
              privileges, whether another user is running some sensitive
              program, whether other users are running any program at
              all, and so on).

Another possibility (found by the poster and added to this answer as reference information) is grsecurity which has a feature for hiding other users' processes from unprivileged users as part of it's file system hardening.

Hide other users' processes for unprivileged users

While the upstream kernel now provides a mount option for /proc to hide other unprivileged users' processes, grsecurity goes beyond this by hiding such information by default, hiding additional sources of sensitive information provided by the kernel in /proc, and hiding private network-related information of all users. Not only is the networking information a violation of the privacy of other users on the system, but it has also been useful in the past for TCP hijacking attacks.

Answer 2

Based on the information from the following posts I have identified 3 possible resolutions.

• htop shows only the processes of the user that's running it? on Unix & Linux
• https://www.centos.org/forums/viewtopic.php?f=14&t=52563

Which states grsecurty can cause the users not being able to see other users pids in /proc/ and that OVH (My hosting company) uses a custom kernel compiled with Grsecurity

$ uname -r
3.14.32-xxxx-grs-ipv6-64

possible resolutions are:

1. remove grsecurity
2. edit policy to permit such use
3. ensure the other admins are made aware of the new security measures in place

For me educating the other admins is be best option as security comes first. Still kinda wish they would have informed us of something like this though. Thank you for all your help!

Answer 3

I do it in multiple steps being logged in as root user.

1. To find all the logged in users last | grep 'logged'

2. Check activity of specific user by top -u username1 and top -u username2 etc.