1

We have a live AWS EC2 instance with Ubuntu 14.04, which I have been handed to maintain. An oddity that I have found is that I have a process of /tmp/systemd using all available CPU:

top - 11:35:20 up  2:34,  1 user,  load average: 1.13, 1.16, 1.15
Tasks: 114 total,   2 running, 111 sleeping,   0 stopped,   1 zombie
%Cpu(s): 32.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 67.3 st
KiB Mem:   2048516 total,   726492 used,  1322024 free,    38784 buffers
KiB Swap:        0 total,        0 used,        0 free.   439788 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 1797 www-data  20   0  192900   4008   1140 S 95.2  0.2 107:42.82 /tmp/systemd
 1145 mysql     20   0  681596 121328   7560 S  3.6  5.9   2:35.84 /usr/sbin/mysqld
 3669 ubuntu    20   0  103084   1904    920 S  0.7  0.1   0:00.06 sshd: ubuntu@pts/0
   40 root      rt   0       0      0      0 S  0.3  0.0   0:04.96 [watchdog/0]
 3692 ubuntu    20   0   23728   1652   1104 R  0.3  0.1   0:00.11 top
    1 root      20   0   33556   2880   1480 S  0.0  0.1   0:03.20 /sbin/init
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 [kthreadd]

I thought the /tmp/systemd seemed a tad odd for a command source, so I checked the /tmp directory - and it's completely empty.

Running an strace seems to indicate that it is polling for something to finish:

ubuntu@ip-10-0-0-157:~$ sudo strace -fvvp 1797
Process 1797 attached with 6 threads
[pid  1801] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid  1802] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid  1803] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid  1800] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 290556741}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] clock_gettime(CLOCK_REALTIME, {1524656311, 141773213}) = 0
[pid  1797] <... epoll_wait resumed> {}, 1024, 27) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 328709201}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 328773526}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1797] <... epoll_wait resumed> {}, 1024, 500) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 839857928}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 839934892}) = 0
[pid  1797] epoll_wait(7, {}, 1024, 18) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 860012749}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 860081346}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1797] <... epoll_wait resumed> {}, 1024, 479) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9449, 350734613}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9449, 350806967}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
q[pid  1797] <... epoll_wait resumed> {}, 1024, 500) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9449, 905946457}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9449, 906007520}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0

I've tried updating and rebooting the server, but it changes nothing. I can kill the process, but approximately 1 hour later it will restart.

Can you please advise on how to locate source of this or how I can find out what it is doing and what is starting it?

  • 3
    There is no /tmp/systemd binary or temporary binary, given that the user which started the process is non-interactive "www-data" I would suggest that your webserver was compromised. Check for any suspicions outbound connections, re-mount /tmp with noexec and check if everything owned by www-data is as it should be. Looks like Tiny XMR mooner - xorl.wordpress.com/2017/12/21/… – Daniel Apr 25 '18 at 13:20
  • @Daniel this answer seems spot on. Should be an answer and not a comment. – Mark Stosberg Apr 25 '18 at 13:46
  • Thank you - and yes, if you create this as an answer, I'll mark it complete - thanks again. I've remounted /tmp with noexec, nodev and nosuid, and I'm looking to see how they have gained entry into the server. – Zoek UK Apr 25 '18 at 14:11
1

There is no /tmp/systemd binary or temporary binary, given that the user which started the process is non-interactive "www-data" I would suggest that your webserver was compromised.

Check for any suspicions outbound connections, re-mount /tmp with noexec and check if everything owned by www-data is as it should be. Look for shell scripts or other executables that should not belong to www-data.

Looks like Tiny XMR mooner - https://xorl.wordpress.com/2017/12/21/the-tiny-xml-mooner-linux-cryptominer-malware/

Best practice is such cases is to make image of the machine (with dd for example) and analyze it offline in environment with no Internet access.

  • Best practice is to take the machine offline immediately, analyze it, and then destroy it. You can never be sure you have fully cleaned a compromised server, and there is little to be gained by trying. serverfault.com/q/218005/153161 – Michael - sqlbot Apr 26 '18 at 0:39

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.