How can I find out which program is executed through a PID which is ended

Question

I found there is a program that killed my Apache regularly, according to the log information, I understand there is a program using "/etc/init.d/apache2 stop" to stop apache, since this process is gone, how can I find out this PID belongs to which program?

journalctl -o verbose _PID=16630
-- Logs begin at Thu 2021-10-28 ... 
    [s=b8b9ba8c0b3a434ab134b8e39ad9a421;...]
    SYSLOG_FACILITY=3
    PRIORITY=6
    _UID=0
    _GID=0
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=dac5e28c70d04920ad8b140efa
    _MACHINE_ID=b66fd2c1b04547d4b1471c0e
    _HOSTNAME=server
    _CAP_EFFECTIVE=3fffffffff
    _TRANSPORT=stdout
    _CMDLINE=/bin/sh /etc/init.d/apache2 stop
    _EXE=/bin/dash
    SYSLOG_IDENTIFIER=apache2
    _COMM=apache2
    _SYSTEMD_UNIT=apache2.service
    MESSAGE= * Stopping Apache httpd web server apache2

Answer 1

You should check dmesg for kernel output. A likely culprit is kernel Out-Of-Memory handler, that picks up a process and ends it when there is system memory pressure.

This event is logged in the kernel log shown by dmesg.

Answer 2

Once a process has terminated all information about it is gone except what has been logged while it was running. The information you are looking for (which command has been running with that PID) simply doesn't exist anymore.

Your only chance is to log the information while the process still runs. If you are sure the killing happens by executing the command /etc/init.d/apache2 stop then one avenue would be to add logging to the script /etc/init.d/apache2. Just inserting a line like

 /bin/ps axf >> /tmp/apachekillertracker.log

somewhere near the beginning of the script might already do the trick as it will dump the process tree to /tmp/apachekillertracker.log each time /etc/init.d/apache2 is run for you to inspect later.

If it turns out the killing is not done through /etc/init.d/apache2 after all then you could look into process accounting to find out what was running at the time the killing happened.