1

I am running my custom service, let's call it "foo.service". It's a gunicorn python process which logs a few things.

I used to be able to get a real-time log overview using journalctl -u <service> -f but now the journal seems to be stuck at some past logs. When I use systemctl status <service> it does show the latest logs though. So my logging does work, but journalctl seems to be stuck and doesn't show any updates.

Example:

journalctl -u foo.service -f

Nov 25 16:19:09 <name> systemd[1]: Started Instance to load up the program and its endpoints.
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,844 [INFO]: Connecting to localhost:9773
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,845 [INFO]: Connecting to localhost:9771
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,846 [INFO]: Connected
Nov 25 16:19:09 <name> gunicorn[28267]: 2019-11-25 16:19:09,846 [INFO]: Connected

systemctl status foo.service

Nov 26 11:39:53 <name> systemd[1]: Started Instance to load up the program and its endpoints.
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,458 [INFO]: Connecting to localhost:9773
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,459 [INFO]: Connecting to localhost:9771
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,460 [INFO]: Connected
Nov 26 11:39:53 <name> gunicorn[29117]: 2019-11-26 11:39:53,460 [INFO]: Connected

The latter is logging that was just created, so that works like a charm, but from my understanding the journalctl seems to be updated as well. This did work in the past, I am encountering these problems since the day before yesterday.

I tried restarting journalctl but that didn't seem to work.

Thanks in advance.

EDIT: When I restart my service I notice gunicorn threw an exception when I look at the status of the systemctl service, I cannot scroll up so I have no clue as to what the cause of the error is (systemctl status):

Nov 26 15:08:27 <name> gunicorn[29390]:     self.log.info("Shutting down: %s", self.master_name)
Nov 26 15:08:27 <name> gunicorn[29390]:   File "/opt/my-program/venv/lib/python3.5/site-packages/gunicorn/glogging.py", line 271, in info
Nov 26 15:08:27 <name> gunicorn[29390]:     self.error_log.info(msg, *args, **kwargs)
Nov 26 15:08:27 <name> gunicorn[29390]: Message: 'Shutting down: %s'
Nov 26 15:08:27 <name> gunicorn[29390]: Arguments: ('Master',)
Nov 26 15:08:27 <name> systemd[1]: Started Instance to load up the program and its endpoints.

My foo.service configuration file:

[Unit]
Description=Instance to load up the program and its endpoints
After=network.target

[Service]
User=root
Group=www-data
WorkingDirectory=/opt/my-program/my-program-thing
Environment="PATH=/opt/my-program/venv/bin"
ExecStart=/opt/my-program/venv/bin/gunicorn --workers 1 --threads 12 --bind unix:foo.sock -m 007 app:app --bind 0.0.0.0:8085 --access-logfile '/var/log/foo.info.log' --error-logfile '/var/log/foo.err.log'
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
1

Try adding these lines to your service file under the [Service] tag

StandardOutput=journal StandardError=journal

Then do a systemctl daemon-reload, then do a systemctl restart your_service.service

I edit the answer for others to see it easily. Problem was lack of space on /var filesystem so journalctl was not able to save logs to disk and read afterwards, that is why there was a difference between systemctl status and journalctl -fu .

4
  • Tried it, I do not notice any differences. I updated my original post! – Dubb Nov 26 '19 at 14:16
  • 1
    You can add -n 30 to systemctl status your_service.service to see more lines, like this: systemctl status your_service.service -n 30 --no-pager – Zatarra Nov 26 '19 at 15:10
  • Yes, thank you! I received an IOError telling me the disk was full. After checking it with 'du h' I noticed the /var/ directory was 100% full. I cleared some logs and it solved the issue. – Dubb Nov 26 '19 at 18:11
  • 1
    Glad that you got it fixed. I've edited the answer so others can identify the solution next time. – Zatarra Nov 27 '19 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.