Configure datadog-agent for gunicorn metrics

Question

I have datadog-agent installed on my Debian 8 server. It is already configured and works well to report metrics about postgres, nginx, system, etc.

I want to monitor my gunicorn daemons (i have 2 Django websites on this server). As far as I understand, datadog-agent already integrates a statsd server, so I don't have to install one:

# ps ax | grep datadog
18816 ?        Ss     0:00 /opt/datadog-agent/embedded/bin/python /opt/datadog-agent/bin/supervisord -c /etc/dd-agent/supervisor.conf
18822 ?        Sl     0:01 /opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/dogstatsd.py --use-local-forwarder
18824 ?        S      0:01 /opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/ddagent.py
18825 ?        S      0:01 /opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/agent.py foreground --use-local-forwarder

I think "dogstatsd.py" is the statsd server, but maybe I am wrong ?

Then, I modified my gunicorn startup script to integrates --name option to set a name to my process and --statsd-host to indicates where to send metrics.

# ps ax | grep gunicorn
18588 ?        Ss     0:00 /var/projects/my_project/venv/bin/python3 /var/projects/my_project/venv/bin/gunicorn --name my_project --statsd-host=localhost:8125 --workers 2 --bind unix:/var/tmp/my_project.sock core.wsgi:application
18630 ?        S      0:00 /var/projects/my_project/venv/bin/python3 /var/projects/my_project/venv/bin/gunicorn --name my_project --statsd-host=localhost:8125 --workers 2 --bind unix:/var/tmp/my_project.sock core.wsgi:application
18632 ?        S      0:00 /var/projects/my_project/venv/bin/python3 /var/projects/my_project/venv/bin/gunicorn --name my_project --statsd-host=localhost:8125 --workers 2 --bind unix:/var/tmp/my_project.sock core.wsgi:application

Now I created gunicorn.yaml from example in /etc/dd-agent/conf.d/ as follow

# NB: This check requires the python environment on which gunicorn runs to
# have the `setproctitle` module installed (https://pypi.python.org/pypi/setproctitle/)

init_config:

instances:
  # The name of the gunicorn process. For the following gunicorn server ...
  #
  #    gunicorn --name my_web_app my_web_app_config.ini
  #
  #  ... we'd use the name `my_web_app`.
  #
  - proc_name: my_project

After restarting agent, I wait a few seconds and I check its status:

# sudo service datadog-agent info
# [...]
  Checks
  ======

    gunicorn
    --------
      - instance #0 [ERROR]: 'Found no master process with name: gunicorn: master [my_project]'
      - Collected 0 metrics, 0 events & 1 service check
      - Dependencies:
          - psutil: 4.4.1

I can't find where I fail to configure this. Can somebody help me ?

Answer 1

After connecting to Datadog IRC, somebody explained that the gunicorn process should have a pretty printed name. The solution was to install setproctitle using pip in the same venv than gunicorn.

Obviously, the answer was in the question, at the top of "gunicorn.yaml" file:

NB: This check requires the python environment on which gunicorn runs to have the setproctitle module installed (https://pypi.python.org/pypi/setproctitle/)

Answer 2

You must install setproctitle package to your app :

# ps aux|grep gunicorn
user 919 0.2 0.2 004 672 0:00 gunicorn --name my_app app:app --workers 5 --bind 
127.0.0.1:5000 --log-level info
user 925 0.6 0.6 052 688 0:01 gunicorn --name my_app app:app --workers 5 --bind 
127.0.0.1:5000 --log-level info
user 926 0.7 0.6 308 032 0:01 gunicorn --name my_app app:app --workers 5 --bind                 
127.0.0.1:5000 --log-level info
user 927 0.4 0.6 884 404 0:00 gunicorn --name my_app app:app --workers 5 --bind 
127.0.0.1:5000 --log-level info

# cd my_app
# source ./venv/bin/activate
# pip install setproctitle
# systemctl restart gunicorn

# ps aux|grep gunicorn
user 468 0.1 0.2 068 712 0:00 gunicorn: master [my_app]
user 471 0.4 0.6 376 852 0:00 gunicorn: worker [my_app]
user 472 0.4 0.6 120 844 0:00 gunicorn: worker [my_app]
user 473 0.4 0.6 944 372 0:00 gunicorn: worker [my_app]
user 474 0.5 0.6 384 976 0:01 gunicorn: worker [my_app]
user 475 0.5 0.6 396 992 0:01 gunicorn: worker [my_app]

You can see first ps where is no master and only workers are visible. After pip install setproctitle to my app environment gunicorn processes are now visible with master/worker names. You can check it with this command : sudo -u dd-agent -- datadog-agent check gunicorn