I would like to monitor request processing latency on my Tomcat server. It would be nice to perform some alerting based on this later. I configured Tomcat valve with %D pattern so it will log time taken to process each request in access log file. What would be correct way to calculate 50, 90, 99 percentiles from these log lines? I already have configured Prometheus and Grafana for basic host metrics(cpu/mem,hdd), but it is not clear to me how to correctly calculate percentiles and graph them with prometheus/grafana? Idea is to detect application hicckups and react before it goes totally unresponsive. Thank you all in advance!
You could use my tomcat_exporter to do this by configuring the filter for the response time metrics. Alternatively you could implement a similar filter by yourself.