0

We've been using Prometheus with fairly solid success so far. However, we've recently noticed that one of the sets of alerts we've set up ends up mislabelled: It shows up as a different environment than it is. This happens with both environments where we have multiple targets, so I'm wondering if this is a syntax (somehow? The rest are fine) or (usage of) blackbox-exporter issue.

Specifically, environments with more than one target listed have the second and subsequent targets labelled as the next environment on the list (ENV-B1's .2 and .3 show as ENV-B2 while ENV-B2's .2 and .3 show as ENV-B3). The rest are correct.

These are my configs (with various bits of info replaced):

# prometheus.yml
rule_files:
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
  # Note: Prometheus lists these in a different order, it's not clear why.
- job_name: "a client (direct)"
  metrics_path: /probe
  params:
    module: [https_health]
  static_configs:
  # ACME
  - targets: [ "http://1.1.1.1:8089/isAlive" ]
    labels:
      environment: S1
      project: ACME
      service: web-api
      importance: moderate
  - targets: [ "http://2.2.2.2:8089/isAlive" ]
    labels:
      environment: S2
      project: ACME
      service: web-api
      importance: moderate
  # B environments run multiple instances; Check all of them.
  - targets:
    - "http://3.3.3.1:8089/isAlive"
    - "http://3.3.3.2:8089/isAlive"
    - "http://3.3.3.3:8089/isAlive"
    labels:
      environment: B1
      project: ACME
      service: web-api
      importance: high
  # B environments run multiple instances; Check all of them.
  - targets:
    - "http://4.4.4.1:8089/isAlive"
    - "http://4.4.4.2:8089/isAlive"
    - "http://4.4.4.3:8089/isAlive"
    labels:
      environment: B2
      project: ACME
      service: web-api
      importance: high
  # B environments run multiple instances . . . except B3.
  - targets: [ "http://5.5.5.5:9089/isAlive" ]
    labels:
      environment: B3
      project: ACME
      service: web-api
      importance: high
# (more)
  relabel_configs:
  - source_labels: [__address__]
    target_label: __param_target
  - source_labels: [__param_target]
    target_label: instance
  - target_label: __address__
    replacement: 10.10.10.10:32222
# (etc)
# alertmanager.yml
global:
receivers:
- name: slack-devteam-acme
  slack_configs:
  - send_resolved: true
    api_url: "...url..."
    username: "Prometheus Monitoring"
    title: '{{if eq .Status "firing"}}:fire:{{else}}:successful:{{end}} {{.CommonAnnotations.summary}} {{if eq .Status "firing"}}failing{{else}}resolved{{end}}'
    title_link: "http://10.10.10.10:32600/alerts"
    text: "{{.CommonAnnotations.description}}\nInstance(s):{{range .Alerts}}\n{{.Labels.instance}} {{.Labels.environment}} {{.Labels.project}} {{.Labels.service}}{{ end }}"
# alerts.yml
- name: blackbox-export-basic
  rules:
  - alert: ProbeFailure
    expr: probe_success < 1
    for: 5m
    # labels:
    #   severity: page
    annotations:
      summary: "{{$labels.job}} probe"
      description: "{{$labels.job}} failed check for 5+ minutes."
      #description: "{{$labels.instance}} ({{$labels.job}}) failed check for 10+ minutes."
# blackbox-exporter.config.yml
modules:
  http_build_number:
    prober: http
    http:
      fail_if_body_not_matches_regexp:
      - "buildNumber"
  https_health:
    prober: http
    http:
      fail_if_body_not_matches_regexp:
      - "Success"
# (etc)
3
  • This smells like a bug, I thought we'd eliminated this class of issue. Can you file one with a minimal example? Oct 23, 2019 at 20:52
  • Oh. Y'know what I didn't consider? We're running an old version, 2.2.1 -- I don't suppose you know which was the last version such a bug was fixed in?
    – Keilaron
    Oct 23, 2019 at 20:58
  • I couldn't tell you offhand, but try the latest version. Oct 24, 2019 at 6:54

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.