Skip to content

Enable Alerts for your Application#

In the modern landscape of software development, monitoring and managing the health and performance of your applications is crucial. To ensure seamless operations and timely responses to issues, setting up alerts and notifications is essential. In this tutorial, we will guide you through the process of enabling alerts and notifications for your application using Prometheus and Alertmanager within your SAAP (Stakater App Agility Platform) cluster.


  • Establish PrometheusRule Custom Resources to define conditions for triggering alerts based on specific metrics.
  • Configure the Alertmanager to handle alerts and send notifications to the appropriate channels, such as Slack.
  • Establish a Secret to securely store and manage the webhook URL for sending alerts to Slack.

Key Results#

  • Confirm that alerts are firing correctly by observing the alerts in the SAAP and notifying Slack.

Now that we have enabled Service Monitor for our application in the previous section, let's create alerts for it. Metrics endpoints are scraped via ServiceMonitor by Prometheus and Prometheus is already installed on your SAAP cluster.


Defining PrometheusRule#

If you want to generate an alert based on some metric, you will need a PrometheusRule Custom Resource. A PrometheusRule defines when an alert should fire.

Now let's add a PrometheusRule for the application. In the previous section, we added a custom metric that records the review. We are going to use the custom metric to write a Prometheus rule that fires when we get too many low ratings.

  1. You need to add the following lines to your deploy/values.yaml file:

        enabled: true
          prometheus: stakater-workload-monitoring
          - name: nordmart-review-low-rating-warning
              - alert: NordmartReviewLowRatingsCritical
                  message: >-
                    Total ratings below 2 has crossed the threshold 1. Total reviews: {{ $value }}.
                expr: >
                  sum by (namespace) (nordmart_review_ratings_total{rating="2"} or nordmart_review_ratings_total{rating="1"}) > 1
                  severity: critical


    The indentation follows by application.prometheusRule.

Alert Manager#

Now we need to tell Alert Manager where to send the alert. For this, we will need to add an AlertManagerConfig.

  1. If you need to send an alert to a Slack channel. You will first need to add a webhook for that channel in Slack Once you have the webhook Url, you can add the AlertManagerConfig. The Alertmanager uses a secret to pick up details of the endpoint to send the alerts to.

  2. Let's create the secret first. Log in to SAAP > Administrator > Workloads > Secrets in your namespace. Create a secret from YAML. Replace "namespace" with the namespace in which your application is deployed and "api_url" with base64 encoded webhook URL:

    kind: Secret
    apiVersion: v1
      name: review-slack-webhook
      namespace: <namespace>
      api_url: >-
    type: Opaque
  3. Let's add the AlertManagerConfig, add this YAML to deploy/values.yaml, and remember to replace "channel-name" with your channel name.

        enabled: true
          alertmanagerConfig: workload
            - name: nordmart-review-receiver
                - apiURL:
                    key: api_url
                    name: review-slack-webhook
                  channel: '<#channel-name>'
                  sendResolved: true
                  text: |2-
                    {{ range .Alerts }}
                    *Alert:* `{{ .Labels.severity | toUpper }}` - {{ .Annotations.summary }}
                    *Description:* {{ .Annotations.description }}
                    *Details:* {{ range $k, $v := .Labels }} - *{{ $k }}:* {{ $v }}
                    {{ end }}
                   {{ end }}
                  title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] SAAP Alertmanager Event Notification'
                      insecureSkipVerify: true
              - alertname
              - severity
            groupInterval: 3m
            groupWait: 30s
            repeatInterval: 1h
              - name: alertname
                value: NordmartReviewLowRatingsCritical
            receiver: nordmart-review-receiver

!!! note The indentation follows by application.alertmanagerConfig.

  1. Save and run tilt up at the root of your directory. Hit the space bar and the browser with TILT logs will be shown. If everything is green then the changes will be deployed on the cluster.

    Now that we have created everything we need, let's see the alerts firing. Log in to SAAP, and change the view to "Developer". You will see the 'Observe' tab in the left panel.

    Alerts firing

    Let's see if you got the notification on your channel:

    slack notification

Isn't it Awesome!!! Let's move to the next chapter.