Skip to content

Logging#

Stakater App Agility Platform uses EFK Stack (ElasticSearch, Fluentd and Kibana) to provide logging for applications. Fluentd daemonsets pick up the logs and send these to ElasticSearch. Kibana dashboards can be used to view/analyze logs

Logging

Components#

Here are the logging stack components:

  • Fluentd
  • ElasticSearch
  • Kibana
  • Event Router

Parse JSON Application Logs#

Logs are parsed by default if applications output logs in JSON format on stdout. Moreover one step nested JSON parsing is also supported additionally.

Consider the following example of a one line event by a java application:

{"timestamp":"2021-04-15 11:41:01.427","level":"WARN","thread":"http-nio-8080-exec-4","mdc":{"breadcrumbId":"441ce707-8096-4aba-a927-0afa8c34802b-by-BOKE","user":"service-account-boke"},"logger":"org.zalando.logbook.Logbook","message":"{\"origin\":\"local\",\"type\":\"response\",\"correlation\":\"ef4f3737f2bcf856\"}"}

This will be parsed as follows:

{
    "timestamp":"2021-04-15 11:41:01.427",
    "level":"WARN",
    "thread":"http-nio-8080-exec-4",
    "mdc.breadcrumbId":"441ce707-8096-4aba-a927-0afa8c34802b-by-BOKE",
    "mdc.user":"service-account-boke",
    "logger":"org.zalando.logbook.Logbook",
    "message":"{\"origin\":\"local\",\"type\":\"response\",\"correlation\":\"ef4f3737f2bcf856\"}",
    "origin": "local",
    "type": "response",
    "correlation": "ef4f3737f2bcf856"
}

Parse non JSON Application Logs#

Parsing application logs which are not in JSON format can be done as follows:

Consider the following example of a one line event by a java application:

2019-11-27 11:04:12.682  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'

The configuration to parse/match/send logs can be specified in the Application Chart, by specifying regular expressions as described below:

Parameter Description
.Values.deployment.fluentdConfigAnnotations.regexFirstLine specify the regex to match the first line of the log
.Values.deployment.fluentdConfigAnnotations.regex specify the regex to parse the complete log entry
.Values.deployment.fluentdConfigAnnotations.timeFormat specify the regex to parse time

Use the following Configuration for parsing java springboot logs :

deployment:
  fluentdConfigAnnotations:
    regex: /^(?<time>\\d+(?:-\\d+){2}\\s+\\d+(?::\\d+){2}\\.\\d+)\\s*(?<level>\\S+)
      (?<pid>\\d+) --- \\[(?<thread>[\\s\\S]*?)\\] (?<class>\\S+)\\s*:\\s*(?<message>[\\s\\S]*?)(?=\\g<time>|\\Z)/
    regexFirstLine: /^\\d+(?:-\\d+){2}\\s+\\d+(?::\\d+){2}\\.\\d+/
    timeFormat: "%Y-%m-%d %H:%M:%S.%L"

This will be parsed as follows:

time: 2019-11-27 11:04:12.682
level: INFO
pid: 1
thread: nio-8080-exec-1
class: o.s.web.servlet.DispatcherServlet
message: Initializing Servlet 'dispatcherServlet'

Log Retention#

By default Application logs are retained for 7 days.

Application alerting#

Alerts can be sent to Slack channels by matching a string against a particular field. e.g. Send an alert to Slack if level==ERROR. These alerts increase operational efficiency. See Application log alerting on how to configure alerts