OpenWare OPEX

rk_logo

OPEX Logging & Monitoring with Stackdriver

img

Table of contents

[TOC]

Introduction

Google Stackdriver provides a wide variety of metrics, dashboards, alerting, log management, reporting, and tracing capabilities. It is natively integrated with Google Cloud Platform.

Logging

GCP Console

GCP kubernetes clusters are natively integrated to Stackdriver. You can browse application and services logs directly from the GCP console.

console logs

Command line tool

One of the possible ways to view logs is using the gcloud logging command line interface from the Google Cloud SDK. It uses Stackdriver Logging filtering syntax to query specific logs.

List the log names of the current project

$ gcloud logging logs list
NAME
projects/helios-stage/logs/barong
projects/helios-stage/logs/cloudaudit.googleapis.com%2Factivity
projects/helios-stage/logs/cloudaudit.googleapis.com%2Fdata_access
projects/helios-stage/logs/cloudsql.googleapis.com%2Fmysql.err
projects/helios-stage/logs/coinhub
projects/helios-stage/logs/compute.googleapis.com%2Factivity_log
projects/helios-stage/logs/count
projects/helios-stage/logs/dnsmasq
projects/helios-stage/logs/docker
projects/helios-stage/logs/enterprise-front
projects/helios-stage/logs/event-exporter
projects/helios-stage/logs/events
projects/helios-stage/logs/fluentd-gcp
projects/helios-stage/logs/heapster
projects/helios-stage/logs/kube-lego
projects/helios-stage/logs/kube-proxy
projects/helios-stage/logs/kubedns
projects/helios-stage/logs/kubelet
projects/helios-stage/logs/kubernetes-dashboard
projects/helios-stage/logs/nginx-ingress-controller
projects/helios-stage/logs/node-problem-detector
projects/helios-stage/logs/peatio
projects/helios-stage/logs/peatio-daemons
projects/helios-stage/logs/peatio-websocket-api
projects/helios-stage/logs/prepare-db
projects/helios-stage/logs/prometheus-to-sd-exporter
projects/helios-stage/logs/redis-redis
projects/helios-stage/logs/relay
projects/helios-stage/logs/run-rake-tasks
projects/helios-stage/logs/sidecar
projects/helios-stage/logs/solvency
projects/helios-stage/logs/sqlproxy
projects/helios-stage/logs/tiller

Read all log of a component:

$ gcloud logging read 'logName = projects/helios-stage/logs/barong' --limit 10

Format the output to get the message only:

$ gcloud logging read 'logName = projects/helios-stage/logs/barong' --limit 10 --format json | jq '.[].textPayload'

Select a specific time period:

$ gcloud logging read 'timestamp >= "2018-05-16T22:34:20Z" AND timestamp <= "2018-05-16T22:35:20Z" AND logName = projects/helios-stage/logs/barong