Get Kubernetes Cluster Metrics with Prometheus in 5 Minutes

By | July 26, 2018

Get Kubernetes Cluster Metrics with Prometheus in 5 Minutes

Having a Kubernetes cluster up and running is pretty easy these days. But, when you start to use the cluster and deploy some applications you might expect some issues over time. Kubernetes being a distributed system is not easy to troubleshoot. You need a good monitoring solution and because the Prometheus is CNCF project as Kubernetes it is probably the best fit. In this post, I will show you how to get the Prometheus running and start monitoring your Kubernetes cluster in 5 minutes.

Prometheus Operator
CoreOS introduced operators as a business logic in the first place. I wrote about Elasticsearch operator and how it works a few months ago so you might check it out. In my opinion, operators are the best way to deploy stateful applications on Kubernetes.

CoreOS team also provided Prometheus operator that I will use for deployment. Here is the official operator workflow and relationships view:

prometheus_operator_workflow

From the picture above you can see that you can create a ServiceMonitor resource which will scrape the Prometheus metrics from the defined set of pods. For example, if you have a frontend app which exposes Prometheus metrics on port web, all you need to do is to create a service monitor which will configure Prometheus server:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: frontend-app
labels:
app: frontend-app
spec:
selector:
matchLabels:
app: frontend-app
endpoints:
– port: web
interval: 10s
Installing operator is pretty easy with Helm. Let’s add CoreOS repository and install it:

⚡ helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/

⚡ helm install \
–name prometheus-operator \
–namespace monitoring \
–set rbacEnable=false \
coreos/prometheus-operator

⚡ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
prometheus-operator-67f87d659c-rtpwq 1/1 Running 0 1m
When you install the Prometheus operator you will get the new Custom Resource Definitions or CRDs. You can check that with this command:

⚡ kubectl get CustomResourceDefinition
NAME AGE
alertmanagers.monitoring.coreos.com 1m
prometheuses.monitoring.coreos.com 1m
servicemonitors.monitoring.coreos.com 1m
As you can see, the Prometheus operator will manage alert manager, Prometheus server, and service monitors.

Prometheus Installation
For Prometheus installation I will use the Helm chart kube-prometheus. This chart has a lot of options, so I encourage you to take a look at default values file and override some values if needed. Among other services, this chart installs Grafana and exporters ready to monitor your cluster. kube-prometheus is an umbrella chart with many dependencies that you can find in requirements file.

I will enable persistent storage for all components and disable RBAC. You should have RBAC, but in my test cluster, it is not enabled. I will expose Grafana with Ingress, so I disabled anonymous authentication and changed the admin password. This is my custom values file:

⚡ cat > custom-values.yaml <

Сomments аrchive