Get Kubernetes Cluster Metrics with Prometheus in 5 Minutes
Having a Kubernetes cluster up and running is pretty easy these days. But, when you start to use the cluster and deploy some applications you might expect some issues over time. Kubernetes being a distributed system is not easy to troubleshoot. You need a good monitoring solution and because the Prometheus is CNCF project as Kubernetes it is probably the best fit. In this post, I will show you how to get the Prometheus running and start monitoring your Kubernetes cluster in 5 minutes.
CoreOS introduced operators as a business logic in the first place. I wrote about Elasticsearch operator and how it works a few months ago so you might check it out. In my opinion, operators are the best way to deploy stateful applications on Kubernetes.
CoreOS team also provided Prometheus operator that I will use for deployment. Here is the official operator workflow and relationships view:
From the picture above you can see that you can create a ServiceMonitor resource which will scrape the Prometheus metrics from the defined set of pods. For example, if you have a frontend app which exposes Prometheus metrics on port web, all you need to do is to create a service monitor which will configure Prometheus server:
– port: web
Installing operator is pretty easy with Helm. Let’s add CoreOS repository and install it:
⚡ helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/
⚡ helm install \
–name prometheus-operator \
–namespace monitoring \
–set rbacEnable=false \
⚡ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
prometheus-operator-67f87d659c-rtpwq 1/1 Running 0 1m
When you install the Prometheus operator you will get the new Custom Resource Definitions or CRDs. You can check that with this command:
⚡ kubectl get CustomResourceDefinition
As you can see, the Prometheus operator will manage alert manager, Prometheus server, and service monitors.
For Prometheus installation I will use the Helm chart kube-prometheus. This chart has a lot of options, so I encourage you to take a look at default values file and override some values if needed. Among other services, this chart installs Grafana and exporters ready to monitor your cluster. kube-prometheus is an umbrella chart with many dependencies that you can find in requirements file.
I will enable persistent storage for all components and disable RBAC. You should have RBAC, but in my test cluster, it is not enabled. I will expose Grafana with Ingress, so I disabled anonymous authentication and changed the admin password. This is my custom values file:
⚡ cat > custom-values.yaml <