Kubernetes Audit Logging Introduction

By | April 25, 2020

Explanation of Kubernetes Audit logging and an example of some policy configurations.

Overview

Kubernetes Auditing is part of the kube-apiserver, and will log all requests that the API Server processes for audit purposes.

This is what an audit log looks like:

{
  "kind":"Event",
  "apiVersion":"audit.k8s.io/v1beta1",
  "metadata":{ "creationTimestamp":"2018-03-21T21:47:07Z" },
  "level":"Metadata",
  "timestamp":"2018-03-21T21:47:07Z",
  "auditID":"20ac14d3-1214-42b8-af3c-31454f6d7dfb",
  "stage":"RequestReceived",
  "requestURI":"/api/v1/namespaces/default/persistentvolumeclaims",
  "verb":"list",
  "user": {
    "username":"[email protected]",
    "groups":[ "system:authenticated" ]
  },
  "sourceIPs":[ "172.20.66.233" ],
  "objectRef": {
    "resource":"persistentvolumeclaims",
    "namespace":"default",
    "apiVersion":"v1"
  },
  "requestReceivedTimestamp":"2018-03-21T21:47:07.603214Z",
  "stageTimestamp":"2018-03-21T21:47:07.603214Z"
}

These logs can give very useful information about what is happening in your cluster, and can even be required for compliance purposes.

This is what a basic audit logging policy looks like:

---
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
 — level: RequestResponse
 omitStages:
 — RequestReceived
 resources:
 — group: ""

level? omitStages? What the heck. Let’s explain what those terms mean!

Stages

  • RequestReceived: The stage for events generated as soon as the audit handler receives the request.
  • ResponseStarted: Once the response headers are sent, but before the response body is sent. This stage is only generated for long-running requests (e.g. watch).
  • ResponseComplete: Once the response body has been completed.
  • Panic: Events generated when a panic occurred.

Levels

  • None: don’t log events that match this rule.
  • Metadata: log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
  • Request: log event metadata and request body but not response body.
  • RequestResponse: log event metadata, request and response bodies.

Configuration

Auditing is configurable at two levels:

  • Policy: What is recorded.
  • Backends: How are records persisted and broadcast.

This is a basic policy that would log everything at the Metadata level.

---
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
 — level: Metadata
 omitStages:
 — RequestReceived

The policy below was created for GCE, but it’s a great starting point for an audit policy. It is much less verbose than the simple audit policy above, which can be costly if you’re using a SaaS centralized logging solution.

Something to note about audit policies is that when an event is processed it is compared against the audit policy rules in order, and the first matching rule sets the audit level of the event. It’s pretty weird. It would make a lot more sense to read the entire policy and apply rules according to most restrictive; like is done with Kubernetes RBAC policies.

---
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # The following requests were manually identified as high-volume and low-risk,
  # so drop them.
  - level: None
    resources:
      - group: ""
        resources:
          - endpoints
          - services
          - services/status
    users:
      - 'system:kube-proxy'
    verbs:
      - watch

  - level: None
    resources:
      - group: ""
        resources:
          - nodes
          - nodes/status
    userGroups:
      - 'system:nodes'
    verbs:
      - get

  - level: None
    namespaces:
      - kube-system
    resources:
      - group: ""
        resources:
          - endpoints
    users:
      - 'system:kube-controller-manager'
      - 'system:kube-scheduler'
      - 'system:serviceaccount:kube-system:endpoint-controller'
    verbs:
      - get
      - update

  - level: None
    resources:
      - group: ""
        resources:
          - namespaces
          - namespaces/status
          - namespaces/finalize
    users:
      - 'system:apiserver'
    verbs:
      - get

  # Don't log HPA fetching metrics.
  - level: None
    resources:
      - group: metrics.k8s.io
    users:
      - 'system:kube-controller-manager'
    verbs:
      - get
      - list

  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - '/healthz*'
      - /version
      - '/swagger*'

  # Don't log events requests.
  - level: None
    resources:
      - group: ""
        resources:
          - events

  # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    users:
      - kubelet
      - 'system:node-problem-detector'
      - 'system:serviceaccount:kube-system:node-problem-detector'
    verbs:
      - update
      - patch

  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    userGroups:
      - 'system:nodes'
    verbs:
      - update
      - patch

  # deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    omitStages:
      - RequestReceived
    users:
      - 'system:serviceaccount:kube-system:namespace-controller'
    verbs:
      - deletecollection

  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - secrets
          - configmaps
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  # Get repsonses can be large; skip them.
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
    verbs:
      - get
      - list
      - watch

  # Default level for known APIs
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
      
  # Default level for all other requests.
  - level: Metadata
    omitStages:
      - RequestReceived

Once we’ve defined a policy like above, we need to apply it to the Kubernetes API Server.

For Kops, we can apply the changes with kops edit cluster <cluster>.

spec:
  fileAssets:
  — name: kubernetes-audit
    path: /srv/kubernetes/audit.yaml
    # which type of instances to appy the file
    roles: [Master]
    content: |
      <audit policy here>
      
  kubeAPIServer:
   auditLogPath: /var/log/kube-apiserver-audit.log
   auditLogMaxAge: 10 # num days
   auditLogMaxBackups: 1 # the num of audit logs to retain
   auditLogMaxSize: 100 # the max size in MB to retain
   auditPolicyFile: /srv/kubernetes/audit.yaml

NOTE: If you’re using a cluster management tool other than Kops, you’ll need to find some way to get the audit policy on the Kubernetes master node.

In the kube configuration above, we’re using a log backend, to read about log backends and webhook backends, check the official Kubernetes documentation.

copied from -> https://medium.com/@noqcks/kubernetes-audit-logging-introduction-464a34a53f6c

Сomments аrchive