Explanation of Kubernetes Audit logging and an example of some policy configurations.
Overview
Kubernetes Auditing is part of the kube-apiserver, and will log all requests that the API Server processes for audit purposes.
This is what an audit log looks like:
{
"kind":"Event",
"apiVersion":"audit.k8s.io/v1beta1",
"metadata":{ "creationTimestamp":"2018-03-21T21:47:07Z" },
"level":"Metadata",
"timestamp":"2018-03-21T21:47:07Z",
"auditID":"20ac14d3-1214-42b8-af3c-31454f6d7dfb",
"stage":"RequestReceived",
"requestURI":"/api/v1/namespaces/default/persistentvolumeclaims",
"verb":"list",
"user": {
"username":"[email protected]",
"groups":[ "system:authenticated" ]
},
"sourceIPs":[ "172.20.66.233" ],
"objectRef": {
"resource":"persistentvolumeclaims",
"namespace":"default",
"apiVersion":"v1"
},
"requestReceivedTimestamp":"2018-03-21T21:47:07.603214Z",
"stageTimestamp":"2018-03-21T21:47:07.603214Z"
}These logs can give very useful information about what is happening in your cluster, and can even be required for compliance purposes.
This is what a basic audit logging policy looks like:
---
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
— level: RequestResponse
omitStages:
— RequestReceived
resources:
— group: ""level? omitStages? What the heck. Let’s explain what those terms mean!
Stages
- RequestReceived: The stage for events generated as soon as the audit handler receives the request.
- ResponseStarted: Once the response headers are sent, but before the response body is sent. This stage is only generated for long-running requests (e.g. watch).
- ResponseComplete: Once the response body has been completed.
- Panic: Events generated when a panic occurred.
Levels
- None: don’t log events that match this rule.
- Metadata: log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
- Request: log event metadata and request body but not response body.
- RequestResponse: log event metadata, request and response bodies.
Configuration
Auditing is configurable at two levels:
- Policy: What is recorded.
- Backends: How are records persisted and broadcast.
This is a basic policy that would log everything at the Metadata level.
---
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
— level: Metadata
omitStages:
— RequestReceivedThe policy below was created for GCE, but it’s a great starting point for an audit policy. It is much less verbose than the simple audit policy above, which can be costly if you’re using a SaaS centralized logging solution.
Something to note about audit policies is that when an event is processed it is compared against the audit policy rules in order, and the first matching rule sets the audit level of the event. It’s pretty weird. It would make a lot more sense to read the entire policy and apply rules according to most restrictive; like is done with Kubernetes RBAC policies.
---
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
# The following requests were manually identified as high-volume and low-risk,
# so drop them.
- level: None
resources:
- group: ""
resources:
- endpoints
- services
- services/status
users:
- 'system:kube-proxy'
verbs:
- watch
- level: None
resources:
- group: ""
resources:
- nodes
- nodes/status
userGroups:
- 'system:nodes'
verbs:
- get
- level: None
namespaces:
- kube-system
resources:
- group: ""
resources:
- endpoints
users:
- 'system:kube-controller-manager'
- 'system:kube-scheduler'
- 'system:serviceaccount:kube-system:endpoint-controller'
verbs:
- get
- update
- level: None
resources:
- group: ""
resources:
- namespaces
- namespaces/status
- namespaces/finalize
users:
- 'system:apiserver'
verbs:
- get
# Don't log HPA fetching metrics.
- level: None
resources:
- group: metrics.k8s.io
users:
- 'system:kube-controller-manager'
verbs:
- get
- list
# Don't log these read-only URLs.
- level: None
nonResourceURLs:
- '/healthz*'
- /version
- '/swagger*'
# Don't log events requests.
- level: None
resources:
- group: ""
resources:
- events
# node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- nodes/status
- pods/status
users:
- kubelet
- 'system:node-problem-detector'
- 'system:serviceaccount:kube-system:node-problem-detector'
verbs:
- update
- patch
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- nodes/status
- pods/status
userGroups:
- 'system:nodes'
verbs:
- update
- patch
# deletecollection calls can be large, don't log responses for expected namespace deletions
- level: Request
omitStages:
- RequestReceived
users:
- 'system:serviceaccount:kube-system:namespace-controller'
verbs:
- deletecollection
# Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
# so only log at the Metadata level.
- level: Metadata
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- secrets
- configmaps
- group: authentication.k8s.io
resources:
- tokenreviews
# Get repsonses can be large; skip them.
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
- group: admissionregistration.k8s.io
- group: apiextensions.k8s.io
- group: apiregistration.k8s.io
- group: apps
- group: authentication.k8s.io
- group: authorization.k8s.io
- group: autoscaling
- group: batch
- group: certificates.k8s.io
- group: extensions
- group: metrics.k8s.io
- group: networking.k8s.io
- group: policy
- group: rbac.authorization.k8s.io
- group: scheduling.k8s.io
- group: settings.k8s.io
- group: storage.k8s.io
verbs:
- get
- list
- watch
# Default level for known APIs
- level: RequestResponse
omitStages:
- RequestReceived
resources:
- group: ""
- group: admissionregistration.k8s.io
- group: apiextensions.k8s.io
- group: apiregistration.k8s.io
- group: apps
- group: authentication.k8s.io
- group: authorization.k8s.io
- group: autoscaling
- group: batch
- group: certificates.k8s.io
- group: extensions
- group: metrics.k8s.io
- group: networking.k8s.io
- group: policy
- group: rbac.authorization.k8s.io
- group: scheduling.k8s.io
- group: settings.k8s.io
- group: storage.k8s.io
# Default level for all other requests.
- level: Metadata
omitStages:
- RequestReceivedOnce we’ve defined a policy like above, we need to apply it to the Kubernetes API Server.
For Kops, we can apply the changes with kops edit cluster <cluster>.
spec:
fileAssets:
— name: kubernetes-audit
path: /srv/kubernetes/audit.yaml
# which type of instances to appy the file
roles: [Master]
content: |
<audit policy here>
kubeAPIServer:
auditLogPath: /var/log/kube-apiserver-audit.log
auditLogMaxAge: 10 # num days
auditLogMaxBackups: 1 # the num of audit logs to retain
auditLogMaxSize: 100 # the max size in MB to retain
auditPolicyFile: /srv/kubernetes/audit.yamlNOTE: If you’re using a cluster management tool other than Kops, you’ll need to find some way to get the audit policy on the Kubernetes master node.
In the kube configuration above, we’re using a log backend, to read about log backends and webhook backends, check the official Kubernetes documentation.
copied from -> https://medium.com/@noqcks/kubernetes-audit-logging-introduction-464a34a53f6c
Views: 46
