Persistent Volumes permission issue for 2+ node clutsters

[albert-salman]

Hello,
If minikube cluster is 2+ nodes then initContainers cannot set volume permissions inside PVCs and pods fail to access data there. Single node cluster do not have this issue.

[RA489]

/kind support

[spowelljr]

Hi @albert-salman, thanks for reporting your issue with minikube!

To help me debug the issue you're experiencing, could you please provide me with a list of instructions to follow to allow me to reproduce this issue?

Thanks!

[albert-salman]

To help me debug the issue you're experiencing, could you please provide me with a list of instructions to follow to allow me to reproduce this issue?

Hi @spowelljr

1. Create 2-node cluster
2. Deploy some manifest or helm chart (e.g. bitnami's elasticstack) which use initContainers to setup permission for containers' PVCs
3. Path which had to have correct permissions (set by initContainers section) is denied for any running app in container

i have a suspicion that PVC is not correctly mapped at node which starts the initContainers's temporary container

i am using
minikube version: v1.22.0
commit: a03fbcf

in bitnami/elasticstack file master-statefulset.yaml

 initContainers:
    - name: volume-permissions
      image: {{ include "elasticsearch.volumePermissions.image" . }}
      imagePullPolicy: {{ .Values.volumePermissions.image.pullPolicy | quote }}
      command:
        - /bin/bash
        - -ec
        - |
          chown -R {{ .Values.master.securityContext.runAsUser }}:{{ .Values.master.securityContext.fsGroup }} /bitnami/elasticsearch/data
      securityContext:
        runAsUser: 0
      {{- if .Values.volumePermissions.resources }}
      resources: {{- toYaml .Values.volumePermissions.resources | nindent 12 }}
      {{- end }}
      volumeMounts:
        - name: data
          mountPath: "/bitnami/elasticsearch/data"

[victor-sudakov]

I think my problem is related. On a 2-node cluster, the spec.template.spec.securityContext.fsGroup=65534 setting is not respected and volume membership is not changed. It remains 755 root:root instead of 2775 root:nobody.

How to reproduce

1. Apply the following manifest

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: ubuntu
  name: ubuntu
spec:
  replicas: 1
  serviceName: ubuntu-headless
  selector:
    matchLabels:
      app: ubuntu
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      containers:
        - image: ubuntu:jammy-20211122
          name: ubuntu
          command: ["/bin/sleep", "365d"]
          volumeMounts:
            - mountPath: /d01
              name: bigdisk
  volumeClaimTemplates:
    - metadata:
        name: bigdisk
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 5G

2. Shell into the ubuntu-0 Pod and check the permissions on the /d01 mountpoint. They will not belong to root:nobody as they should according to the fsGroup setting, but will remain root:root.

Minikube version
minikube version: v1.24.0 commit: 76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b-dirty

Minikube was started as
minikube start --disk-size=50g --nodes=2 --insecure-registry "192.168.88.0/24" --mount --mount-string ~/.volumes:/volumes

[victor-sudakov]

@albert-salman Do you see the problem if you use the fsGroup security context instead of initContainers? I do.

BTW I think using initContainers just for setting permissions is a bit of overkill.

[victor-sudakov]

On a single-node minikube cluster, the setting of spec.template.spec.securityContext.fsGroup=65534 results in volume permissions 777 root:root which is not correct either. fsGroup should not work this way IMHO.

nobody@ubuntu-0:/$ ls -ald /d01
drwxrwxrwx 2 root root 6 Dec  6 02:25 /d01
nobody@ubuntu-0:/$ 

Expected: root:nobody

@albert-salman can you confirm this?

[victor-sudakov]

@albert-salman ping!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale