Overview
Kubernetes Pod Security Standards: Safe Rollout Without Breaking Workloads is the kind of operational challenge that catches teams off guard — not because the feature is obscure, but because the path from “nothing enforced” to “restricted everywhere” is littered with subtle breakage if you move too fast. PSS replaced the deprecated PodSecurityPolicy admission plugin starting in Kubernetes 1.25 and became the stable, built-in mechanism for controlling what Pods can and cannot do at the OS boundary.
The model is deliberately simple: three policy levels, three enforcement modes, and a handful of namespace labels. That simplicity is deceptive. The moment you flip a namespace to enforce: restricted without preparation, any Pod spec that mounts a hostPath volume, runs as root, or retains default Linux capabilities will be rejected at admission time. In a production cluster, that means deployment failures and on-call pages.
The three policy levels are:
- privileged — no restrictions; intended for system-level workloads and infrastructure components.
- baseline — blocks the most dangerous escalation vectors (host namespaces, privileged containers, dangerous volume types) while remaining broadly compatible with containerized applications.
- restricted — enforces current hardening best practices: non-root execution, dropped capabilities, seccomp profiles, and no privilege escalation.
Each level can be applied in three independent modes per namespace. enforce blocks non-compliant Pods outright. audit records violations to the Kubernetes audit log without blocking anything. warn returns a warning header in the API response so that kubectl callers see the problem immediately. The practical rollout strategy exploits all three modes in sequence, which is exactly what this tutorial covers.
For background on how namespace-scoped controls fit into a broader cluster security posture, see the DevOps_DayS knowledge base for related Kubernetes hardening articles. The authoritative reference for the full field list and version semantics lives in the official Pod Security Standards documentation.
Before you start
Before touching any namespace labels, verify the following prerequisites.
Cluster version. PSS admission is stable from Kubernetes 1.25 onward. Anything earlier requires the alpha feature gate or a third-party admission webhook. Check your server version:
kubectl version --shortAudit logging enabled. The audit mode is only useful if your cluster actually writes audit events. On kubeadm-provisioned clusters, the file to watch is /var/log/kubernetes/audit/audit.log. Confirm the audit policy includes RequestResponse or at minimum Metadata level for the pods resource.
Existing namespace labels. Many clusters already carry labels from previous experiments or admission webhook configurations. Inspect what is currently set before adding anything new:
kubectl get ns payments --show-labelsIf you see an existing pod-security.kubernetes.io/enforce label set to a permissive level, note it — you will be tightening it, not starting from scratch.
System namespace awareness. Never apply baseline or restricted enforcement to kube-system, kube-public, or any CNI/CSI namespace. Core components routinely run as root, mount host paths, and use host networking. Restricting these namespaces will break your cluster. Leave them explicitly at privileged or unlabeled.
Inventory your workloads. Produce a quick summary of which namespaces contain application workloads versus infrastructure tooling. The enforcement rollout should target application namespaces first, where you have full control over Pod specs.
Step 1: Audit Existing Workloads with warn and audit Modes
The safest first move is to apply warn and audit labels simultaneously. Neither mode blocks traffic. Together they give you two independent signal sources: real-time warnings in your terminal and a persistent record in the audit log.
The following manifest applies both modes to the payments namespace targeting the restricted policy level. The enforce labels are present but commented out — they will be activated in Step 3 after remediation is complete. Note the version-pin labels: pinning to a specific Kubernetes version (here v1.29) prevents the policy semantics from silently changing when you upgrade the control plane.
# pss-namespace-rollout.yaml
# Phase 1: apply warn + audit to detect violations safely
# Phase 2: promote to enforce after workload remediation
---
apiVersion: v1
kind: Namespace
metadata:
name: payments
labels:
# Audit mode: violations recorded in audit log, nothing blocked
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: v1.29
# Warn mode: API server returns warning headers to kubectl callers
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: v1.29
# Enforce: uncomment only after all workloads pass audit/warn clean
# pod-security.kubernetes.io/enforce: restricted
# pod-security.kubernetes.io/enforce-version: v1.29Apply it and immediately attempt a rollout of any existing Deployment in that namespace:
kubectl apply -f pss-namespace-rollout.yaml
kubectl rollout restart deployment/payments-api -n paymentsIf violations exist, kubectl will print warning lines before returning. They look like this:
Warning: would violate PodSecurity "restricted:v1.29": \
allowPrivilegeEscalation != false (container "api" must set \
securityContext.allowPrivilegeEscalation=false), \
unrestricted capabilities (container "api" must set \
securityContext.capabilities.drop=["ALL"])Each warning names the exact field that needs to change. Collect these across all Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs in the namespace. For audit log entries, filter by the PodSecurity source and FailedCreate reason:
grep 'PodSecurity' /var/log/kubernetes/audit/audit.log | \
jq -r '.annotations["pod-security.kubernetes.io/audit-violations"] // empty'Build a remediation list from both sources before proceeding.
Step 2: Remediate Violating Workloads
With a concrete list of violations in hand, update each Pod template to satisfy the restricted policy level. The changes cluster into three categories: security context fields, capability management, and volume substitution.
The following Deployment spec demonstrates a fully compliant configuration. Walk through each field — this is not boilerplate to copy blindly, but a reference to understand what the policy actually requires.
---
# Example compliant Deployment for the restricted policy level
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
namespace: payments
spec:
replicas: 2
selector:
matchLabels:
app: payments-api
template:
metadata:
labels:
app: payments-api
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: api
image: payments-api:1.4.2
ports:
- containerPort: 8080
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
# tmpfs for writable scratch space without hostPath
- name: tmp
emptyDir: {}
automountServiceAccountToken: falseSeveral decisions here deserve explicit explanation:
- seccompProfile: RuntimeDefault — Required by
restrictedsince Kubernetes 1.25. TheRuntimeDefaultprofile applies the container runtime’s default syscall filter, which is conservative without being application-breaking for most workloads. - capabilities.drop: [“ALL”] — Drops every Linux capability the kernel would otherwise grant. If your application genuinely needs
NET_BIND_SERVICE(binding to ports below 1024), add it back explicitly undercapabilities.add. Most applications running on port 8080 or higher need nothing. - readOnlyRootFilesystem: true — Forces the container filesystem to be immutable. Applications that write to
/tmpor need a cache directory must useemptyDirvolumes mounted at specific paths, as shown above. - runAsUser: 1000 — A non-zero UID satisfies
runAsNonRoot: true. The UID must exist in the container image or be set here explicitly. If your image’sUSERdirective already sets a non-root user, you can omitrunAsUserfrom the Pod spec and rely on the image default. - automountServiceAccountToken: false — Not strictly required by PSS, but a sensible hardening measure for workloads that do not call the Kubernetes API. It eliminates an unnecessary credential from the Pod filesystem.
After updating each workload manifest, apply it and watch for warnings. When a rollout completes with no warning output and the audit log produces no new PodSecurity entries for that namespace, that workload is clean.
Step 3: Enforce the Policy Level on Namespaces
Once warn and audit modes produce no violations across all workloads in a namespace, you are ready to promote to enforcement. Do not rush this step across all namespaces simultaneously. Roll out namespace by namespace, and validate before moving to the next.
First, perform a server-side dry run. This asks the API server to evaluate the label change as if it were real, without committing it:
kubectl label ns payments \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/enforce-version=v1.29 \
--dry-run=serverIf the dry run returns cleanly with no warnings, you can apply the change. The cleanest approach is to update the Namespace manifest you already have in version control — uncomment the enforce lines and apply:
kubectl apply -f pss-namespace-rollout.yamlVerify the labels are set correctly:
kubectl get ns payments --show-labelsImmediately after enforcement is active, attempt to create a deliberately non-compliant Pod to confirm the admission controller is actually blocking:
kubectl run test-violation \
--image=nginx \
--namespace=payments \
--restart=NeverThe default nginx image runs as root and does not set allowPrivilegeEscalation: false. Under restricted enforcement, this command should be rejected with a clear error message referencing PodSecurity. If it succeeds, the label was not applied correctly — re-check with --show-labels.
Repeat this process for each application namespace. Track progress in your GitOps repository so the namespace labels are source-controlled and drift is detectable.
Troubleshooting
Pods stuck in Pending after enforcement is enabled. This is the most common symptom of a premature enforce promotion. The Pod was admitted but the ReplicaSet controller is failing to create it. Check the ReplicaSet events:
kubectl describe rs -n payments <replicaset-name>Look for events with reason: FailedCreate and a message from PodSecurity. The message will list every violated field. Address those fields in the Pod template and roll out again.
Audit log entries are missing. If you applied the audit label but see nothing in the audit log, the audit policy on the control plane likely does not capture pods at the required verbosity. Check the --audit-policy-file argument on the kube-apiserver process and ensure it includes a rule covering pods at Metadata level or higher. On managed clusters (EKS, GKE, AKS), audit log delivery is configured through the cloud provider’s control plane logging settings, not a local file.
Accidentally enforced a namespace containing a critical workload. If you need to immediately roll back enforcement without removing the label entirely, downgrade the enforce level to privileged:
kubectl label ns payments \
pod-security.kubernetes.io/enforce=privileged \
--overwriteThis unblocks all Pods instantly. Then investigate the violation, fix the workload spec, and re-promote to restricted via the dry-run path described in Step 3.
Seccomp profile causing runtime failures. If a workload passes PSS admission but crashes at runtime with permission denied errors on syscalls, the RuntimeDefault seccomp profile may be filtering a syscall the application requires. Switch to Unconfined temporarily to confirm the diagnosis, then work with your security team to craft a custom seccomp profile that allows only the required syscalls.
Third-party operators and controllers. Helm charts and operators often deploy Pods with elevated privileges. Before enforcing on a namespace that contains operator-managed resources, check the operator’s documentation for PSS compatibility. Many popular operators have added restricted-compliant configurations in recent versions. If the operator cannot be updated, consider isolating it in a dedicated namespace at baseline enforcement rather than leaving the entire application namespace permissive.
Going further
With enforcement active on application namespaces, the next concern is preventing regressions. A developer pushing a new Deployment with a hostPath volume should discover the violation in CI, not in production.
The kubectl --dry-run=server flag works against any cluster with PSS labels active. Integrate it into your CI pipeline by running it against a dedicated staging namespace that mirrors production enforcement labels. Any manifest that would violate the policy will fail the pipeline step before merge.
For policy requirements that go beyond the three built-in PSS levels — custom allowed registries, mandatory resource limits, required label schemas — OPA Gatekeeper provides a Kubernetes-native admission webhook backed by Rego policies. PSS and Gatekeeper are complementary: PSS handles the security baseline efficiently at the admission controller level, while Gatekeeper handles organization-specific policy that PSS cannot express.
If you manage namespace labels through GitOps with Argo CD or Flux, the namespace manifest approach demonstrated in this tutorial integrates naturally. Store each namespace’s YAML in your cluster configuration repository, and the label rollout becomes a pull request with a review gate rather than an imperative kubectl label command. This also gives you a clear audit trail of when enforcement was promoted and who approved it.
Finally, revisit the version-pin labels when you upgrade Kubernetes. A label pinned to v1.29 will continue to enforce the 1.29 policy semantics even on a 1.31 cluster. That stability is intentional, but it means you need a deliberate process to advance the version pins as part of your cluster upgrade runbook — otherwise policy drift accumulates silently over time.
