The Scenario
A single --privileged Docker flag on your Jenkins agent hands an attacker full root on your build host — and in most default installs, that flag is sitting there, enabled by a well-meaning engineer who just needed Docker-in-Docker to work. I’ve walked into more than a few environments where the Jenkins controller was locked down tight, certificates everywhere, SSO enforced, and then the agent was running as root in a privileged container with AWS credentials mounted directly into the workspace. All that perimeter work, undone by one flag.
The threat model here is concrete. Your builds pull third-party dependencies. A compromised npm package or poisoned Maven artifact runs arbitrary code inside your agent. From there, it can read the workspace filesystem, exfiltrate credentials injected as environment variables, call back to the controller over the JNLP socket, or — if the container is privileged — mount the host filesystem and own the node entirely. A malicious PR that gets auto-tested has the same access. The attack surface is the agent itself, and most teams treat it as a throwaway environment rather than a security boundary.
This is the setup we hardened: Jenkins LTS controller on a dedicated VM, agents running as Docker containers on separate hosts, builds pulling secrets from the Credentials Binding plugin. The same principles apply to EC2-provisioned agents and Kubernetes pod agents — I’ll cover the Kubernetes variant in the pod spec below.
Prerequisites
- Jenkins LTS 2.440+ — older versions lack some agent security hardening flags; check your version at Manage Jenkins → About Jenkins
- Agent hosts running Linux — Ubuntu 22.04 or RHEL 9 used here; kernel 5.11+ required if you plan to use rootless Docker (verify with
uname -r) - Docker 24.x for containerized agents
- Plugins: Role-based Authorization Strategy 727.vd9f8e5a_f4f5a, Credentials Binding 657.v68a, Pipeline 596.v8c21c963d92d
- Controller and agents on separate hosts — single-host setups have a different threat model and some of these controls won’t apply cleanly
- For the Kubernetes variant: a cluster with PodSecurityAdmission available and the Jenkins Kubernetes Plugin 3900+
Step 1 — Isolate the Agent OS User and Filesystem
The goal here is simple: a build process should be able to touch its workspace and nothing else. Start by creating a dedicated system user with no login shell and no home directory outside the workspace root.
# Create a non-login system user for all agent processes
useradd -r -s /sbin/nologin -d /var/lib/jenkins-agent jenkins-agent
# Set workspace root — agent user owns it, nothing else can write here
mkdir -p /var/lib/jenkins-agent/workspace
chown jenkins-agent:jenkins-agent /var/lib/jenkins-agent/workspace
chmod 750 /var/lib/jenkins-agent/workspaceMount the workspace on a separate volume. A runaway build filling the root filesystem will take down the agent host entirely — I’ve seen this cause a 2 AM incident that turned out to be a test suite generating gigabytes of debug output. The small EBS cost is worth it. Then lock down /tmp so build processes can’t write and execute files there.
# Add to /etc/fstab — prevents execution of binaries written to /tmp
# Watch out: noexec on /tmp breaks Maven Surefire and some Node.js scripts
# that write temp executables. Test on a non-critical agent first.
tmpfs /tmp tmpfs defaults,noexec,nosuid,nodev 0 0Also set AllowAgentForwarding no in /etc/ssh/sshd_config on every agent host. It’s off by default on newer OpenSSH builds, but I’ve seen it flipped on during “troubleshooting” and never turned back off.
Step 2 — Lock Down the JNLP Connection and Agent-to-Controller Security
This is the highest-impact change you can make in under five minutes, and it’s disabled by default in most Jenkins installs. Go to Manage Jenkins → Security and enable Agent → Controller Security. Without this, an agent can make arbitrary calls back to the controller, including reading files and executing code. Enabling it restricts what agents are allowed to do on the controller side.
While you’re there, switch from raw TCP port 50000 to WebSocket transport. It reduces your firewall exposure significantly — you’re going through the same HTTPS port as the UI instead of exposing a separate TCP listener to the network. If you must keep TCP 50000 open, scope it to the controller IP only in your iptables rules.
# /etc/iptables/rules.v4 — restrict JNLP TCP port to controller IP only
# Replace 10.0.1.10 with your actual controller IP
-A INPUT -p tcp --dport 50000 -s 10.0.1.10 -j ACCEPT
-A INPUT -p tcp --dport 50000 -j DROPPin the agent JAR version in your provisioning script. A version mismatch between the agent JAR and the controller produces java.io.IOException: Unexpected termination of the channel — people chase this as a network issue for hours. It’s almost never the network.
# Always pull the agent JAR from the controller itself — version is guaranteed to match
curl -fsSL http://jenkins-controller:8080/jnlpJars/agent.jar -o /opt/jenkins-agent/agent.jarStore the agent secret token in Vault or AWS Secrets Manager. The tokens are 64-character hex strings sitting in $JENKINS_HOME/nodes/<name>/config.xml. Rotate them on a schedule by deleting and re-provisioning the agent node via the API.
Step 3 — Harden the Build Container
Never run agent containers with --privileged. I stopped using Docker-in-Docker entirely after seeing a build escape to the host through a CVE in the Docker daemon. We use Kaniko for image builds now — it runs unprivileged and produces identical results for our use case. If your team genuinely needs DinD, rootless Docker is the right path, not --privileged.
For Kubernetes-based agents, this pod spec applies the controls at the manifest level — seccomp profile, dropped capabilities, read-only root filesystem, and ephemeral workspace volume. Pin the image to a digest, not a tag, so a registry push can’t silently swap what you’re running.
# jenkins-agent-pod.yaml
# Kubernetes Pod template for a hardened Jenkins inbound agent
# Compatible with Jenkins Kubernetes Plugin 3900.va_dce992317b_4+
# Apply namespace-level PodSecurityAdmission: enforce=restricted
apiVersion: v1
kind: Pod
metadata:
name: jenkins-agent
labels:
app: jenkins-agent
spec:
# Run as a non-root user — UID 1000 matches eclipse-temurin default
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault # applies Docker default seccomp — blocks ~44 syscalls
# Do not automount the service account token — agents don't need k8s API access
automountServiceAccountToken: false
containers:
- name: jnlp
# Pin to a specific digest, not just a tag — prevents silent image swaps
image: jenkins/inbound-agent:3256.v88a_f6e922152-1@sha256:REPLACE_WITH_ACTUAL_DIGEST
imagePullPolicy: Always
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL # drop every Linux capability — inbound agent needs none
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "4Gi" # hard limit — OOM kill shows as exit 137 in build log
cpu: "2"
env:
- name: JENKINS_URL
value: "https://jenkins.internal.kuryzhev.cloud"
- name: JENKINS_AGENT_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
# Secret token injected from k8s Secret, never hardcoded here
- name: JENKINS_SECRET
valueFrom:
secretKeyRef:
name: jenkins-agent-secret
key: token
volumeMounts:
# Workspace is the only writable path
- name: workspace
mountPath: /home/jenkins/agent
# /tmp must be writable for JVM — use emptyDir, not host path
- name: tmp-dir
mountPath: /tmp
volumes:
- name: workspace
emptyDir: {} # ephemeral — wiped after pod terminates, no cross-build leakage
- name: tmp-dir
emptyDir:
medium: Memory # tmpfs — faster and never persists to node disk
sizeLimit: 512Mi
nodeSelector:
role: ci-agent # dedicate a node group to build workloads only
restartPolicy: Never # agent pods should not restart — Jenkins provisions a new oneWatch out for memory limits. Setting --memory 4g with --memory-swap 4g (no swap) causes silent OOM kills that appear as exit code 137. Without explicit handling in your Jenkinsfile, the build just disappears. Add an error handler that logs exit codes so you can tell OOM kills from actual test failures.
Step 4 — Credential Hygiene and Secret Injection
Secrets should never touch disk or appear in logs. Use the Credentials Binding plugin and scope secrets to the withCredentials block only — they’re unset when the block exits. Enable maskPasswords globally in Jenkins so a set -x in a shell step doesn’t print them to the console log. The CPU overhead is around 3-5% on large log output — an acceptable tradeoff.
// Jenkinsfile — credentials scoped tightly, never passed as parameters
pipeline {
agent { label 'hardened-agent' }
stages {
stage('Deploy') {
steps {
// Secret is available only within this block — unset on exit
withCredentials([string(credentialsId: 'aws-deploy-token', variable: 'AWS_TOKEN')]) {
sh '''
# AWS_TOKEN is masked in logs even if set -x is active
aws sts get-caller-identity
'''
}
}
}
}
}Never pass secrets as build parameters. Parameters are stored in plaintext at $JENKINS_HOME/jobs/<job>/builds/<n>/build.xml on the controller filesystem. Anyone with filesystem access — or a backup restore — can read them. Run find $JENKINS_HOME/credentials.xml and audit which agents have access to which credential domains while you’re at it.
Verify and Test
Hardening that isn’t verified is just documentation. Run these checks before calling any agent production-ready.
For Docker agents, inspect the container and confirm your security settings actually applied — it’s easy to have a compose file override something you set in the provisioning script.
# Verify container security config — check CapAdd, CapDrop, ReadonlyRootfs
docker inspect <agent-container-id> | jq '.[0].HostConfig | {CapAdd, CapDrop, ReadonlyRootfs, Memory}'Run a test pipeline step that tries to write outside the workspace. It should fail immediately with Permission denied. If it succeeds, your filesystem isolation isn’t working.
# Test pipeline step — should fail with Permission denied
sh 'touch /etc/pwned-test || echo "Write blocked — isolation confirmed"'Confirm Agent → Controller Security is enforced by triggering a Groovy step that calls Jenkins.instance. You should see org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException — that’s the correct behavior. If it runs, go back to Step 2. After any build run, check /var/log/auth.log on the agent host — no sudo calls should appear. Finally, scan your base image before promoting it.
# Scan agent base image for CVEs before promoting to production
trivy image eclipse-temurin:21-jre-alpine --severity HIGH,CRITICALThe eclipse-temurin:21-jre-alpine image comes in around 180MB versus jenkins/inbound-agent:latest at roughly 580MB. Smaller image, smaller attack surface, faster pull times. Worth the occasional Alpine compatibility issue.
Closing
Agent hardening is a layered problem — no single control is enough on its own. OS user isolation limits filesystem blast radius. JNLP lockdown prevents agents from becoming a pivot point into the controller. Container security context controls cap escalation at the process level. Credential hygiene ensures that even if a build is compromised, the secrets it touches don’t persist anywhere recoverable. Together, these controls turn your build nodes from an open door into a genuine security boundary. The natural next step is moving to fully ephemeral agents — Kubernetes pod agents with PodSecurityAdmission set to enforce=restricted at the namespace level, provisioned fresh for every build and gone the moment it completes. We cover related CI/CD security patterns over at kuryzhev.cloud if you want to keep going in that direction.
