diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/README.md b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/README.md new file mode 100644 index 0000000..ff92e1e --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/README.md @@ -0,0 +1,568 @@ +# Findings: native Kubernetes user namespaces with rootful DinD + +Experiment run: 2026-08-06 + +Record last updated: 2026-08-08 + +## Question tested + +Can a Coder workspace run a normal **rootful** Docker daemon and BuildKit in a +native Kubernetes user-namespace Pod, without Envbox or Sysbox, while keeping +Pod UID 0 mapped to an unprivileged host UID range? + +This was not a rootless-Docker test. The ultimately successful shape was: + +```text +EKS AL2023 node + └─ containerd RuntimeClass: stock runc + cgroup_writable = true + └─ Pod: hostUsers: false, privileged: false + ├─ workspace processes in /workspace-processes cgroup + └─ rootful dockerd managing sibling /docker cgroup hierarchy + └─ ordinary Docker containers / BuildKit workers +``` + +The original fully successful Pod used `capabilities.add: ["ALL"]`, +`procMount: Unmasked`, an unconfined seccomp profile, and +`allowPrivilegeEscalation: true`. The capability-minimization follow-up below +shows that the runtime's default capability set is insufficient, while adding +only `SYS_ADMIN` and `NET_ADMIN` produces a full workload pass. `ALL` is +therefore not required. Those powers were inside the Pod's user namespace: +container UID 0 mapped to a nonzero host UID range. +The resulting workspace was effectively privileged over resources owned by +that user namespace, but it was neither a Kubernetes `privileged: true` +container nor privileged in the host's initial user namespace. Consequently, +`privileged: false` here must not be read as the security posture of a +conventionally restricted application Pod. + +## Environment + +- EKS Kubernetes `v1.36.2-eks-254016e` in `us-east-2`. +- Amazon Linux 2023 `m6i.large` managed node-group nodes. +- Node kernel: `6.18.38-76.139.amzn2023.x86_64`. +- Node containerd: `2.2.5+unknown`. +- EBS CSI driver, with an experiment-specific `gp3-csi` StorageClass using + `ebs.csi.aws.com` and `WaitForFirstConsumer`. +- Docker test image: `docker:27-dind`, which resolved to Docker Engine + `27.5.1`; the replay manifest now pins `docker:27.5.1-dind`. +- Final Docker data root: an EBS/ext4 PVC. + +The MNG was selected as the debugging-friendly baseline before considering +EKS Auto Mode/Bottlerocket. It permits explicit node bootstrap and containerd +configuration. + +## Baseline results on the stock runtime + +### Native user namespace with an EBS PVC: pass + +`userns-volume-probe.yaml` ran with `hostUsers: false`, bound the EBS PVC, and +wrote and read `/workspace/probe.txt` successfully: + +```text +uid=0(root) +/proc/self/uid_map: + 0 3130523648 65536 +``` + +This proved both non-host UID mapping and compatibility with the CSI-mounted +EBS/ext4 workspace volume. + +### Namespaced privileged probe: pass + +The initial capability probe used `privileged: true` and +`procMount: Unmasked`. It retained a non-host UID mapping and successfully +created a private tmpfs mount. This established that the requested kernel +operations were available inside the user namespace, but the final DinD Pod +did not need Kubernetes `privileged: true`. + +### Rootful Docker on the stock runtime: fail + +Dockerd started and initialized `overlay2` on the PVC, and image pulling +worked. Every attempt to start a child container failed with: + +```text +unable to apply cgroup configuration: +mkdir /sys/fs/cgroup/docker: permission denied +``` + +A focused probe confirmed that the stock runtime exposed no cgroup directory +writable by the Pod. Therefore `hostUsers: false` alone was insufficient for +rootful DinD. + +## Cgroup-writable RuntimeClass follow-up + +The follow-up added a second AL2023 MNG. Its containerd configuration +registered a named handler using the stock `io.containerd.runc.v2` runtime: + +```toml +[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc-cgroup-writable] + runtime_type = 'io.containerd.runc.v2' + cgroup_writable = true +``` + +`runc-cgroup-writable` is a local RuntimeClass/handler name, not a custom runc +binary. + +### Writable-cgroup probe: pass + +A non-privileged, user-namespaced Pod scheduled through this RuntimeClass and +created a child cgroup successfully: + +```text +/proc/self/uid_map: + 0 2088894464 65536 +cgroup on /sys/fs/cgroup type cgroup2 (rw,...,nsdelegate,...) +cgroup-writable-probe-ok +``` + +This fixed the original permission-denied failure. + +### First rootful-Docker attempt: partial pass + +Without preparing the cgroup topology, the following worked: + +- dockerd startup; +- `overlay2` on the EBS/ext4 PVC; +- image pulls and ordinary `docker run`; +- BuildKit `RUN` steps; +- Docker bridge networking between nested containers. + +Only a container using `--memory=64m --pids-limit=64` failed. The hierarchy +showed: + +```text +/sys/fs/cgroup: domain threaded +/sys/fs/cgroup/docker: threaded +``` + +PID 1, dockerd, and containerd occupied the delegated root while threaded +controllers were enabled. This forced a threaded topology in which Docker +could not apply the domain `memory` controller. + +### Domain-cgroup topology: full workload pass + +Before starting dockerd, the final entrypoint: + +1. created `/sys/fs/cgroup/workspace-processes`; +2. moved PID 1 and the workspace processes into it; +3. left the delegated cgroup root empty; +4. enabled `cpuset cpu io memory pids` in the root's + `cgroup.subtree_control`; +5. started dockerd in `workspace-processes`, with Docker children under the + sibling `/docker` hierarchy. + +The recorded result was: + +```text +/proc/self/uid_map: + 0 1990918144 65536 +root cgroup type: domain +root cgroup processes: +root subtree controllers: cpuset cpu io memory pids +workspace cgroup type: domain +Docker storage driver: overlay2 +BuildKit result: buildkit-ok +resource-limited container launch: pass +``` + +Image pull, ordinary nested execution, BuildKit, bridge networking, and a +nested container configured with memory and PID limits all completed. + +### Capability-minimization follow-up: two added capabilities pass + +On 2026-08-08, the cluster and cgroup-writable MNG were recreated and the +writable-cgroup control probe passed again. The final DinD manifest was then +rerun unchanged except that `capabilities.add: ["ALL"]` was removed. This +left the container with the runtime's default capability set: + +```text +CapPrm: 00000000a80425fb +CapEff: 00000000a80425fb +CapBnd: 00000000a80425fb +``` + +Dockerd did not become ready. Its log identified two capability-sensitive +failures: + +```text +failed to mount overlay: operation not permitted +failed to create NAT chain DOCKER: iptables ... Permission denied +``` + +The earlier root-propagation setup also reported `operation not permitted`. +These results show that the default capability set is insufficient for this +rootful-DinD configuration. They point to `CAP_SYS_ADMIN` for mount and +overlay operations and `CAP_NET_ADMIN` for iptables/NAT setup. They do **not** +show that every capability in `ALL` is necessary. + +A second variant retained the runtime's default capability set and added only +`SYS_ADMIN` and `NET_ADMIN`: + +```yaml +capabilities: + add: ["SYS_ADMIN", "NET_ADMIN"] +``` + +Its effective capability mask was: + +```text +CapPrm: 00000000a82435fb +CapEff: 00000000a82435fb +CapBnd: 00000000a82435fb +``` + +The complete embedded suite passed: dockerd became ready with `overlay2`, an +ordinary child container ran, BuildKit completed a build, bridge networking +and HTTP worked, memory and PID limits were accepted, Compose service DNS and +HTTP worked, outbound networking worked, and the published port was reachable +through the workspace loopback address. + +The companion peer test was then scheduled on the original MNG node, distinct +from the workspace's cgroup-writable MNG node. It reached the Compose-published +server through both the Kubernetes ClusterIP Service and the workspace Pod IP: + +```text +cross-node-clusterip-service-ok +cross-node-direct-pod-ip-ok +compose-cross-node-network-tests-ok +``` + +This disproves `ALL` as a requirement. `SYS_ADMIN` plus `NET_ADMIN` is the +smallest **tested** successful addition set. The experiment has not yet rerun +the suite with either capability individually removed, so it does not by +itself formally prove that both are independently necessary. The observed +mount/overlay and iptables failures strongly explain why each is present. + +The workspace container also explicitly sets +`allowPrivilegeEscalation: true`. This is not currently an independent setting +that can simply be tightened while retaining the tested capability set: +Kubernetes treats privilege escalation as enabled whenever a container has +`CAP_SYS_ADMIN`. In this design that authority remains scoped by +`hostUsers: false` to the Pod's user namespace, but it still removes the +`no_new_privs` defense-in-depth boundary inside the workspace. + +### `/proc`-masking follow-up: Kubernetes default fails + +A third variant retained the successful `SYS_ADMIN` and `NET_ADMIN` additions, +unconfined seccomp, and writable cgroups, but removed +`procMount: Unmasked`. Kubernetes consequently restored its default protected +`/proc` submounts, including a read-only `/proc/sys` and masked sensitive files +such as `/proc/kcore` and `/proc/keys`. + +Dockerd itself started, initialized `overlay2`, and answered `docker info`. +The first fresh nested `docker run`, however, failed while Docker configured +the container's veth interface: + +```text +failed to add interface ... to sandbox: +failed to configure ipv6: failed to disable IPv6 on container's interface +eth0: unknown +``` + +For the tested Docker 27.5.1 bridge-network configuration, default `/proc` +masking is therefore insufficient: Docker needs a writable namespaced +`/proc/sys` path during nested-container network setup. This result does not +prove that exposing every path covered by `procMount: Unmasked` is inherently +necessary. A differently configured Docker network or a future mechanism for +narrower `/proc` exposure may reduce this requirement. + +### Seccomp follow-up: `RuntimeDefault` fails + +A fourth variant restored `procMount: Unmasked` and retained the successful +`SYS_ADMIN` and `NET_ADMIN` additions and writable cgroups, but replaced the +outer workspace container's `Unconfined` seccomp profile with +`RuntimeDefault`. `/proc/self/status` confirmed one active seccomp filter: + +```text +Seccomp: 2 +Seccomp_filters: 1 +``` + +Dockerd started, initialized `overlay2`, and answered `docker info`. The first +fresh nested `docker run` failed during nested runc initialization: + +```text +unable to join session keyring: +unable to create session key: operation not permitted +``` + +The tested runtime-default profile is therefore insufficient for this nested +runc path. This does not prove that the outer workspace must be fully +unconfined. A tailored seccomp profile that permits the required keyring +syscalls, or a validated configuration that tells nested runc not to create a +new session keyring, may retain most default filtering while allowing DinD. + +This topology is not unique to the native MNG experiment. Current Envbox and +Sysbox solve the same cgroup-v2 no-internal-process constraint at two levels: + +- Envbox's outer-dockerd wrapper (`cli/wrap_dockerd.sh`) creates an `/init` + leaf, moves processes out of the visible cgroup root, and enables its + controllers before starting the outer dockerd. This keeps inner-container + cgroups beneath the Envbox Pod's host cgroup tree. +- Sysbox-runc creates an `init.scope` leaf for the system container, places + its init and exec processes there, and delegates ownership of the cgroup-v2 + control files so inner systemd or Docker can create domain sub-cgroups. + +The native wrapper's `workspace-processes` leaf and sibling `/docker` +hierarchy explicitly reproduce the latter delegation pattern using stock runc +and containerd's `cgroup_writable = true` handler. The wrapper is therefore an +explicit replacement for behavior that Sysbox normally supplies invisibly, +not an unrelated workaround. + +### Docker Compose networking follow-up: pass + +A targeted Docker Compose test created a user-defined bridge network with an +`nginx:1.27-alpine` server and an `alpine:3.21` client. It demonstrated: + +- Compose network creation and attachment of both service containers; +- Docker embedded DNS and bare service-name resolution through libc; +- HTTP from the client to `http://server`; +- outbound HTTP from the nested client to the internet; +- a nested server published as `0.0.0.0:18080->80/tcp`; +- access to that published port from the workspace through + `127.0.0.1:18080`; +- access from another Kubernetes Pod through the workspace Pod IP; and +- access through a Kubernetes ClusterIP Service targeting the published port. + +The external peer check first passed on the workspace node and then passed +from the original MNG node. The cross-node probe reached both the workspace +Pod IP (`192.168.83.149:18080`) and the ClusterIP Service, demonstrating that +Docker's nested bridge/NAT and port-publishing rules interoperated with EKS +Pod routing and Service forwarding across nodes. + +One diagnostic nuance was observed. BusyBox `nslookup server` tried the +Kubernetes search domains inherited by the nested container with `ndots:5` +and returned failure, while `nslookup server.`, `getent hosts server`, and +HTTP to the bare name `server` all resolved the Compose service correctly. +This did not prevent ordinary libc-based application resolution, but clients +with unusual raw-DNS/search-list behavior may require separate validation. + +The successful +[`rootful-dind.yaml`](cgroup-writable-runtime/rootful-dind.yaml) replay +manifest now automates the Compose service-name, HTTP, outbound-network, and +workspace-loopback checks and records explicit completion artifacts. The +companion +[`compose-network-peer.yaml`](cgroup-writable-runtime/compose-network-peer.yaml) +declaratively creates the ClusterIP and headless Services and pins a restricted +peer Pod to the original MNG. The peer resolves the headless Service to the +workspace Pod IP, accesses that IP directly, and separately accesses the +ClusterIP Service. The Docker patch release and observed Alpine and Nginx +digests are pinned for repeatability. + +These manifests encode checks that passed interactively during the recorded +experiment. Their newly combined automated orchestration has not yet itself +been rerun; a future replay must still verify the completion files, peer log, +and distinct workspace/peer node placement before treating the manifests as a +fresh pass. + +## Interpretation + +Native Kubernetes user namespaces can support rootful DinD on this EKS 1.36 +AL2023 MNG without a host-privileged workspace Pod, provided that all of the +following are supplied: + +1. `hostUsers: false` and the broad in-user-namespace security context needed + by dockerd; +2. a containerd RuntimeClass with `cgroup_writable = true`; +3. a startup wrapper that constructs a valid delegated domain-cgroup + topology before starting dockerd; +4. compatible writable storage; EBS/ext4 with `overlay2` worked here. + +The stock EKS runtime remains insufficient. The positive result depends on +purpose-built node/runtime configuration and is currently demonstrated only +on a configurable managed node group. + +### AMI compatibility boundary + +The positive result was demonstrated on the AWS EKS-optimized Amazon Linux +2023 AMI. That AMI runs `nodeadm` during boot, and `nodeadm` supports merging +additional inline containerd TOML from a `NodeConfig`. The experiment used +that supported bootstrap path to register the handler; it did not modify or +rebuild the AMI itself. + +This result does not establish compatibility with every custom, certified, or +hardened AMI. A candidate AMI must preserve the `nodeadm`/`NodeConfig` +bootstrap path, permit the containerd override, provide a containerd version +that supports `cgroup_writable`, and use the matching containerd configuration +schema. In particular, containerd 1.x and 2.x use different CRI plugin paths. +AMI hardening or compliance policy may also prohibit writable delegated +cgroups even when the image can technically accept the configuration. + +Therefore the current compatibility boundary is: + +- AWS EKS-optimized AL2023 MNG: compatible and proven by this experiment; +- custom AL2023 AMI derived from it: plausible only if the required bootstrap + and runtime behavior are preserved, and must be tested; +- arbitrary certified or hardened AMI: not guaranteed and requires vendor or + compliance validation; +- EKS Auto Mode Bottlerocket: this AL2023 bootstrap mechanism is unavailable. + +### Storage compatibility boundary + +The MNG stack supports idmapped mounts; the `hostUsers: false` volume probe +successfully mounted and wrote to an EBS/ext4 PVC. Ext4 therefore provides a +proven storage path for the normal Coder shape of one workspace Pod using one +RWO persistent volume. + +NFS volumes are not supported for Kubernetes user-namespace Pods. Kubernetes +1.36 explicitly documents that the Linux NFS client does not support idmapped +mounts, which these Pods require for every filesystem used by a Pod volume. +This also excludes standard EFS CSI volumes because EFS is mounted through +NFS. See the upstream +[user-namespace filesystem requirements](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#filesystem-support). + +This is a meaningful compatibility restriction, but not a general blocker for +an EBS-backed Envbox replacement. It becomes blocking for templates that +require NFS/EFS semantics such as RWX storage, concurrently shared home +directories or datasets, or storage without EBS availability-zone affinity. + +### Namespace and nested-networking boundaries + +Kubernetes disallows combining `hostUsers: false` with `hostNetwork: true`, +`hostPID: true`, or `hostIPC: true`. This is a native user-namespace +restriction and therefore applies to the MNG design. It is unlikely to block +an ordinary Coder workspace, which normally uses Pod networking and isolated +PID and IPC namespaces, but it excludes specialized workspaces that require +direct host networking or host process/IPC inspection. See the upstream +[user-namespace limitations](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#limitations). + +The claim that nested networking necessarily uses userspace NAT is not true +for the rootful-Docker design tested here. Dockerd can use Linux bridges, veth +interfaces, and kernel iptables/nftables NAT within the Pod's network +namespace using its namespaced `CAP_NET_ADMIN`; traffic then passes through +the normal Pod CNI and node/VPC networking. Userspace networking such as +`slirp4netns` is principally associated with rootless Docker. See Docker's +[packet-filtering and firewall documentation](https://docs.docker.com/engine/network/packet-filtering-firewalls/). + +The experiment and Compose follow-up proved nested-container outbound +connectivity, Compose service-name resolution through libc, published-port +reachability from the workspace, and same-node and cross-node reachability +through both the workspace Pod IP and a ClusterIP Service. Still untested are +CNI NetworkPolicy behavior, large-packet/MTU correctness, IPv6, and external +NodePort, LoadBalancer, or Ingress exposure. + +### Security comparison with Envbox/Sysbox + +This approach demonstrated the same fundamental user-namespace property as +the Envbox inner container: workspace UID 0 maps to an unprivileged host UID, +and the user-controlled workspace does not run as a host-privileged container. +It is therefore reasonable to describe the two approaches as pursuing the +same core isolation objective. + +The complete security postures are not yet proven equivalent. The native +approach removes Envbox's privileged outer container and the Sysbox manager, +filesystem service, and custom runtime from each workspace's trusted stack. +Kubernetes also assigned a distinct high host-UID range to each tested Pod, +rather than using Envbox's fixed `100000` user-namespace offset. These may be +security advantages. + +Conversely, the successful native Pod needs broad authority inside its user +namespace: the runtime's default capabilities plus the tested additions +`CAP_SYS_ADMIN` and `CAP_NET_ADMIN`, an unmasked `/proc`, an unconfined seccomp +profile, and a writable delegated cgroup hierarchy. The native approach also +lacks Sysbox-specific virtualization and mediation of system-container +behavior. Those differences must be evaluated rather than assumed equivalent. + +More precisely, the successful workspace was effectively privileged inside +its own sandbox. It could administer the Pod's mounts, network namespace, +processes, delegated cgroups, nested containers, PVC contents, credentials, +and reachable network resources. This broad authority is expected for a +Docker-capable developer workspace, where the developer is intentionally +allowed complete control inside the workspace. The relevant security +requirement is therefore containment: that authority must not extend to the +node, other workspaces or their storage, cluster-wide credentials, or network +resources the workspace is not authorized to reach. + +It was not effectively host-privileged. `hostUsers: false` mapped UID 0 to an +unprivileged high host UID and scoped namespaced capabilities such as +`CAP_SYS_ADMIN` and `CAP_NET_ADMIN` to resources owned by the Pod's user +namespace; capabilities such as `CAP_SYS_MODULE` cannot affect the host from +that namespace. The manifest also did not automatically grant host UID 0, +host namespaces, arbitrary host mounts, or unrestricted host-device access. +Those are meaningful differences from a Kubernetes `privileged: true` +container. See the upstream documentation on +[user-namespace capability boundaries](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/) +and +[privileged-container behavior](https://kubernetes.io/docs/concepts/security/linux-kernel-security-constraints/#privileged-containers). + +The remaining risk is still material because all containers share the node's +kernel. An unconfined seccomp profile permits the full syscall surface, +unmasked `/proc` exposes interfaces normally hidden by the runtime, and +namespaced `CAP_SYS_ADMIN` and `CAP_NET_ADMIN` plus +`allowPrivilegeEscalation: true` remove substantial defense in depth inside +the namespace. A kernel or user-namespace vulnerability could cross the +intended boundary. This design therefore relies heavily on the Linux user +namespace as its primary host-security boundary: it is meaningfully safer than +host-privileged DinD, but it is not equivalent to a conventional restricted +Pod and still requires a focused security review. + +The supported conclusion is therefore that the native MNG design reproduces +Envbox's fundamental non-host-root workspace boundary and may have a smaller +trusted stack, but full security equivalence requires focused escape, +cross-workspace, `/proc`, cgroup, device, mount, networking, and kernel attack- +surface testing. + +In a separate +[EKS Auto Mode/Bottlerocket experiment](../eks-auto-mode-bottlerocket-userns-rootful-dind-experiment/findings.md), +the first `hostUsers: false` probe failed because the AWS-managed Bottlerocket +node had `user.max_user_namespaces = 0`. A privileged node-preparation +DaemonSet, ordered with a NodePool startup taint, successfully raised that +sysctl and allowed a `hostUsers: false` Pod to use an EBS PVC. The subsequent cgroup +probes nevertheless found no writable delegated hierarchy, including in the +user-namespaced privileged control. Auto Mode's supported NodeClass interface +still exposes no equivalent of the custom `cgroup_writable = true` containerd +handler used by this successful MNG experiment. + +## Decision and remaining validation + +This approach is now a technically credible Envbox/Sysbox alternative for +Coder workspaces on configurable EKS MNGs. It is not yet a production-readiness +or security-equivalence result. + +Before recommending it, test at least: + +1. actual enforcement of memory, CPU, PID, and IO limits under load, rather + than only successful creation with limits; +2. multiple concurrent workspaces on one node, including resource-exhaustion + and cross-Pod isolation attempts; +3. the Coder agent and representative workspace images, broader Compose + configurations, Testcontainers, and devcontainer workflows; +4. Pod restart, node reboot, autoscaling, eviction, PVC reattachment, and + cleanup behavior; +5. admission-policy requirements and whether dedicating/gating the custom + RuntimeClass is operationally acceptable; +6. perform a focused security review of the smallest tested working profile: + the runtime-default capabilities plus namespaced `SYS_ADMIN` and + `NET_ADMIN`, effective `allowPrivilegeEscalation: true`, unmasked `/proc`, + unconfined seccomp, writable cgroups, and nested networking. The Pod user + namespace limits the authority of this profile but does not eliminate its + shared-kernel attack surface; +7. if a customer policy requires further minimization, test `SYS_ADMIN` and + `NET_ADMIN` individually and investigate whether a different Docker network + configuration or runtime mechanism can avoid fully unmasking `/proc`. + Also test whether a tailored seccomp profile or a no-new-keyring runtime + configuration can replace `Unconfined`. These are hardening opportunities, + not unresolved functional pass criteria; +8. monitor for a future supported EKS Auto Mode/Bottlerocket integration. A + privileged preparation DaemonSet overcame the tested node's initial + `user.max_user_namespaces = 0`, but writable cgroup delegation remained + unavailable and Auto Mode exposed no supported equivalent of the MNG's + custom containerd handler. + +The runtime-wide `cgroup_writable` handler also lacks the finer per-Pod policy +and cgroup-depth/descendant controls expected from a future first-class +Kubernetes writable-cgroups API. Until such an API is available and validated, +the custom handler should be limited to dedicated nodes and explicitly +authorized workloads. + +## Runbook corrections made during setup + +- The experiment node must not carry an untolerated custom `NoSchedule` taint + that prevents required EKS add-ons from scheduling. +- The test uses an explicit `gp3-csi` StorageClass rather than the legacy + in-tree `gp2` StorageClass. +- A new WFFC PVC was used for the second MNG to avoid binding the Docker test + to the first node's availability zone. +- The Docker test does not hide pipeline exit codes. +- A writable cgroup mount alone is insufficient: the workspace entrypoint + must keep processes out of the delegated root before enabling domain + controllers. diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-userns-probe.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-userns-probe.yaml new file mode 100644 index 0000000..052a294 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-userns-probe.yaml @@ -0,0 +1,37 @@ +apiVersion: v1 +kind: Pod +metadata: + name: cgroup-userns-probe + namespace: userns-rootful-dind +spec: + hostUsers: false + restartPolicy: Never + nodeSelector: + experiment.coder.com/userns-dind: "true" + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + securityContext: + privileged: true + procMount: Unmasked + command: ["/bin/sh", "-c"] + args: + - | + set -eux + echo '--- uid map ---' + cat /proc/self/uid_map + echo '--- cgroup membership ---' + cat /proc/self/cgroup + echo '--- cgroup mounts ---' + mount | grep -E 'cgroup|/sys/fs/cgroup' || true + echo '--- cgroup root metadata ---' + ls -ld /sys/fs/cgroup + echo '--- cgroup-root write probe ---' + test_dir=/sys/fs/cgroup/userns-dind-write-probe + if mkdir "$test_dir"; then + echo writable + rmdir "$test_dir" + else + echo not-writable + fi + sleep 600 diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/cgroup-probe.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/cgroup-probe.yaml new file mode 100644 index 0000000..d6d9f49 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/cgroup-probe.yaml @@ -0,0 +1,41 @@ +apiVersion: v1 +kind: Pod +metadata: + name: cgroup-writable-probe + namespace: userns-rootful-dind +spec: + runtimeClassName: runc-cgroup-writable + hostUsers: false + restartPolicy: Never + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + securityContext: + privileged: false + allowPrivilegeEscalation: true + procMount: Unmasked + capabilities: + add: ["ALL"] + seccompProfile: + type: Unconfined + command: ["/bin/sh", "-c"] + args: + - | + set -eux + echo '--- identity and user namespace ---' + id + cat /proc/self/uid_map + cat /proc/self/gid_map + echo '--- cgroup namespace and mount ---' + cat /proc/self/cgroup + mount | grep -E 'cgroup|/sys/fs/cgroup' + ls -ldn /sys/fs/cgroup + echo '--- available/delegated controllers ---' + cat /sys/fs/cgroup/cgroup.controllers + cat /sys/fs/cgroup/cgroup.subtree_control + echo '--- create child cgroup ---' + mkdir /sys/fs/cgroup/userns-dind-write-probe + ls -ldn /sys/fs/cgroup/userns-dind-write-probe + rmdir /sys/fs/cgroup/userns-dind-write-probe + echo cgroup-writable-probe-ok + sleep 600 diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/compose-network-peer.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/compose-network-peer.yaml new file mode 100644 index 0000000..bab8cc4 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/compose-network-peer.yaml @@ -0,0 +1,94 @@ +apiVersion: v1 +kind: Service +metadata: + name: compose-network-target + namespace: userns-rootful-dind +spec: + selector: + experiment.coder.com/compose-network-target: "true" + ports: + - name: http + port: 18080 + targetPort: 18080 +--- +# This headless Service provides a stable discovery mechanism for the +# workspace Pod IP. The peer resolves it, extracts the Pod IP, and then sends +# HTTP directly to that IP rather than through the ClusterIP Service. +apiVersion: v1 +kind: Service +metadata: + name: compose-network-target-headless + namespace: userns-rootful-dind +spec: + clusterIP: None + selector: + experiment.coder.com/compose-network-target: "true" + ports: + - name: http + port: 18080 + targetPort: 18080 +--- +apiVersion: v1 +kind: Pod +metadata: + name: compose-network-peer-probe + namespace: userns-rootful-dind +spec: + # The successful workspace is forced onto the cgroup-writable MNG by its + # RuntimeClass. Pinning this peer to the original MNG makes the two HTTP + # checks cross-node rather than merely same-node checks. + nodeSelector: + experiment.coder.com/userns-dind: "true" + automountServiceAccountToken: false + restartPolicy: Never + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + runAsNonRoot: true + runAsUser: 65534 + seccompProfile: + type: RuntimeDefault + env: + - name: PEER_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + command: ["/bin/sh", "-c"] + args: + - | + set -eux + + fetch_and_verify() { + url="$1" + output="$2" + attempts=0 + until wget -T 5 -t 1 -qO "$output" "$url"; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 120 ] || return 1 + sleep 2 + done + grep -q 'Welcome to nginx' "$output" + } + + service_url='http://compose-network-target.userns-rootful-dind.svc.cluster.local:18080/' + fetch_and_verify "$service_url" /tmp/service-body + echo "cross-node-clusterip-service-ok node=$PEER_NODE_NAME url=$service_url" + + headless_host='compose-network-target-headless.userns-rootful-dind.svc.cluster.local' + attempts=0 + pod_ip='' + until pod_ip="$(getent hosts "$headless_host" | awk 'NR == 1 { print $1 }')" && + [ -n "$pod_ip" ]; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 120 ] || exit 1 + sleep 2 + done + + pod_url="http://${pod_ip}:18080/" + fetch_and_verify "$pod_url" /tmp/pod-ip-body + echo "cross-node-direct-pod-ip-ok node=$PEER_NODE_NAME url=$pod_url" + echo compose-cross-node-network-tests-ok diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/nodegroup.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/nodegroup.yaml new file mode 100644 index 0000000..010a321 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/nodegroup.yaml @@ -0,0 +1,35 @@ +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig + +metadata: + name: userns-rootful-dind-mng-136 + region: us-east-2 + version: "1.36" + +managedNodeGroups: + - name: userns-dind-cgroup-writable + amiFamily: AmazonLinux2023 + instanceType: m6i.large + minSize: 1 + desiredCapacity: 1 + maxSize: 1 + labels: + experiment.coder.com/userns-dind-cgroup-writable: "true" + iam: + withAddonPolicies: + ebs: true + # eksctl prepends this partial NodeConfig to the EKS-generated NodeConfig; + # nodeadm merges it with the cluster connection details and normal AL2023 + # defaults. Containerd 2.x uses the config-v3 CRI runtime plugin path. + overrideBootstrapCommand: | + apiVersion: node.eks.aws/v1alpha1 + kind: NodeConfig + spec: + containerd: + config: | + [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc-cgroup-writable] + runtime_type = 'io.containerd.runc.v2' + cgroup_writable = true + + [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc-cgroup-writable.options] + SystemdCgroup = true diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/pvc.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/pvc.yaml new file mode 100644 index 0000000..647bac8 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/pvc.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: workspace-data-cgroup-writable + namespace: userns-rootful-dind +spec: + accessModes: [ReadWriteOnce] + storageClassName: gp3-csi + resources: + requests: + storage: 30Gi diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-default-capabilities.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-default-capabilities.yaml new file mode 100644 index 0000000..90f27c3 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-default-capabilities.yaml @@ -0,0 +1,202 @@ +apiVersion: v1 +kind: Pod +metadata: + name: rootful-dind-default-caps + namespace: userns-rootful-dind + labels: + experiment.coder.com/compose-network-target: "true" +spec: + runtimeClassName: runc-cgroup-writable + hostUsers: false + restartPolicy: Never + containers: + - name: workspace + # The original docker:27-dind tag resolved to Docker Engine 27.5.1. + # Pin the patch release so a replay does not silently change engines. + image: docker:27.5.1-dind + securityContext: + privileged: false + allowPrivilegeEscalation: true + procMount: Unmasked + seccompProfile: + type: Unconfined + env: + - name: DOCKER_TLS_CERTDIR + value: "" + - name: DOCKER_HOST + value: unix:///run/docker.sock + command: ["/bin/sh", "-c"] + args: + - | + set -eu + grep "^Cap" /proc/self/status + mkdir -p /run /workspace/docker-data /workspace/docker-artifacts-default-caps + artifacts=/workspace/docker-artifacts-default-caps + rm -f "$artifacts/test-complete" + alpine_image='alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d' + nginx_image='nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10' + + # Keep the delegated cgroup root free of processes so that domain + # controllers such as memory and io can be enabled for Docker's + # child-container hierarchy. All workspace processes, including + # this PID 1 shell and the dockerd it starts, live in a sibling + # cgroup instead. + mkdir /sys/fs/cgroup/workspace-processes + while read -r pid < /sys/fs/cgroup/cgroup.procs; do + [ -n "$pid" ] || break + echo "$pid" > /sys/fs/cgroup/workspace-processes/cgroup.procs + done + + for controller in cpu cpuset io memory pids; do + if grep -qw "$controller" /sys/fs/cgroup/cgroup.controllers; then + echo "+$controller" > /sys/fs/cgroup/cgroup.subtree_control + fi + done + + cat /sys/fs/cgroup/cgroup.type > "$artifacts/root-cgroup-type.txt" + cat /sys/fs/cgroup/cgroup.controllers > "$artifacts/root-cgroup-controllers.txt" + cat /sys/fs/cgroup/cgroup.subtree_control > "$artifacts/root-cgroup-subtree-control.txt" + cat /sys/fs/cgroup/cgroup.procs > "$artifacts/root-cgroup-procs.txt" + cat /sys/fs/cgroup/workspace-processes/cgroup.type > "$artifacts/workspace-cgroup-type.txt" + + id > "$artifacts/id.txt" + cat /proc/self/uid_map > "$artifacts/uid-map.txt" + cat /proc/self/gid_map > "$artifacts/gid-map.txt" + cat /proc/self/cgroup > "$artifacts/cgroup.txt" + mount | grep -E 'cgroup|/sys/fs/cgroup' > "$artifacts/cgroup-mount.txt" + mkdir /sys/fs/cgroup/dind-preflight + rmdir /sys/fs/cgroup/dind-preflight + + dockerd \ + --host=unix:///run/docker.sock \ + --data-root=/workspace/docker-data \ + --pidfile=/run/dockerd.pid \ + --exec-opt=native.cgroupdriver=cgroupfs \ + --cgroup-parent=docker \ + > "$artifacts/dockerd.log" 2>&1 & + dockerd_pid=$! + trap 'kill "$dockerd_pid" 2>/dev/null || true; wait "$dockerd_pid" 2>/dev/null || true' EXIT + + i=0 + until docker info > "$artifacts/docker-info.txt" 2>&1; do + i=$((i + 1)) + if [ "$i" -ge 90 ]; then + echo 'dockerd did not become ready' >&2 + cat "$artifacts/dockerd.log" >&2 || true + exit 1 + fi + sleep 2 + done + + docker version + docker info + docker info --format '{{.Driver}}' > "$artifacts/storage-driver.txt" + + docker pull "$alpine_image" + docker run --rm "$alpine_image" id > "$artifacts/child-id.txt" + cat "$artifacts/child-id.txt" + + mkdir -p /tmp/build-context + printf '%s\n' \ + "FROM $alpine_image" \ + 'RUN id' \ + 'CMD ["/bin/sh", "-c", "echo buildkit-ok"]' \ + > /tmp/build-context/Dockerfile + DOCKER_BUILDKIT=1 docker build -t userns-dind-smoke:3 /tmp/build-context \ + > "$artifacts/build.log" 2>&1 + cat "$artifacts/build.log" + docker run --rm userns-dind-smoke:3 > "$artifacts/build-result.txt" + cat "$artifacts/build-result.txt" + + docker rm -f httpd >/dev/null 2>&1 || true + docker run -d --name httpd "$nginx_image" + httpd_ip="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' httpd)" + docker run --rm "$alpine_image" wget -qO- "http://${httpd_ip}" \ + > "$artifacts/network-result.txt" + head -c 100 "$artifacts/network-result.txt" + echo + docker rm -f httpd + + if docker run --rm --memory=64m --pids-limit=64 "$alpine_image" true \ + > "$artifacts/resource-limit-result.txt" 2>&1; then + echo pass > "$artifacts/resource-limit-status.txt" + else + echo fail > "$artifacts/resource-limit-status.txt" + cat "$artifacts/resource-limit-result.txt" >&2 + exit 1 + fi + + # Reproduce the Docker Compose networking checks that were + # originally run interactively against this workspace. + compose_dir=/tmp/compose-network-test + compose_file="$compose_dir/compose.yaml" + mkdir -p "$compose_dir" + printf '%s\n' \ + 'services:' \ + ' server:' \ + " image: $nginx_image" \ + ' ports:' \ + ' - "18080:80"' \ + ' client:' \ + " image: $alpine_image" \ + ' command: ["sh", "-c", "sleep 3600"]' \ + > "$compose_file" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + up -d --remove-orphans + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + ps > "$artifacts/compose-ps.txt" + docker network inspect coder-nettest_default \ + > "$artifacts/compose-network-inspect.json" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + exec -T client sh -ec ' + attempts=0 + until getent hosts server >/tmp/server-hosts && + wget -qO /tmp/internal.html http://server/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q "Welcome to nginx" /tmp/internal.html + echo compose-service-dns-and-http-ok + + attempts=0 + until wget -qO /tmp/outbound.html http://example.com/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -qi "Example Domain" /tmp/outbound.html + echo compose-outbound-network-ok + ' > "$artifacts/compose-inner-network-result.txt" + cat "$artifacts/compose-inner-network-result.txt" + + attempts=0 + until wget -qO "$artifacts/compose-published.html" \ + http://127.0.0.1:18080/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q 'Welcome to nginx' "$artifacts/compose-published.html" + echo workspace-loopback-published-port-ok \ + > "$artifacts/compose-published-port-status.txt" + cat "$artifacts/compose-published-port-status.txt" + + touch "$artifacts/test-complete" /tmp/rootful-dind-test-complete + echo 'Domain-cgroup rootful DinD and Compose tests complete; keeping dockerd alive.' + wait "$dockerd_pid" + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data-cgroup-writable diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-default-proc.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-default-proc.yaml new file mode 100644 index 0000000..a01d173 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-default-proc.yaml @@ -0,0 +1,204 @@ +apiVersion: v1 +kind: Pod +metadata: + name: rootful-dind-default-proc + namespace: userns-rootful-dind + labels: + experiment.coder.com/compose-network-target: "true" +spec: + runtimeClassName: runc-cgroup-writable + hostUsers: false + restartPolicy: Never + containers: + - name: workspace + # The original docker:27-dind tag resolved to Docker Engine 27.5.1. + # Pin the patch release so a replay does not silently change engines. + image: docker:27.5.1-dind + securityContext: + privileged: false + allowPrivilegeEscalation: true + capabilities: + add: ["SYS_ADMIN", "NET_ADMIN"] + seccompProfile: + type: Unconfined + env: + - name: DOCKER_TLS_CERTDIR + value: "" + - name: DOCKER_HOST + value: unix:///run/docker.sock + command: ["/bin/sh", "-c"] + args: + - | + set -eu + grep "^Cap" /proc/self/status + grep -E " /proc($|/)" /proc/self/mountinfo | head -40 + mkdir -p /run /workspace/docker-data /workspace/docker-artifacts-default-proc + artifacts=/workspace/docker-artifacts-default-proc + rm -f "$artifacts/test-complete" + alpine_image='alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d' + nginx_image='nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10' + + # Keep the delegated cgroup root free of processes so that domain + # controllers such as memory and io can be enabled for Docker's + # child-container hierarchy. All workspace processes, including + # this PID 1 shell and the dockerd it starts, live in a sibling + # cgroup instead. + mkdir /sys/fs/cgroup/workspace-processes + while read -r pid < /sys/fs/cgroup/cgroup.procs; do + [ -n "$pid" ] || break + echo "$pid" > /sys/fs/cgroup/workspace-processes/cgroup.procs + done + + for controller in cpu cpuset io memory pids; do + if grep -qw "$controller" /sys/fs/cgroup/cgroup.controllers; then + echo "+$controller" > /sys/fs/cgroup/cgroup.subtree_control + fi + done + + cat /sys/fs/cgroup/cgroup.type > "$artifacts/root-cgroup-type.txt" + cat /sys/fs/cgroup/cgroup.controllers > "$artifacts/root-cgroup-controllers.txt" + cat /sys/fs/cgroup/cgroup.subtree_control > "$artifacts/root-cgroup-subtree-control.txt" + cat /sys/fs/cgroup/cgroup.procs > "$artifacts/root-cgroup-procs.txt" + cat /sys/fs/cgroup/workspace-processes/cgroup.type > "$artifacts/workspace-cgroup-type.txt" + + id > "$artifacts/id.txt" + cat /proc/self/uid_map > "$artifacts/uid-map.txt" + cat /proc/self/gid_map > "$artifacts/gid-map.txt" + cat /proc/self/cgroup > "$artifacts/cgroup.txt" + mount | grep -E 'cgroup|/sys/fs/cgroup' > "$artifacts/cgroup-mount.txt" + mkdir /sys/fs/cgroup/dind-preflight + rmdir /sys/fs/cgroup/dind-preflight + + dockerd \ + --host=unix:///run/docker.sock \ + --data-root=/workspace/docker-data \ + --pidfile=/run/dockerd.pid \ + --exec-opt=native.cgroupdriver=cgroupfs \ + --cgroup-parent=docker \ + > "$artifacts/dockerd.log" 2>&1 & + dockerd_pid=$! + trap 'kill "$dockerd_pid" 2>/dev/null || true; wait "$dockerd_pid" 2>/dev/null || true' EXIT + + i=0 + until docker info > "$artifacts/docker-info.txt" 2>&1; do + i=$((i + 1)) + if [ "$i" -ge 90 ]; then + echo 'dockerd did not become ready' >&2 + cat "$artifacts/dockerd.log" >&2 || true + exit 1 + fi + sleep 2 + done + + docker version + docker info + docker info --format '{{.Driver}}' > "$artifacts/storage-driver.txt" + + docker pull "$alpine_image" + docker run --rm "$alpine_image" id > "$artifacts/child-id.txt" + cat "$artifacts/child-id.txt" + + mkdir -p /tmp/build-context + printf '%s\n' \ + "FROM $alpine_image" \ + 'RUN id' \ + 'CMD ["/bin/sh", "-c", "echo buildkit-ok"]' \ + > /tmp/build-context/Dockerfile + DOCKER_BUILDKIT=1 docker build -t userns-dind-smoke:3 /tmp/build-context \ + > "$artifacts/build.log" 2>&1 + cat "$artifacts/build.log" + docker run --rm userns-dind-smoke:3 > "$artifacts/build-result.txt" + cat "$artifacts/build-result.txt" + + docker rm -f httpd >/dev/null 2>&1 || true + docker run -d --name httpd "$nginx_image" + httpd_ip="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' httpd)" + docker run --rm "$alpine_image" wget -qO- "http://${httpd_ip}" \ + > "$artifacts/network-result.txt" + head -c 100 "$artifacts/network-result.txt" + echo + docker rm -f httpd + + if docker run --rm --memory=64m --pids-limit=64 "$alpine_image" true \ + > "$artifacts/resource-limit-result.txt" 2>&1; then + echo pass > "$artifacts/resource-limit-status.txt" + else + echo fail > "$artifacts/resource-limit-status.txt" + cat "$artifacts/resource-limit-result.txt" >&2 + exit 1 + fi + + # Reproduce the Docker Compose networking checks that were + # originally run interactively against this workspace. + compose_dir=/tmp/compose-network-test + compose_file="$compose_dir/compose.yaml" + mkdir -p "$compose_dir" + printf '%s\n' \ + 'services:' \ + ' server:' \ + " image: $nginx_image" \ + ' ports:' \ + ' - "18080:80"' \ + ' client:' \ + " image: $alpine_image" \ + ' command: ["sh", "-c", "sleep 3600"]' \ + > "$compose_file" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + up -d --remove-orphans + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + ps > "$artifacts/compose-ps.txt" + docker network inspect coder-nettest_default \ + > "$artifacts/compose-network-inspect.json" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + exec -T client sh -ec ' + attempts=0 + until getent hosts server >/tmp/server-hosts && + wget -qO /tmp/internal.html http://server/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q "Welcome to nginx" /tmp/internal.html + echo compose-service-dns-and-http-ok + + attempts=0 + until wget -qO /tmp/outbound.html http://example.com/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -qi "Example Domain" /tmp/outbound.html + echo compose-outbound-network-ok + ' > "$artifacts/compose-inner-network-result.txt" + cat "$artifacts/compose-inner-network-result.txt" + + attempts=0 + until wget -qO "$artifacts/compose-published.html" \ + http://127.0.0.1:18080/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q 'Welcome to nginx' "$artifacts/compose-published.html" + echo workspace-loopback-published-port-ok \ + > "$artifacts/compose-published-port-status.txt" + cat "$artifacts/compose-published-port-status.txt" + + touch "$artifacts/test-complete" /tmp/rootful-dind-test-complete + echo 'Domain-cgroup rootful DinD and Compose tests complete; keeping dockerd alive.' + wait "$dockerd_pid" + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data-cgroup-writable diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-runtime-default-seccomp.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-runtime-default-seccomp.yaml new file mode 100644 index 0000000..99db190 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-runtime-default-seccomp.yaml @@ -0,0 +1,205 @@ +apiVersion: v1 +kind: Pod +metadata: + name: rootful-dind-runtime-default-seccomp + namespace: userns-rootful-dind + labels: + experiment.coder.com/compose-network-target: "true" +spec: + runtimeClassName: runc-cgroup-writable + hostUsers: false + restartPolicy: Never + containers: + - name: workspace + # The original docker:27-dind tag resolved to Docker Engine 27.5.1. + # Pin the patch release so a replay does not silently change engines. + image: docker:27.5.1-dind + securityContext: + privileged: false + allowPrivilegeEscalation: true + procMount: Unmasked + capabilities: + add: ["SYS_ADMIN", "NET_ADMIN"] + seccompProfile: + type: RuntimeDefault + env: + - name: DOCKER_TLS_CERTDIR + value: "" + - name: DOCKER_HOST + value: unix:///run/docker.sock + command: ["/bin/sh", "-c"] + args: + - | + set -eu + grep "^Cap" /proc/self/status + grep "^Seccomp" /proc/self/status + mkdir -p /run /workspace/docker-data /workspace/docker-artifacts-runtime-default-seccomp + artifacts=/workspace/docker-artifacts-runtime-default-seccomp + rm -f "$artifacts/test-complete" + alpine_image='alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d' + nginx_image='nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10' + + # Keep the delegated cgroup root free of processes so that domain + # controllers such as memory and io can be enabled for Docker's + # child-container hierarchy. All workspace processes, including + # this PID 1 shell and the dockerd it starts, live in a sibling + # cgroup instead. + mkdir /sys/fs/cgroup/workspace-processes + while read -r pid < /sys/fs/cgroup/cgroup.procs; do + [ -n "$pid" ] || break + echo "$pid" > /sys/fs/cgroup/workspace-processes/cgroup.procs + done + + for controller in cpu cpuset io memory pids; do + if grep -qw "$controller" /sys/fs/cgroup/cgroup.controllers; then + echo "+$controller" > /sys/fs/cgroup/cgroup.subtree_control + fi + done + + cat /sys/fs/cgroup/cgroup.type > "$artifacts/root-cgroup-type.txt" + cat /sys/fs/cgroup/cgroup.controllers > "$artifacts/root-cgroup-controllers.txt" + cat /sys/fs/cgroup/cgroup.subtree_control > "$artifacts/root-cgroup-subtree-control.txt" + cat /sys/fs/cgroup/cgroup.procs > "$artifacts/root-cgroup-procs.txt" + cat /sys/fs/cgroup/workspace-processes/cgroup.type > "$artifacts/workspace-cgroup-type.txt" + + id > "$artifacts/id.txt" + cat /proc/self/uid_map > "$artifacts/uid-map.txt" + cat /proc/self/gid_map > "$artifacts/gid-map.txt" + cat /proc/self/cgroup > "$artifacts/cgroup.txt" + mount | grep -E 'cgroup|/sys/fs/cgroup' > "$artifacts/cgroup-mount.txt" + mkdir /sys/fs/cgroup/dind-preflight + rmdir /sys/fs/cgroup/dind-preflight + + dockerd \ + --host=unix:///run/docker.sock \ + --data-root=/workspace/docker-data \ + --pidfile=/run/dockerd.pid \ + --exec-opt=native.cgroupdriver=cgroupfs \ + --cgroup-parent=docker \ + > "$artifacts/dockerd.log" 2>&1 & + dockerd_pid=$! + trap 'kill "$dockerd_pid" 2>/dev/null || true; wait "$dockerd_pid" 2>/dev/null || true' EXIT + + i=0 + until docker info > "$artifacts/docker-info.txt" 2>&1; do + i=$((i + 1)) + if [ "$i" -ge 90 ]; then + echo 'dockerd did not become ready' >&2 + cat "$artifacts/dockerd.log" >&2 || true + exit 1 + fi + sleep 2 + done + + docker version + docker info + docker info --format '{{.Driver}}' > "$artifacts/storage-driver.txt" + + docker pull "$alpine_image" + docker run --rm "$alpine_image" id > "$artifacts/child-id.txt" + cat "$artifacts/child-id.txt" + + mkdir -p /tmp/build-context + printf '%s\n' \ + "FROM $alpine_image" \ + 'RUN id' \ + 'CMD ["/bin/sh", "-c", "echo buildkit-ok"]' \ + > /tmp/build-context/Dockerfile + DOCKER_BUILDKIT=1 docker build -t userns-dind-smoke:3 /tmp/build-context \ + > "$artifacts/build.log" 2>&1 + cat "$artifacts/build.log" + docker run --rm userns-dind-smoke:3 > "$artifacts/build-result.txt" + cat "$artifacts/build-result.txt" + + docker rm -f httpd >/dev/null 2>&1 || true + docker run -d --name httpd "$nginx_image" + httpd_ip="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' httpd)" + docker run --rm "$alpine_image" wget -qO- "http://${httpd_ip}" \ + > "$artifacts/network-result.txt" + head -c 100 "$artifacts/network-result.txt" + echo + docker rm -f httpd + + if docker run --rm --memory=64m --pids-limit=64 "$alpine_image" true \ + > "$artifacts/resource-limit-result.txt" 2>&1; then + echo pass > "$artifacts/resource-limit-status.txt" + else + echo fail > "$artifacts/resource-limit-status.txt" + cat "$artifacts/resource-limit-result.txt" >&2 + exit 1 + fi + + # Reproduce the Docker Compose networking checks that were + # originally run interactively against this workspace. + compose_dir=/tmp/compose-network-test + compose_file="$compose_dir/compose.yaml" + mkdir -p "$compose_dir" + printf '%s\n' \ + 'services:' \ + ' server:' \ + " image: $nginx_image" \ + ' ports:' \ + ' - "18080:80"' \ + ' client:' \ + " image: $alpine_image" \ + ' command: ["sh", "-c", "sleep 3600"]' \ + > "$compose_file" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + up -d --remove-orphans + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + ps > "$artifacts/compose-ps.txt" + docker network inspect coder-nettest_default \ + > "$artifacts/compose-network-inspect.json" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + exec -T client sh -ec ' + attempts=0 + until getent hosts server >/tmp/server-hosts && + wget -qO /tmp/internal.html http://server/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q "Welcome to nginx" /tmp/internal.html + echo compose-service-dns-and-http-ok + + attempts=0 + until wget -qO /tmp/outbound.html http://example.com/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -qi "Example Domain" /tmp/outbound.html + echo compose-outbound-network-ok + ' > "$artifacts/compose-inner-network-result.txt" + cat "$artifacts/compose-inner-network-result.txt" + + attempts=0 + until wget -qO "$artifacts/compose-published.html" \ + http://127.0.0.1:18080/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q 'Welcome to nginx' "$artifacts/compose-published.html" + echo workspace-loopback-published-port-ok \ + > "$artifacts/compose-published-port-status.txt" + cat "$artifacts/compose-published-port-status.txt" + + touch "$artifacts/test-complete" /tmp/rootful-dind-test-complete + echo 'Domain-cgroup rootful DinD and Compose tests complete; keeping dockerd alive.' + wait "$dockerd_pid" + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data-cgroup-writable diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-targeted-capabilities.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-targeted-capabilities.yaml new file mode 100644 index 0000000..088b795 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind-targeted-capabilities.yaml @@ -0,0 +1,204 @@ +apiVersion: v1 +kind: Pod +metadata: + name: rootful-dind-targeted-caps + namespace: userns-rootful-dind + labels: + experiment.coder.com/compose-network-target: "true" +spec: + runtimeClassName: runc-cgroup-writable + hostUsers: false + restartPolicy: Never + containers: + - name: workspace + # The original docker:27-dind tag resolved to Docker Engine 27.5.1. + # Pin the patch release so a replay does not silently change engines. + image: docker:27.5.1-dind + securityContext: + privileged: false + allowPrivilegeEscalation: true + procMount: Unmasked + capabilities: + add: ["SYS_ADMIN", "NET_ADMIN"] + seccompProfile: + type: Unconfined + env: + - name: DOCKER_TLS_CERTDIR + value: "" + - name: DOCKER_HOST + value: unix:///run/docker.sock + command: ["/bin/sh", "-c"] + args: + - | + set -eu + grep "^Cap" /proc/self/status + mkdir -p /run /workspace/docker-data /workspace/docker-artifacts-targeted-caps + artifacts=/workspace/docker-artifacts-targeted-caps + rm -f "$artifacts/test-complete" + alpine_image='alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d' + nginx_image='nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10' + + # Keep the delegated cgroup root free of processes so that domain + # controllers such as memory and io can be enabled for Docker's + # child-container hierarchy. All workspace processes, including + # this PID 1 shell and the dockerd it starts, live in a sibling + # cgroup instead. + mkdir /sys/fs/cgroup/workspace-processes + while read -r pid < /sys/fs/cgroup/cgroup.procs; do + [ -n "$pid" ] || break + echo "$pid" > /sys/fs/cgroup/workspace-processes/cgroup.procs + done + + for controller in cpu cpuset io memory pids; do + if grep -qw "$controller" /sys/fs/cgroup/cgroup.controllers; then + echo "+$controller" > /sys/fs/cgroup/cgroup.subtree_control + fi + done + + cat /sys/fs/cgroup/cgroup.type > "$artifacts/root-cgroup-type.txt" + cat /sys/fs/cgroup/cgroup.controllers > "$artifacts/root-cgroup-controllers.txt" + cat /sys/fs/cgroup/cgroup.subtree_control > "$artifacts/root-cgroup-subtree-control.txt" + cat /sys/fs/cgroup/cgroup.procs > "$artifacts/root-cgroup-procs.txt" + cat /sys/fs/cgroup/workspace-processes/cgroup.type > "$artifacts/workspace-cgroup-type.txt" + + id > "$artifacts/id.txt" + cat /proc/self/uid_map > "$artifacts/uid-map.txt" + cat /proc/self/gid_map > "$artifacts/gid-map.txt" + cat /proc/self/cgroup > "$artifacts/cgroup.txt" + mount | grep -E 'cgroup|/sys/fs/cgroup' > "$artifacts/cgroup-mount.txt" + mkdir /sys/fs/cgroup/dind-preflight + rmdir /sys/fs/cgroup/dind-preflight + + dockerd \ + --host=unix:///run/docker.sock \ + --data-root=/workspace/docker-data \ + --pidfile=/run/dockerd.pid \ + --exec-opt=native.cgroupdriver=cgroupfs \ + --cgroup-parent=docker \ + > "$artifacts/dockerd.log" 2>&1 & + dockerd_pid=$! + trap 'kill "$dockerd_pid" 2>/dev/null || true; wait "$dockerd_pid" 2>/dev/null || true' EXIT + + i=0 + until docker info > "$artifacts/docker-info.txt" 2>&1; do + i=$((i + 1)) + if [ "$i" -ge 90 ]; then + echo 'dockerd did not become ready' >&2 + cat "$artifacts/dockerd.log" >&2 || true + exit 1 + fi + sleep 2 + done + + docker version + docker info + docker info --format '{{.Driver}}' > "$artifacts/storage-driver.txt" + + docker pull "$alpine_image" + docker run --rm "$alpine_image" id > "$artifacts/child-id.txt" + cat "$artifacts/child-id.txt" + + mkdir -p /tmp/build-context + printf '%s\n' \ + "FROM $alpine_image" \ + 'RUN id' \ + 'CMD ["/bin/sh", "-c", "echo buildkit-ok"]' \ + > /tmp/build-context/Dockerfile + DOCKER_BUILDKIT=1 docker build -t userns-dind-smoke:3 /tmp/build-context \ + > "$artifacts/build.log" 2>&1 + cat "$artifacts/build.log" + docker run --rm userns-dind-smoke:3 > "$artifacts/build-result.txt" + cat "$artifacts/build-result.txt" + + docker rm -f httpd >/dev/null 2>&1 || true + docker run -d --name httpd "$nginx_image" + httpd_ip="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' httpd)" + docker run --rm "$alpine_image" wget -qO- "http://${httpd_ip}" \ + > "$artifacts/network-result.txt" + head -c 100 "$artifacts/network-result.txt" + echo + docker rm -f httpd + + if docker run --rm --memory=64m --pids-limit=64 "$alpine_image" true \ + > "$artifacts/resource-limit-result.txt" 2>&1; then + echo pass > "$artifacts/resource-limit-status.txt" + else + echo fail > "$artifacts/resource-limit-status.txt" + cat "$artifacts/resource-limit-result.txt" >&2 + exit 1 + fi + + # Reproduce the Docker Compose networking checks that were + # originally run interactively against this workspace. + compose_dir=/tmp/compose-network-test + compose_file="$compose_dir/compose.yaml" + mkdir -p "$compose_dir" + printf '%s\n' \ + 'services:' \ + ' server:' \ + " image: $nginx_image" \ + ' ports:' \ + ' - "18080:80"' \ + ' client:' \ + " image: $alpine_image" \ + ' command: ["sh", "-c", "sleep 3600"]' \ + > "$compose_file" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + up -d --remove-orphans + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + ps > "$artifacts/compose-ps.txt" + docker network inspect coder-nettest_default \ + > "$artifacts/compose-network-inspect.json" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + exec -T client sh -ec ' + attempts=0 + until getent hosts server >/tmp/server-hosts && + wget -qO /tmp/internal.html http://server/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q "Welcome to nginx" /tmp/internal.html + echo compose-service-dns-and-http-ok + + attempts=0 + until wget -qO /tmp/outbound.html http://example.com/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -qi "Example Domain" /tmp/outbound.html + echo compose-outbound-network-ok + ' > "$artifacts/compose-inner-network-result.txt" + cat "$artifacts/compose-inner-network-result.txt" + + attempts=0 + until wget -qO "$artifacts/compose-published.html" \ + http://127.0.0.1:18080/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q 'Welcome to nginx' "$artifacts/compose-published.html" + echo workspace-loopback-published-port-ok \ + > "$artifacts/compose-published-port-status.txt" + cat "$artifacts/compose-published-port-status.txt" + + touch "$artifacts/test-complete" /tmp/rootful-dind-test-complete + echo 'Domain-cgroup rootful DinD and Compose tests complete; keeping dockerd alive.' + wait "$dockerd_pid" + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data-cgroup-writable diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind.yaml new file mode 100644 index 0000000..c319544 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind.yaml @@ -0,0 +1,203 @@ +apiVersion: v1 +kind: Pod +metadata: + name: rootful-dind-cgroup-writable + namespace: userns-rootful-dind + labels: + experiment.coder.com/compose-network-target: "true" +spec: + runtimeClassName: runc-cgroup-writable + hostUsers: false + restartPolicy: Never + containers: + - name: workspace + # The original docker:27-dind tag resolved to Docker Engine 27.5.1. + # Pin the patch release so a replay does not silently change engines. + image: docker:27.5.1-dind + securityContext: + privileged: false + allowPrivilegeEscalation: true + procMount: Unmasked + capabilities: + add: ["ALL"] + seccompProfile: + type: Unconfined + env: + - name: DOCKER_TLS_CERTDIR + value: "" + - name: DOCKER_HOST + value: unix:///run/docker.sock + command: ["/bin/sh", "-c"] + args: + - | + set -eu + mkdir -p /run /workspace/docker-data /workspace/docker-artifacts-cgroup-domain + artifacts=/workspace/docker-artifacts-cgroup-domain + rm -f "$artifacts/test-complete" + alpine_image='alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d' + nginx_image='nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10' + + # Keep the delegated cgroup root free of processes so that domain + # controllers such as memory and io can be enabled for Docker's + # child-container hierarchy. All workspace processes, including + # this PID 1 shell and the dockerd it starts, live in a sibling + # cgroup instead. + mkdir /sys/fs/cgroup/workspace-processes + while read -r pid < /sys/fs/cgroup/cgroup.procs; do + [ -n "$pid" ] || break + echo "$pid" > /sys/fs/cgroup/workspace-processes/cgroup.procs + done + + for controller in cpu cpuset io memory pids; do + if grep -qw "$controller" /sys/fs/cgroup/cgroup.controllers; then + echo "+$controller" > /sys/fs/cgroup/cgroup.subtree_control + fi + done + + cat /sys/fs/cgroup/cgroup.type > "$artifacts/root-cgroup-type.txt" + cat /sys/fs/cgroup/cgroup.controllers > "$artifacts/root-cgroup-controllers.txt" + cat /sys/fs/cgroup/cgroup.subtree_control > "$artifacts/root-cgroup-subtree-control.txt" + cat /sys/fs/cgroup/cgroup.procs > "$artifacts/root-cgroup-procs.txt" + cat /sys/fs/cgroup/workspace-processes/cgroup.type > "$artifacts/workspace-cgroup-type.txt" + + id > "$artifacts/id.txt" + cat /proc/self/uid_map > "$artifacts/uid-map.txt" + cat /proc/self/gid_map > "$artifacts/gid-map.txt" + cat /proc/self/cgroup > "$artifacts/cgroup.txt" + mount | grep -E 'cgroup|/sys/fs/cgroup' > "$artifacts/cgroup-mount.txt" + mkdir /sys/fs/cgroup/dind-preflight + rmdir /sys/fs/cgroup/dind-preflight + + dockerd \ + --host=unix:///run/docker.sock \ + --data-root=/workspace/docker-data \ + --pidfile=/run/dockerd.pid \ + --exec-opt=native.cgroupdriver=cgroupfs \ + --cgroup-parent=docker \ + > "$artifacts/dockerd.log" 2>&1 & + dockerd_pid=$! + trap 'kill "$dockerd_pid" 2>/dev/null || true; wait "$dockerd_pid" 2>/dev/null || true' EXIT + + i=0 + until docker info > "$artifacts/docker-info.txt" 2>&1; do + i=$((i + 1)) + if [ "$i" -ge 90 ]; then + echo 'dockerd did not become ready' >&2 + cat "$artifacts/dockerd.log" >&2 || true + exit 1 + fi + sleep 2 + done + + docker version + docker info + docker info --format '{{.Driver}}' > "$artifacts/storage-driver.txt" + + docker pull "$alpine_image" + docker run --rm "$alpine_image" id > "$artifacts/child-id.txt" + cat "$artifacts/child-id.txt" + + mkdir -p /tmp/build-context + printf '%s\n' \ + "FROM $alpine_image" \ + 'RUN id' \ + 'CMD ["/bin/sh", "-c", "echo buildkit-ok"]' \ + > /tmp/build-context/Dockerfile + DOCKER_BUILDKIT=1 docker build -t userns-dind-smoke:3 /tmp/build-context \ + > "$artifacts/build.log" 2>&1 + cat "$artifacts/build.log" + docker run --rm userns-dind-smoke:3 > "$artifacts/build-result.txt" + cat "$artifacts/build-result.txt" + + docker rm -f httpd >/dev/null 2>&1 || true + docker run -d --name httpd "$nginx_image" + httpd_ip="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' httpd)" + docker run --rm "$alpine_image" wget -qO- "http://${httpd_ip}" \ + > "$artifacts/network-result.txt" + head -c 100 "$artifacts/network-result.txt" + echo + docker rm -f httpd + + if docker run --rm --memory=64m --pids-limit=64 "$alpine_image" true \ + > "$artifacts/resource-limit-result.txt" 2>&1; then + echo pass > "$artifacts/resource-limit-status.txt" + else + echo fail > "$artifacts/resource-limit-status.txt" + cat "$artifacts/resource-limit-result.txt" >&2 + exit 1 + fi + + # Reproduce the Docker Compose networking checks that were + # originally run interactively against this workspace. + compose_dir=/tmp/compose-network-test + compose_file="$compose_dir/compose.yaml" + mkdir -p "$compose_dir" + printf '%s\n' \ + 'services:' \ + ' server:' \ + " image: $nginx_image" \ + ' ports:' \ + ' - "18080:80"' \ + ' client:' \ + " image: $alpine_image" \ + ' command: ["sh", "-c", "sleep 3600"]' \ + > "$compose_file" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + up -d --remove-orphans + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + ps > "$artifacts/compose-ps.txt" + docker network inspect coder-nettest_default \ + > "$artifacts/compose-network-inspect.json" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + exec -T client sh -ec ' + attempts=0 + until getent hosts server >/tmp/server-hosts && + wget -qO /tmp/internal.html http://server/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q "Welcome to nginx" /tmp/internal.html + echo compose-service-dns-and-http-ok + + attempts=0 + until wget -qO /tmp/outbound.html http://example.com/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -qi "Example Domain" /tmp/outbound.html + echo compose-outbound-network-ok + ' > "$artifacts/compose-inner-network-result.txt" + cat "$artifacts/compose-inner-network-result.txt" + + attempts=0 + until wget -qO "$artifacts/compose-published.html" \ + http://127.0.0.1:18080/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q 'Welcome to nginx' "$artifacts/compose-published.html" + echo workspace-loopback-published-port-ok \ + > "$artifacts/compose-published-port-status.txt" + cat "$artifacts/compose-published-port-status.txt" + + touch "$artifacts/test-complete" /tmp/rootful-dind-test-complete + echo 'Domain-cgroup rootful DinD and Compose tests complete; keeping dockerd alive.' + wait "$dockerd_pid" + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data-cgroup-writable diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runbook.md b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runbook.md new file mode 100644 index 0000000..f82ea4c --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runbook.md @@ -0,0 +1,170 @@ +# Cgroup-writable RuntimeClass follow-up + +This follow-up keeps the existing EKS 1.36 cluster and original AL2023 MNG as +the control. It adds a second AL2023 MNG whose containerd 2.x configuration +registers a stock-runc handler with `cgroup_writable = true`. + +The name `runc-cgroup-writable` is local configuration, not another runtime +binary. The handler still uses the node's stock `io.containerd.runc.v2`. + +## 1. Create the second MNG + +```bash +eksctl create nodegroup -f nodegroup.yaml + +kubectl get nodes \ + -L eks.amazonaws.com/nodegroup,experiment.coder.com/userns-dind-cgroup-writable +``` + +Do not continue until the new node is `Ready` and has the expected label. + +## 2. Register the RuntimeClass + +```bash +kubectl apply -f runtimeclass.yaml +kubectl get runtimeclass runc-cgroup-writable -o yaml +``` + +## 3. Prove cgroup delegation + +```bash +kubectl apply -f cgroup-probe.yaml +kubectl -n userns-rootful-dind wait \ + --for=condition=Ready pod/cgroup-writable-probe --timeout=15m +kubectl -n userns-rootful-dind logs cgroup-writable-probe +``` + +The required result is `cgroup-writable-probe-ok`. If `mkdir` under +`/sys/fs/cgroup` fails, stop and inspect the new node's generated containerd +configuration; do not proceed to Docker. + +## 4. Run rootful Docker + +Delete the probe, create the new WFFC PVC, and start the Docker Pod: + +```bash +kubectl -n userns-rootful-dind delete pod cgroup-writable-probe --wait=true +kubectl -n userns-rootful-dind delete pod rootful-dind-cgroup-writable \ + --ignore-not-found --wait=true +kubectl apply -f pvc.yaml +kubectl apply -f rootful-dind.yaml +kubectl -n userns-rootful-dind wait \ + --for=condition=Ready pod/rootful-dind-cgroup-writable --timeout=15m +kubectl -n userns-rootful-dind logs -f rootful-dind-cgroup-writable +``` + +Stop following logs after this completion message: + +```text +Domain-cgroup rootful DinD and Compose tests complete; keeping dockerd alive. +``` + +The Pod becoming Ready only proves that its container started; it does not +prove that the embedded Docker, BuildKit, resource-limit, and Compose tests +finished. Verify the completion artifact before interpreting the result: + +```bash +until kubectl -n userns-rootful-dind exec rootful-dind-cgroup-writable -- \ + test -f /tmp/rootful-dind-test-complete; do + sleep 2 +done +``` + +Inspect results with: + +```bash +kubectl -n userns-rootful-dind exec rootful-dind-cgroup-writable -- sh -c ' + artifacts=/workspace/docker-artifacts-cgroup-domain + cat "$artifacts/uid-map.txt" + cat "$artifacts/cgroup.txt" + cat "$artifacts/root-cgroup-type.txt" + cat "$artifacts/root-cgroup-subtree-control.txt" + cat "$artifacts/storage-driver.txt" + cat "$artifacts/build-result.txt" + cat "$artifacts/resource-limit-status.txt" + cat "$artifacts/compose-inner-network-result.txt" + cat "$artifacts/compose-published-port-status.txt" +' +``` + +The successful manifest pins Docker Engine 27.5.1 and the observed Alpine and +Nginx image digests. It leaves the Compose server running with port 18080 +published into the workspace Pod network namespace for the peer tests below. + +## 5. Reproduce same-node and cross-node networking + +The embedded checks above prove Compose service-name resolution, HTTP between +Compose services, outbound networking, and access to the published port from +the workspace itself. The companion manifest adds: + +- a ClusterIP Service selecting the workspace Pod; +- a headless Service used only to discover the workspace Pod IP; and +- a restricted peer Pod pinned to the original MNG, which accesses the Nginx + container through both the ClusterIP and the workspace Pod IP. + +Delete any previous completed peer Pod, apply the manifest, and wait for the +new peer to finish: + +```bash +kubectl -n userns-rootful-dind delete pod compose-network-peer-probe \ + --ignore-not-found --wait=true +kubectl apply -f compose-network-peer.yaml +kubectl -n userns-rootful-dind wait \ + --for=jsonpath='{.status.phase}'=Succeeded \ + pod/compose-network-peer-probe --timeout=10m +kubectl -n userns-rootful-dind logs compose-network-peer-probe +``` + +The required final line is: + +```text +compose-cross-node-network-tests-ok +``` + +Confirm that the peer and workspace actually ran on different nodes; do not +call this a cross-node pass based only on the peer log: + +```bash +workspace_node="$(kubectl -n userns-rootful-dind get pod \ + rootful-dind-cgroup-writable -o jsonpath='{.spec.nodeName}')" +peer_node="$(kubectl -n userns-rootful-dind get pod \ + compose-network-peer-probe -o jsonpath='{.spec.nodeName}')" +printf 'workspace_node=%s\npeer_node=%s\n' "$workspace_node" "$peer_node" +test "$workspace_node" != "$peer_node" + +kubectl -n userns-rootful-dind get \ + service/compose-network-target \ + service/compose-network-target-headless \ + pod/rootful-dind-cgroup-writable \ + pod/compose-network-peer-probe -o wide +``` + +## Interpretation + +- Cgroup probe fails: the node/runtime configuration did not produce a usable + delegation boundary. +- Cgroup probe passes but Docker fails: capture `dockerd.log`; the next + boundary is likely device, network, or controller delegation rather than + image storage. +- Docker run/build/network/resource-limit and embedded Compose tests pass, + but peer test fails: preserve the peer log and distinguish Service, DNS, + Pod-IP routing, node placement, and CNI policy failures. +- All embedded and cross-node tests pass: the native-userns design is + technically viable on a purpose-configured MNG, but the runtime-wide + writable-cgroup handler and broad in-user-namespace security context still + need production security and exhaustion reviews. + +## Cleanup + +The Service objects can be applied repeatedly. The completed peer Pod must be +deleted before each replay because a Pod's command is immutable. + +```bash +kubectl delete -f compose-network-peer.yaml --ignore-not-found +kubectl delete -f rootful-dind.yaml --ignore-not-found +kubectl delete -f pvc.yaml --ignore-not-found +kubectl delete -f runtimeclass.yaml --ignore-not-found +``` + +Delete the `userns-dind-cgroup-writable` node group separately when the entire +experiment is complete. diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runtimeclass.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runtimeclass.yaml new file mode 100644 index 0000000..8952c00 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runtimeclass.yaml @@ -0,0 +1,8 @@ +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: runc-cgroup-writable +handler: runc-cgroup-writable +scheduling: + nodeSelector: + experiment.coder.com/userns-dind-cgroup-writable: "true" diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/test-capabilities-default.log b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/test-capabilities-default.log new file mode 100644 index 0000000..3c8680e --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/test-capabilities-default.log @@ -0,0 +1,83 @@ +CapInh: 0000000000000000 +CapPrm: 00000000a80425fb +CapEff: 00000000a80425fb +CapBnd: 00000000a80425fb +CapAmb: 0000000000000000 +dockerd did not become ready +time="2026-08-08T22:39:35.239098329Z" level=info msg="Starting up" +time="2026-08-08T22:39:35.242384681Z" level=warning msg="Error while setting daemon root propagation, this is not generally critical but may cause some functionality to not work or fallback to less desirable behavior" dir=/workspace/docker-data error="could not setup daemon root propagation to shared: mount /workspace/docker-data:/workspace/docker-data, flags: 0x1000: operation not permitted" +time="2026-08-08T22:39:35.242901651Z" level=info msg="containerd not running, starting managed containerd" +time="2026-08-08T22:39:35.246082899Z" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=51 +time="2026-08-08T22:39:35.272280257Z" level=info msg="starting containerd" revision=bcc810d6b9066471b0b6fa75f557a15a1cbf31bb version=v1.7.25 +time="2026-08-08T22:39:35.286074929Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1 +time="2026-08-08T22:39:35.286107857Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1 +time="2026-08-08T22:39:35.286300837Z" level=info msg="loading plugin \"io.containerd.warning.v1.deprecations\"..." type=io.containerd.warning.v1 +time="2026-08-08T22:39:35.286314735Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.286378237Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." error="no scratch file generator: skip plugin" type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.286386586Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.286394896Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." error="devmapper not configured: skip plugin" type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.286400742Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.286442713Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.286615493Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.288708952Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"ip: can't find device 'aufs'\\nmodprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.288729849Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.288816307Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /workspace/docker-data/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1 +time="2026-08-08T22:39:35.288824955Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1 +time="2026-08-08T22:39:35.288835998Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1 +time="2026-08-08T22:39:35.288884362Z" level=info msg="metadata content store policy set" policy=shared +time="2026-08-08T22:39:35.292855026Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1 +time="2026-08-08T22:39:35.292888927Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1 +time="2026-08-08T22:39:35.292914976Z" level=info msg="loading plugin \"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1 +time="2026-08-08T22:39:35.292926704Z" level=info msg="loading plugin \"io.containerd.streaming.v1.manager\"..." type=io.containerd.streaming.v1 +time="2026-08-08T22:39:35.292938963Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1 +time="2026-08-08T22:39:35.293078983Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1 +time="2026-08-08T22:39:35.293311362Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2 +time="2026-08-08T22:39:35.293458065Z" level=info msg="loading plugin \"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2 +time="2026-08-08T22:39:35.293478258Z" level=info msg="loading plugin \"io.containerd.sandbox.store.v1.local\"..." type=io.containerd.sandbox.store.v1 +time="2026-08-08T22:39:35.293495261Z" level=info msg="loading plugin \"io.containerd.sandbox.controller.v1.local\"..." type=io.containerd.sandbox.controller.v1 +time="2026-08-08T22:39:35.293509772Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1 +time="2026-08-08T22:39:35.293523774Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1 +time="2026-08-08T22:39:35.293536487Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1 +time="2026-08-08T22:39:35.293568369Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1 +time="2026-08-08T22:39:35.293589654Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1 +time="2026-08-08T22:39:35.293605338Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1 +time="2026-08-08T22:39:35.293616349Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1 +time="2026-08-08T22:39:35.293624241Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1 +time="2026-08-08T22:39:35.293637756Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293646833Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293655262Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293663978Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293672287Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293685114Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293695735Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293704043Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293712074Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293721874Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293728926Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293736353Z" level=info msg="loading plugin \"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293745411Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293757698Z" level=info msg="loading plugin \"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1 +time="2026-08-08T22:39:35.293772570Z" level=info msg="loading plugin \"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293784442Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293791489Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1 +time="2026-08-08T22:39:35.293830916Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1 +time="2026-08-08T22:39:35.293849736Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.tracing.processor.v1 +time="2026-08-08T22:39:35.293856399Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1 +time="2026-08-08T22:39:35.293864036Z" level=info msg="skip loading plugin \"io.containerd.internal.v1.tracing\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.internal.v1 +time="2026-08-08T22:39:35.293869563Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1 +time="2026-08-08T22:39:35.293877310Z" level=info msg="loading plugin \"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1 +time="2026-08-08T22:39:35.293883434Z" level=info msg="NRI interface is disabled by configuration." +time="2026-08-08T22:39:35.294100172Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock +time="2026-08-08T22:39:35.294181551Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc +time="2026-08-08T22:39:35.294241069Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock +time="2026-08-08T22:39:35.294259494Z" level=info msg="containerd successfully booted in 0.022632s" +time="2026-08-08T22:39:36.250311157Z" level=info msg="OTEL tracing is not configured, using no-op tracer provider" +time="2026-08-08T22:39:36.264997886Z" level=error msg="failed to mount overlay: operation not permitted" storage-driver=overlay2 +time="2026-08-08T22:39:36.265079302Z" level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fuse-overlayfs +time="2026-08-08T22:39:36.266663615Z" level=info msg="Loading containers: start." +time="2026-08-08T22:39:36.268840998Z" level=info msg="unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.11 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)`" error="exit status 4" +time="2026-08-08T22:39:36.278588270Z" level=info msg="stopping event stream following graceful shutdown" error="" module=libcontainerd namespace=moby +time="2026-08-08T22:39:36.278966620Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd +time="2026-08-08T22:39:36.279019265Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby +failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.11 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root) + (exit status 4) diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/test-capabilities-sys-admin-net-admin.log b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/test-capabilities-sys-admin-net-admin.log new file mode 100644 index 0000000..8552539 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/test-capabilities-sys-admin-net-admin.log @@ -0,0 +1,189 @@ +CapInh: 0000000000000000 +CapPrm: 00000000a82435fb +CapEff: 00000000a82435fb +CapBnd: 00000000a82435fb +CapAmb: 0000000000000000 +Client: + Version: 27.5.1 + API version: 1.47 + Go version: go1.22.11 + Git commit: 9f9e405 + Built: Wed Jan 22 13:40:02 2025 + OS/Arch: linux/amd64 + Context: default + +Server: Docker Engine - Community + Engine: + Version: 27.5.1 + API version: 1.47 (minimum version 1.24) + Go version: go1.22.11 + Git commit: 4c9b3b0 + Built: Wed Jan 22 13:41:24 2025 + OS/Arch: linux/amd64 + Experimental: false + containerd: + Version: v1.7.25 + GitCommit: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb + runc: + Version: 1.2.4 + GitCommit: v1.2.4-0-g6c52b3f + docker-init: + Version: 0.19.0 + GitCommit: de40ad0 +Client: + Version: 27.5.1 + Context: default + Debug Mode: false + Plugins: + buildx: Docker Buildx (Docker Inc.) + Version: v0.20.1 + Path: /usr/local/libexec/docker/cli-plugins/docker-buildx + compose: Docker Compose (Docker Inc.) + Version: v2.33.0 + Path: /usr/local/libexec/docker/cli-plugins/docker-compose + +Server: + Containers: 0 + Running: 0 + Paused: 0 + Stopped: 0 + Images: 0 + Server Version: 27.5.1 + Storage Driver: overlay2 + Backing Filesystem: extfs + Supports d_type: true + Using metacopy: false + Native Overlay Diff: true + userxattr: true + Logging Driver: json-file + Cgroup Driver: cgroupfs + Cgroup Version: 2 + Plugins: + Volume: local + Network: bridge host ipvlan macvlan null overlay + Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog + Swarm: inactive + Runtimes: io.containerd.runc.v2 runc + Default Runtime: runc + Init Binary: docker-init + containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb + runc version: v1.2.4-0-g6c52b3f + init version: de40ad0 + Security Options: + seccomp + Profile: builtin + cgroupns + Kernel Version: 6.18.38-76.139.amzn2023.x86_64 + Operating System: Alpine Linux v3.21 (containerized) + OSType: linux + Architecture: x86_64 + CPUs: 2 + Total Memory: 7.597GiB + Name: rootful-dind-targeted-caps + ID: 48af6e61-b8e2-462c-9062-6b7c5cfbcfb9 + Docker Root Dir: /workspace/docker-data + Debug Mode: false + Experimental: false + Insecure Registries: + 127.0.0.0/8 + Live Restore Enabled: false + Product License: Community Engine + +docker.io/library/alpine@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d: Pulling from library/alpine +897d797d2723: Pulling fs layer +897d797d2723: Verifying Checksum +897d797d2723: Download complete +897d797d2723: Pull complete +Digest: sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d +Status: Downloaded newer image for alpine@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d +docker.io/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d +uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) +#0 building with "default" instance using docker driver + +#1 [internal] load build definition from Dockerfile +#1 transferring dockerfile: 230B done +#1 DONE 0.0s + +#2 [internal] load metadata for docker.io/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d +#2 DONE 0.0s + +#3 [internal] load .dockerignore +#3 transferring context: 2B done +#3 DONE 0.0s + +#4 [1/2] FROM docker.io/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d +#4 DONE 0.0s + +#5 [2/2] RUN id +#5 0.159 uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) +#5 DONE 0.2s + +#6 exporting to image +#6 exporting layers 0.0s done +#6 writing image sha256:32dc6c7151c4fe9f94f0db948b4b7d16ef456f5f570e534568ddb30a4fd1437d done +#6 naming to docker.io/library/userns-dind-smoke:3 done +#6 DONE 0.1s +buildkit-ok +Unable to find image 'nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10' locally +docker.io/library/nginx@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10: Pulling from library/nginx +f18232174bc9: Pulling fs layer +61ca4f733c80: Pulling fs layer +b464cfdf2a63: Pulling fs layer +d7e507024086: Pulling fs layer +81bd8ed7ec67: Pulling fs layer +197eb75867ef: Pulling fs layer +34a64644b756: Pulling fs layer +39c2ddfd6010: Pulling fs layer +81bd8ed7ec67: Waiting +197eb75867ef: Waiting +34a64644b756: Waiting +39c2ddfd6010: Waiting +d7e507024086: Waiting +b464cfdf2a63: Verifying Checksum +b464cfdf2a63: Download complete +61ca4f733c80: Verifying Checksum +61ca4f733c80: Download complete +f18232174bc9: Verifying Checksum +f18232174bc9: Download complete +d7e507024086: Verifying Checksum +d7e507024086: Download complete +81bd8ed7ec67: Verifying Checksum +81bd8ed7ec67: Download complete +197eb75867ef: Verifying Checksum +197eb75867ef: Download complete +34a64644b756: Verifying Checksum +34a64644b756: Download complete +f18232174bc9: Pull complete +39c2ddfd6010: Verifying Checksum +39c2ddfd6010: Download complete +61ca4f733c80: Pull complete +b464cfdf2a63: Pull complete +d7e507024086: Pull complete +81bd8ed7ec67: Pull complete +197eb75867ef: Pull complete +34a64644b756: Pull complete +39c2ddfd6010: Pull complete +Digest: sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10 +Status: Downloaded newer image for nginx@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10 +2e208751865f5f9c6d7f9afc2c47ebeda375611b504bf79c68e15d80abc0e39d + + + +Welcome to nginx! +