Linux Namespaces and Cgroups: The Building Blocks of Containers

Namespaces

A namespace wraps a global system resource in a layer of abstraction so that processes inside the namespace believe they have their own private copy of that resource. Changes to the resource inside one namespace are invisible to processes in other namespaces. Every process on a Linux system is always inside a set of namespaces. On a host with no containerisation happening, all processes share the same namespaces. When you start a container, the container runtime creates new namespaces and launches the container processes inside them.

Linux currently has eight namespace types. A process can be in a different namespace for each one independently. The kernel tracks namespace membership through file descriptors in /proc/<pid>/ns/. Two processes sharing the same inode number for a given namespace type are in the same namespace.

How namespaces are created

Namespaces are created through three syscalls. clone() creates a new process in new namespaces in one step. unshare() moves the calling process into new namespaces without forking. setns() moves a process into an existing namespace by passing a file descriptor. Container runtimes use combinations of all three when setting up a container.

Cgroups

Control groups organise processes into a hierarchy and apply resource limits to each node in that hierarchy. Every process belongs to exactly one cgroup at each level. Limits set on a parent apply to everything under it. You can give a group of processes a combined memory limit without setting individual limits on each one. The kernel does the accounting for you.

The interface to cgroups is the filesystem. Everything under /sys/fs/cgroup/is a virtual filesystem that the kernel populates. You read resource usage by reading files. You set limits by writing to files. You create a new cgroup by creating a directory. Removing a cgroup means removing the directory after all its processes have moved elsewhere.

Cgroup v1 vs cgroup v2

Cgroup v1 grew organically over the years and it shows. Each subsystem had its own independent hierarchy mounted separately. A process could be in one part of the CPU hierarchy but a completely different part of the memory hierarchy. In practice this created inconsistencies that were difficult to reason about.

Cgroup v2 replaced that with a single unified hierarchy. All subsystems share the same tree. A process is in exactly one location in the hierarchy and all resource limits apply at that location. Most modern distributions default to v2 now. Docker switched its default to v2 in recent versions. If you are reading documentation written before roughly 2021 that talks about cgroups, assume it describes v1 unless it says otherwise.

Working with cgroups

You rarely manipulate cgroups directly in production. Systemd manages them for services. Container runtimes manage them for containers. But knowing how to inspect the hierarchy and read what limits are in place is genuinely useful when you need to understand why a process got killed or why a service is being throttled.

How namespaces and cgroups work together

A container is a process with a full set of namespaces applied plus a cgroup to constrain its resource usage. The namespaces build the walls. The cgroup puts a ceiling on the room. Neither mechanism knows about the other at the kernel level. A process can have namespaces without cgroup limits, or cgroup limits without namespace isolation. Container runtimes use both together because isolation without resource limits leaves you vulnerable to noisy-neighbour problems and resource limits without isolation do not prevent processes from interfering with each other in other ways.

When the memory limit is hit

When a process in a cgroup exceeds its memory limit, the kernel runs the out-of-memory killer against processes inside that group. It picks a victim based on a score that accounts for how much memory the process is using relative to its size. The chosen process is sent SIGKILL. This is why containers sometimes die without warning. From inside the container it looks like the process simply disappeared. From the host you will see an OOM kill event in the kernel log.

The event shows up in dmesg as a line containing oom-kill: followed by the cgroup path, the process name, the PID killed and the amount of memory involved. This is the first place to look when a container exits unexpectedly with no error in its own logs.

oom-kill: constraint=CONSTRAINT_MEMCG, nodemask=(null),
  cpuset=mycontainer, mems_allowed=0,
  oom_memcg=/system.slice/docker-abc123.scope,
  task_memcg=/system.slice/docker-abc123.scope,
  task=node, pid=3141, uid=0