Memory kills, CPU throttles, pids refuses

A cgroup is an accounting boundary

Not a sandbox. A cgroup counts what the processes in it are using and refuses to let them go past a limit, and the way it refuses is different for every limit.

memory.max kills. When the total goes over, the OOM killer picks a process in the cgroup and takes it — the process gets no warning and no chance to clean up, because it is killed with a signal it cannot catch.

cpu.max throttles. Nothing dies; everything gets slower. A CPU-limited container looks exactly like a slow disk, and nobody thinks to look at the cgroup.

pids.max refuses. fork returns EAGAIN, and most programs do not check fork's return value at all.

Three failures that look nothing alike, and one of them is silent unless the program checks a return value.

The quiet one

pids.max is the limit nobody notices, because fork failing is something most programs do not check for. A supervisor that forks a worker per job and never tests the return value will silently stop starting work, with no error anywhere and every process apparently healthy.

#include <unistd.h>
#include <sys/wait.h>

int main(void) {
  for (int i = 0; i < 6; i = i + 1) {
    pid_t pid = fork();
    if (pid == 0) {
      _exit(0);
    }
    if (pid < 0) {
      printf("fork failed at %d\n", i);
      return 1;
    }
    int status;
    waitpid(pid, &status, 0);
  }
  return 0;
}

The limit applies to everything underneath

A limit is on the subtree, not on the one directory. Splitting a workload into child cgroups does not give it more memory, and a child with no limit of its own is still bounded by its parent's.

Writing a larger number into a child changes nothing and reads back as the number you wrote — which is why cat memory.max inside a container is not the answer to "how much memory do I have". The answer is the tightest limit on the path to the root, and nothing shows it to you.

Before you run it

A cgroup holds a supervisor that allocated almost nothing and a worker that allocated nearly all of the limit. The worker allocates a little more and the cgroup goes over.