In the ever-evolving world of container orchestration with Kubernetes, changes are a constant. One of the most significant shifts is the transition from cgroup v1 to cgroup v2. If you’re managing Kubernetes clusters, this is not just a technical detail—it's a fundamental change that impacts resource management, security, and overall performance. Let’s dive into what this means for you.

What are cgroups?

At its core, a cgroup (control group) is a Linux kernel feature that allows for the allocation, prioritization, and monitoring of system resources like CPU, memory, disk I/O, and network bandwidth. In Kubernetes, cgroups are critical for managing the resources of containers running in pods. They ensure that no single container can hog all the resources and disrupt the entire system.

Think of it this way: if your applications are like cars on a highway (your computer’s resources), then cgroups are the traffic management system. They set speed limits, designate lanes, and prevent one application from taking over all the resources and causing a system-wide traffic jam. Cgroups are the foundation for fair resource distribution in your cluster.

Here’s how Kubernetes uses cgroups:

• Resource Allocation: Ensuring containers don’t exceed their allocated CPU and memory limits.
• Isolation: Isolating containers from each other to prevent resource contention.
• Monitoring: Tracking resource usage for each container to provide insights and metrics.

cgroup v1 vs. cgroup v2: A Major Upgrade

Just like traffic management systems have evolved, so have cgroups. Cgroup v1, the older version, is like an old filing system—a bit clunky and not as organized as it could be. Cgroup v2 is a sleek, modern upgrade, streamlining everything for greater efficiency and control.

cgroup v1: The Old Guard

Cgroup v1 has been the workhorse for resource management in Linux for years. It allowed for the management of various resources, but it had some significant limitations. One of the main issues was its complex and inconsistent structure, with different controllers for different resource types. This lack of a unified structure made management more complicated than it needed to be.

cgroup v2: The Modern Solution

Enter cgroup v2, the next generation of the cgroup API. This is not just a minor update but a complete overhaul designed to address the shortcomings of v1. Cgroup v2 provides a unified control system with enhanced resource management capabilities. It’s designed to be more efficient, secure, and easier to manage.

Here’s why cgroup v2 is a significant improvement:

• Unified Hierarchy: Unlike v1, which had multiple hierarchies, v2 uses a single unified hierarchy. This simplifies resource management and makes it more consistent.
• Improved Resource Control: Cgroup v2 offers more granular control over resource allocation. It also provides unified accounting for different types of memory allocations. It’s like having a dimmer switch instead of an on/off light switch, allowing you to fine-tune how much CPU, memory, or network bandwidth each application receives.
• Enhanced Security: Cgroup v2 provides safer sub-tree delegation to containers. It also has features such as rootless support, which improves the security of containerized applications.
• New Features: Cgroup v2 introduces new features like Pressure Stall Information (PSI).
• Better Performance: The unified structure and better control mechanisms contribute to better performance of workloads.

How to Migrate from cgroup v1 to v2

To migrate to cgroup v2, here's what you need to do:

1. Check System Requirements:
   • Your OS must have cgroup v2 enabled.
   • Many newer distributions use cgroup v2 by default.
   • Your Linux kernel version is recommended to be 5.8+.
   • Your container runtime (like containerd or cri-o) must support cgroup v2.
   • Kubelet and the container runtime should be configured to use the systemd cgroup driver.
2. Upgrade your kernel: Upgrade to a kernel version that has cgroup v2 enabled by default.
3. Verify the cgroup version: Use the command stat -fc %T /sys/fs/cgroup/ to check if your distribution uses cgroup v2. The output cgroup2fs indicates cgroup v2, while tmpfs indicates cgroup v1.
4. Update applications:
   • Update any tools that directly interact with the cgroup filesystem.
   • Third-party monitoring and security agents need to be updated to versions that support cgroup v2.
   • If you use cAdvisor, update it to v0.43.0 or later.
   • For Java applications, use versions with full cgroup v2 support.
   • If you use uber-go/automaxprocs, ensure it's version v1.5.1 or higher.
5. Memory usage: Be aware that cgroup v2 may report memory usage differently than v1, potentially showing higher usage, even if actual consumption is the same.

After completing these steps, kubelet will automatically detect cgroup v2 and adjust accordingly, with no additional configuration needed. Switching to cgroup v2 shouldn't noticeably change user experience unless you directly access the cgroup filesystem.

The Kubernetes Transition

Kubernetes is fully embracing cgroup v2 for enhanced resource management and isolation. The community has decided to move cgroup v1 support into maintenance mode in v1.31. This means:

• Feature Freeze: No new features will be added to cgroup v1.
• Security Fixes: Critical security fixes will still be provided.
• Best-Effort Bug Fixes: Major bugs may be fixed if feasible.

This transition is driven by the need to align with the broader Linux ecosystem and encourage the adoption of cgroup v2. It’s important to note that maintenance mode is not deprecation; cgroup v1 will continue to receive necessary fixes.

What This Means For You

If you’re using Kubernetes on Linux nodes, here’s what you need to know about this transition:

1. Check Your cgroup Version: Many newer Linux distributions, like Ubuntu 21.10+ and Debian 11+, use cgroup v2 by default. As we mentioned above, you can check your system’s cgroup version by running the command:
   stat -fc %T /sys/fs/cgroup/

The output will be cgroup2fs for cgroup v2, and tmpfs for cgroup v1.

2. Ensure Compatibility: Make sure your applications and tools are compatible with cgroup v2.
   
   • Third-party tools: Update monitoring and security agents to versions that support cgroup v2. If you run cAdvisor, update it to v0.43.0 or later.
   • Java applications: Use versions that fully support cgroup v2. Some recommended versions include OpenJDK / HotSpot (jdk8u372, 11.0.16, 15 and later), IBM Semeru Runtimes (8.0.382.0, 11.0.20.0, 17.0.8.0, and later), and IBM Java (8.0.8.6 and later).
   • uber-go/automaxprocs package: Make sure the version you use is v1.5.1 or higher.

3. Ensure no application is using cgroup filesystem APIs from v1: If you have custom applications, ensure they’re not using outdated cgroup v1 tricks. Upgrading to the cgroup v2 will break those applications.