Pantavisor 031 adds support for cgroup v2, the newer way Linux limits what a container may use. Most people do not need to do anything. If your containers do not set any cgroup options, they keep working and run on whichever version your device uses. If they do set device lists or limits, the section “Migrating your containers” below shows the few lines to add.
These changes ship with Pantavisor 031 (the next release, not yet out) and pvr 055 (released). meta-pantavisor changes ship with its next release. This post goes together with LXC 7.0 arrives, LXC 6 becomes the default, LXC 3 is deprecated, which explains why cgroup v2 matters now.
What are cgroups?
Cgroups (“control groups”) are a Linux kernel feature. They let the system limit two kinds of things for a container:
- Limits: how much memory, CPU time or how many processes the container may use.
- Device rules: which devices the container may open, for example a serial port, GPIO pins or a disk.
The kernel offers two interfaces for this. cgroup v1 is the older one and is deprecated. cgroup v2 is the newer one, and current Linux distributions use it.
Today, Pantavisor on a device sets up a mixed (“hybrid”) layout: all controls live in v1, with an empty v2 tree next to them.
The new setting
pantavisor.config gets a new setting, PV_SYSTEM_CGROUP_VERSION:
PV_SYSTEM_CGROUP_VERSION=auto
| Value | Meaning |
|---|---|
1 |
The hybrid layout used today. |
2 |
A pure cgroup v2 layout. |
auto |
v2 if the kernel can do it, otherwise v1. |
default |
Same as not setting it: 1 on devices, auto in appengine (Pantavisor running inside a container on a PC or server, for development and testing). |
If you set 1 or 2 and the system cannot provide it, Pantavisor stops with a clear error. This is deliberate: the alternative would be to run containers without the protection they ask for. On v1, Pantavisor logs a deprecation warning.
Kernel requirements for v2
On v2 the kernel restricts device access with a small BPF program. Without it, device rules would silently not apply. So v2 needs these kernel options:
CONFIG_BPF_SYSCALL=y
CONFIG_CGROUP_BPF=y
With auto, Pantavisor checks this at boot. As an example from testing: a Banana Pi M2 Berry with its stock kernel (no CONFIG_CGROUP_BPF) stays on v1 with auto. With those options enabled, it runs v2.
Defaults in meta-pantavisor
The default depends on which LXC version the build uses (see the LXC post):
| LXC feature | LXC version | PV_SYSTEM_CGROUP_VERSION |
|---|---|---|
lxc-stable (default) |
LXC 6 | auto |
lxc-next |
LXC 7 | 2 |
lxc-deprecated |
LXC 3 | 1 |
The layer adds no kernel options. A build that asks for 2 stops at build time with a clear message if the kernel lacks the two options above. To override the default in your build:
PANTAVISOR_CGROUP_VERSION = "1"
Use PANTAVISOR_CGROUP_VERSION:<machine> = "1" to change it for one machine only.
How containers state their cgroup rules
A container lists its cgroup lines in its LXC config. Lines starting with lxc.cgroup. are for v1, and lines starting with lxc.cgroup2. are for v2.
Before starting a container, Pantavisor removes the lines for the version it is not running. This is the same for every LXC version.
- Safety rule: if lines were removed and none are left for the running version, the container does not start, and the log names the removed lines. If an update brings such a container, the update fails and the device goes back to the previous revision.
- One exception:
lxc.cgroup.devices.allow = aorlxc.cgroup2.devices.allow = a(“allow all devices”) is not needed, because a container with no device lines gets every device anyway. Removing it does not trigger the safety rule; a warning says the line can be dropped. - Your files stay as they are. The originals in the revision are never changed. The config LXC actually reads is written to
/run/pantavisor/lxc/<container>.conf(settingPV_SYSTEM_LXCDIR), a directory no container can see. To see what was removed:
diff /storage/trails/current/<container>/lxc.container.conf /run/pantavisor/lxc/<container>.conf
Migrating your containers
pvr 055 knows the v2 template arguments. They go in the "args" section of a container’s src.json (see PVR_TEMPLATES.md):
| Device list | Limits | |
|---|---|---|
| v1 | PVR_LXC_CGROUP_DEVICES_WHITE |
PVR_LXC_CGROUP_V1 |
| v2 (new) | PVR_LXC_CGROUP2_DEVICES_WHITE |
PVR_LXC_CGROUP_V2 |
A device list writes deny = a and then one allow = <rule> line per entry, for each version. pvr 055 no longer writes lxc.cgroup.devices.allow = a into every container. Containers built by meta-pantavisor set no cgroup args, so they have no cgroup lines and work on both versions.
Which case are you in?
- No cgroup args: nothing to do. Old revisions that still carry the old
allow = aline keep working, with a warning. - v1 device lists or limits: add the v2 args next to the v1 ones. Use the same device list, and the v2 names for limits. For example, v1
memory.limit_in_bytesbecomes v2memory.max, whilepids.maxkeeps its name. Keep the v1 args for as long as devices running v1 must run the container.
Example
"args": {
"PVR_LXC_CGROUP_DEVICES_WHITE": ["c 1:3 rwm", "c 1:5 rwm"],
"PVR_LXC_CGROUP2_DEVICES_WHITE": ["c 1:3 rwm", "c 1:5 rwm"],
"PVR_LXC_CGROUP_V1": ["pids.max = 100"],
"PVR_LXC_CGROUP_V2": ["pids.max = 100"]
}
A device rule such as c 1:3 rwm reads: character device 1:3 (that is /dev/null), allowed to read, write and create the device node. This produces these lines in the container’s LXC config:
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.pids.max = 100
lxc.cgroup2.devices.deny = a
lxc.cgroup2.devices.allow = c 1:3 rwm
lxc.cgroup2.devices.allow = c 1:5 rwm
lxc.cgroup2.pids.max = 100
Running the new containers on older Pantavisor
Pantavisor before 031 does not remove any lines. What happens there:
| Container has | On Pantavisor before 031 |
|---|---|
| no cgroup lines | starts as before |
| v1 and v2 device lists | starts; the v1 rules apply |
v2 limit lines (for example memory.max) |
does not start on v1/hybrid devices, because the v2 limit files do not exist there |
So v2 device lists are safe to add today. v2 limits need Pantavisor 031. A revision that brings the new Pantavisor (the BSP) together with v2 limits works.
Checking what a device runs
New in 031: two device metadata keys tell you the layout and version in use.
pantavisor.cgroup.layout:hybrid,legacyorunifiedpantavisor.cgroup.version:1or2
On the device:
pvcontrol devmeta ls
They also show in the device’s metadata on Pantacor Hub. The Pantavisor log has one line with the decision:
cgroup layout hybrid, using cgroup version 1 (PV_SYSTEM_CGROUP_VERSION=auto)
On v2, you can read a container’s limit from inside it:
cat /sys/fs/cgroup/pids.max
The v2 layout
On v2, Pantavisor and its own processes live in init.scope, each container in lxc/<name>, and LXC’s helper process for each container in lxc.monitor.<name>. The layout is the same on devices and in appengine.
Questions and results from your own boards are welcome in this thread.