Bug #3071
docker-in-lxc setup results in high lxcfs CPU load
Start date:
03/17/2018
Due date:
% Done:
0%
Spec Reference:
Description
When looking at e.g. build2, I see the following entry in top:
686 root 20 0 2749704 5840 1328 S 370.5 0.0 15262:35 lxcfs
This means 370% CPU used on the lxcfs daemon.
When I strace lxcfs, it's always about the deb8build container, executing build jobs:
[pid 4254] openat(13, "./lxc/deb8build/memory.usage_in_bytes", O_RDONLY) = 17 [pid 4254] openat(13, "./lxc/deb8build/memory.stat", O_RDONLY) = 17 [pid 20990] openat(13, "./lxc/deb8build/memory.limit_in_bytes", O_RDONLY) = 16 [pid 20990] openat(13, "./lxc/memory.limit_in_bytes", O_RDONLY) = 16 [pid 20990] openat(13, ".//memory.limit_in_bytes", O_RDONLY) = 16 [pid 20990] openat(13, "./lxc/deb8build/memory.usage_in_bytes", O_RDONLY) = 16 [pid 20990] openat(13, "./lxc/deb8build/memory.stat", O_RDONLY) = 16 [pid 4140] openat(13, ".//memory.limit_in_bytes", O_RDONLY) = 18 [pid 551] openat(13, "./lxc/deb8build/memory.limit_in_bytes", O_RDONLY) = 20 [pid 551] openat(13, "./lxc/memory.limit_in_bytes", O_RDONLY) = 20 [pid 551] openat(13, ".//memory.limit_in_bytes", O_RDONLY) = 20 [pid 551] openat(13, "./lxc/deb8build/memory.usage_in_bytes", O_RDONLY) = 20 [pid 551] openat(13, "./lxc/deb8build/memory.stat", O_RDONLY) = 20 [pid 20991] openat(13, "./lxc/deb8build/memory.limit_in_bytes", O_RDONLY) = 22 [pid 20991] openat(13, "./lxc/memory.limit_in_bytes", O_RDONLY) = 22 [pid 20991] openat(13, ".//memory.limit_in_bytes", O_RDONLY) = 22 [pid 20991] openat(13, "./lxc/deb8build/memory.usage_in_bytes", O_RDONLY) = 22 [pid 20991] openat(13, "./lxc/deb8build/memory.stat", O_RDONLY) = 22 [pid 3976] openat(13, "./lxc/deb8build/memory.memsw.limit_in_bytes", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 15513] openat(13, "./lxc/deb8build/memory.limit_in_bytes", O_RDONLY) = 19 [pid 15513] openat(13, "./lxc/memory.limit_in_bytes", O_RDONLY) = 19 [pid 15513] openat(13, ".//memory.limit_in_bytes", O_RDONLY) = 19 [pid 15513] openat(13, "./lxc/deb8build/memory.usage_in_bytes", O_RDONLY) = 19 [pid 15513] openat(13, "./lxc/deb8build/memory.stat", O_RDONLY) = 19 [pid 4140] openat(13, "./lxc/deb8build/memory.usage_in_bytes", O_RDONLY) = 18 [pid 23069] openat(13, "./lxc/deb8build/memory.limit_in_bytes", O_RDONLY) = 21 [pid 23069] openat(13, "./lxc/memory.limit_in_bytes", O_RDONLY) = 21 [pid 23069] openat(13, ".//memory.limit_in_bytes", O_RDONLY) = 21 [pid 23069] openat(13, "./lxc/deb8build/memory.usage_in_bytes", O_RDONLY) = 21 [pid 23069] openat(13, "./lxc/deb8build/memory.stat", O_RDONLY) = 21 [pid 4302] openat(13, "./lxc/deb8build/memory.limit_in_bytes", O_RDONLY) = 23 [pid 4302] openat(13, "./lxc/memory.limit_in_bytes", O_RDONLY) = 23 [pid 4302] openat(13, ".//memory.limit_in_bytes", O_RDONLY) = 23 [pid 4302] openat(13, "./lxc/deb8build/memory.usage_in_bytes", O_RDONLY) = 23 [pid 4302] openat(13, "./lxc/deb8build/memory.stat", O_RDONLY) = 23
The current theory is that dockers libcontainer is heavily reading cgroupfs (https://github.com/docker/libcontainer/blob/master/cgroups/fs/memory.go), probably to get memory usage statistics.
As cgroupfs is not provided by the kernel on the host OS, but emulated via FUSE + lxcfs, those frequent read accesses are causing an enormous load on the lxcfs daemon.
Now the question is:- can we live without lxcfs ?
- can we disable docker from reading all those [unused] stats all the time
- do we have to use docker natively on the machine (deb9), rather than inside containers (deb8, deb9, ...)?
History
#1 Updated by laforge almost 3 years ago
- Description updated (diff)