[Question] The system may be stuck if there is a cpu cgroup cpu.cfs_quato_us is very low

From: Zhang Qiao
Date: Mon Jun 27 2022 - 02:50:33 EST


Hi all,

I'm working on debuging a problem.
The testcase does follew operations:
1) create a test task cgroup, set cpu.cfs_quota_us=2000,cpu.cfs_period_us=100000.
2) run 20 test_fork[1] test process in the test task cgroup.
3) create 100 new containers:
for i in {1..100}; do docker run -itd --health-cmd="ls" --health-interval=1s ubuntu:latest bash; done

These operations are expected to succeed and 100 containers create success. however, when creating containers,
the system will get stuck and create container failed.

After debug this, I found the test_fork process frequently sleep in freezer_fork()->mutex_lock()->might_sleep()
with taking the cgroup_threadgroup_rw_sem lock, as follow:

copy_process():
cgroup_can_fork() ---> lock cgroup_threadgroup_rw_sem
sched_cgroup_fork();
->task_fork_fair(){
->update_curr(){
->__account_cfs_rq_runtime() {
resched_curr(); ---> the quota is used up, and set flag TIF_NEED_RESCHED to current
}
cgroup_post_fork();
->feezer_fork()
->mutex_lock() {
->might_sleep() ---> schedule() and the current task will be throttled long time.

->cgroup_css_set_put_fork() ---> unlock cgroup_threadgroup_rw_sem


Becuase the task cgroup's cpu.cfs_quota_us is very small and test_fork's load is very heavy, the test_fork
may be throttled long time, therefore, the cgroup_threadgroup_rw_sem read lock is held for a long time, other
processes will get stuck waiting for the lock:

1) a task fork child, will wait at copy_process()->cgroup_can_fork();

2) a task exiting will wait at exit_signals();

3) a task write cgroup.procs file will wait at cgroup_file_write()->__cgroup1_procs_write();
...

even the whole system will get stuck.

Anyone know how to slove this? Except for changing the cpu.cfs_quota_us.


[1] test_fork.c

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>

int main(int argc, char **argv)
{
pid_t pid;
int count = 20;

while(1) {
for (int i = 0; i < count; i++) {
if ((pid = fork()) <0) {
printf("fork error");
return 1;
} else if (pid ==0) {
exit(0);
}
}

for (int i = 0; i < count; i++) {
wait(NULL);
}
sleep(1);
}
return 0;
}

Thanks a lot.
-Qiao
-