Performance regression in ip_set_swap on 6.1.69

[Ale Crismani]

Dear all,

When upgrading some of our Debian hosts that compose a Kubernetes cluster we found a regression in ip_set_swap on 6.1.69. Calls to ip_set_swap now take roughly 15ms, while they used to take just tens of microseconds before.

The issue is very visible for use, since we use kube-router as our Kubernetes networking interface, and it uses ipset swap all the time to populate sets that enforce firewall policies between containers.

We tracked the issue down with strace, and then took stats with bpftrace running:
---
kfunc:ip_set:ip_set_swap {
        @start[tid] = nsecs;
}

kretfunc:ip_set:ip_set_swap {
        if (@start[tid]) {
                @srlat = hist((nsecs - @start[tid])/1000);
                delete(@start[tid]);
        }
}

interval:s:20 {
        printf("ip_set_swap() latency, milliseconds:\n");
---

On 6.1.69 results look like:
 
ip_set_swap() latency, milliseconds:
[8K, 16K)           1848 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[16K, 32K)         1017 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[32K, 64K)         38     |@

while on 6.1.67:
ip_set_swap() latency, milliseconds:

[0]                  166   |@
[1]                  378   |@@
[2, 4)              762   |@@@@@
[4, 8)              1624 |@@@@@@@@@@@
[8, 16)            3493 |@@@@@@@@@@@@@@@@@@@@@@@@
[16, 32)          7308 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[32, 64)          6412 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[64, 128)              1 |

We tried compiling commits between 6.1.67 and 6.1.69 and it seems the performance regression was introduced by 875ee3a, ip_set_swap is fast on 602505 that precedes it, and slow on it.

First time I post here, hope the format is appropriate, and thanks for any help with this! Also, if possible, I'd appreciate if any reply could CC me, as I am not subscribed.

Alessandro Crismani