RE: [PATCH net 3/3] net: fec: increase the size of tx ring and update thresholds of tx ring

From: Wei Fang
Date: Wed Jul 05 2023 - 02:20:40 EST

> -----Original Message-----
> From: Andrew Lunn <andrew@xxxxxxx>
> Sent: 2023年7月5日 8:25
> To: Wei Fang <wei.fang@xxxxxxx>
> Cc: davem@xxxxxxxxxxxxx; edumazet@xxxxxxxxxx; kuba@xxxxxxxxxx;
> pabeni@xxxxxxxxxx; ast@xxxxxxxxxx; daniel@xxxxxxxxxxxxx;
> hawk@xxxxxxxxxx; john.fastabend@xxxxxxxxx; Shenwei Wang
> <shenwei.wang@xxxxxxx>; Clark Wang <xiaoning.wang@xxxxxxx>;
> netdev@xxxxxxxxxxxxxxx; dl-linux-imx <linux-imx@xxxxxxx>;
> linux-kernel@xxxxxxxxxxxxxxx; bpf@xxxxxxxxxxxxxxx
> Subject: Re: [PATCH net 3/3] net: fec: increase the size of tx ring and update
> thresholds of tx ring
>
> > After trying various methods, I think that increase the size of tx BD
> > ring is simple and effective. Maybe the best resolution is that
> > allocate NAPI for each queue to improve the efficiency of the NAPI
> > callback, but this change is a bit big and I didn't try this method.
> > Perheps this method will be implemented in a future patch.
>
> How does this affect platforms like Vybrid with its fast Ethernet?
Sorry, I don't have the Vybrid platform, but I think I don't think it has much
impact, at most it just takes up some more memory.

> Does the burst latency go up?
No, for fec, when a packet is attached to the BDs, the software will immediately
trigger the hardware to send the packet. In addition, I think it may improve the
latency, because the size of the tx ring becomes larger, and more packets can be
attached to the BD ring for burst traffic.

>
> > In addtion, this patch also updates the tx_stop_threshold and the
> > tx_wake_threshold of the tx ring. In previous logic, the value of
> > tx_stop_threshold is 217, however, the value of tx_wake_threshold is
> > 147, it does not make sense that tx_wake_threshold is less than
> > tx_stop_threshold.
>
> What do these actually mean? I could imagine that as the ring fills you don't
> want to stop until it is 217/512 full. There is then some hysteresis, such that it
> has to drop below 147/512 before more can be added?
>
You must have misunderstood, let me explain more clearly, the queue will be
stopped when the available BDs are less than tx_stop_threshold (217 BDs). And
the queue will be waked when the available BDs are greater than tx_wake_threshold
(147 BDs). So in most cases, the available BDs are greater than tx_wake_threshold
when the queue is stopped, the only effect is to delay packet sending.
In my opinion, tx_wake_threshold should be greater than tx_stop_threshold, we
should stop queue when the available BDs are not enough for a skb to be attached.
And wake the queue when the available BDs are sufficient for a skb.

> > Besides, both XDP path and 'slow path' share the tx BD rings. So if
> > tx_stop_threshold is 217, in the case of heavy XDP traffic, the slow
> > path is easily to be stopped, this will have a serious impact on the
> > slow path.
>
> Please post your iperf results for various platforms, so we can see the effects of
> this. We generally don't accept tuning patches without benchmarks which
> prove the improvements, and also show there is no regression. And given the
> wide variety of SoCs using the FEC, i expect testing on a number of SoCs, but
> Fast and 1G.
>
Below are the results on i.MX6UL/8MM/8MP/8ULP/93 platforms, i.MX6UL and
8ULP only support Fast ethernet. Others support 1G.
1.1 i.MX6UL with tx ring size 512
root@imx6ul7d:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 47148 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.1 MBytes 84.9 Mbits/sec 0 103 KBytes
[ 5] 1.00-2.00 sec 9.88 MBytes 82.6 Mbits/sec 0 103 KBytes
[ 5] 2.00-3.00 sec 9.82 MBytes 82.7 Mbits/sec 0 103 KBytes
[ 5] 3.00-4.00 sec 9.82 MBytes 82.4 Mbits/sec 0 103 KBytes
[ 5] 4.00-5.00 sec 9.88 MBytes 82.9 Mbits/sec 0 103 KBytes
[ 5] 5.00-6.00 sec 9.94 MBytes 83.4 Mbits/sec 0 103 KBytes
[ 5] 6.00-7.00 sec 10.1 MBytes 84.3 Mbits/sec 0 103 KBytes
[ 5] 7.00-8.00 sec 9.82 MBytes 82.4 Mbits/sec 0 103 KBytes
[ 5] 8.00-9.00 sec 9.82 MBytes 82.4 Mbits/sec 0 103 KBytes
[ 5] 9.00-10.00 sec 9.88 MBytes 82.9 Mbits/sec 0 103 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 99.0 MBytes 83.1 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 98.8 MBytes 82.8 Mbits/sec receiver

1.2 i.MX6UL with tx ring size 1024
root@imx6ul7d:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 55670 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.2 MBytes 85.4 Mbits/sec 0 236 KBytes
[ 5] 1.00-2.00 sec 10.1 MBytes 84.6 Mbits/sec 0 236 KBytes
[ 5] 2.00-3.00 sec 10.2 MBytes 85.5 Mbits/sec 0 249 KBytes
[ 5] 3.00-4.00 sec 10.1 MBytes 85.1 Mbits/sec 0 249 KBytes
[ 5] 4.00-5.00 sec 10.1 MBytes 84.7 Mbits/sec 0 249 KBytes
[ 5] 5.00-6.00 sec 10.0 MBytes 84.1 Mbits/sec 0 249 KBytes
[ 5] 6.00-7.00 sec 10.1 MBytes 85.1 Mbits/sec 0 249 KBytes
[ 5] 7.00-8.00 sec 10.1 MBytes 84.9 Mbits/sec 0 249 KBytes
[ 5] 8.00-9.00 sec 10.3 MBytes 85.9 Mbits/sec 0 249 KBytes
[ 5] 9.00-10.00 sec 10.2 MBytes 85.6 Mbits/sec 0 249 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 101 MBytes 85.1 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 101 MBytes 84.5 Mbits/sec receiver

2.1 i.MX8ULP with tx ring size 512
root@imx8ulpevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 54342 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.8 MBytes 90.3 Mbits/sec 0 99.0 KBytes
[ 5] 1.00-2.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes
[ 5] 2.00-3.00 sec 10.2 MBytes 85.5 Mbits/sec 0 99.0 KBytes
[ 5] 3.00-4.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes
[ 5] 4.00-5.00 sec 10.2 MBytes 85.5 Mbits/sec 0 99.0 KBytes
[ 5] 5.00-6.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes
[ 5] 6.00-7.00 sec 9.69 MBytes 81.3 Mbits/sec 0 99.0 KBytes
[ 5] 7.00-8.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes
[ 5] 8.00-9.00 sec 9.69 MBytes 81.3 Mbits/sec 0 99.0 KBytes
[ 5] 9.00-10.00 sec 10.2 MBytes 85.5 Mbits/sec 0 99.0 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 100 MBytes 84.3 Mbits/sec 0 sender
[ 5] 0.00-9.90 sec 100 MBytes 84.7 Mbits/sec receiver

2.1 i.MX8ULP with tx ring size 1024
root@imx8ulpevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 44770 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.7 MBytes 90.1 Mbits/sec 0 93.3 KBytes
[ 5] 1.00-2.00 sec 9.94 MBytes 83.4 Mbits/sec 0 93.3 KBytes
[ 5] 2.00-3.00 sec 10.2 MBytes 85.5 Mbits/sec 0 93.3 KBytes
[ 5] 3.00-4.00 sec 10.1 MBytes 85.0 Mbits/sec 0 93.3 KBytes
[ 5] 4.00-5.00 sec 9.94 MBytes 83.4 Mbits/sec 0 100 KBytes
[ 5] 5.00-6.00 sec 10.2 MBytes 85.5 Mbits/sec 0 100 KBytes
[ 5] 6.00-7.00 sec 9.69 MBytes 81.3 Mbits/sec 0 100 KBytes
[ 5] 7.00-8.00 sec 9.94 MBytes 83.4 Mbits/sec 0 100 KBytes
[ 5] 8.00-9.00 sec 10.2 MBytes 85.5 Mbits/sec 0 100 KBytes
[ 5] 9.00-10.00 sec 9.69 MBytes 81.3 Mbits/sec 0 100 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 101 MBytes 84.4 Mbits/sec 0 sender
[ 5] 0.00-9.92 sec 100 MBytes 84.8 Mbits/sec receiver

3.1 i.MX8MM with tx ring size 512
root@imx8mmevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 55734 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 111 MBytes 934 Mbits/sec 0 577 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec 0 577 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 0 609 KBytes
[ 5] 3.00-4.00 sec 113 MBytes 945 Mbits/sec 0 638 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 638 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 0 638 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 638 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 943 Mbits/sec 0 638 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 943 Mbits/sec 0 638 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 0 638 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec receiver

3.2 i.MX8MM with tx ring size 1024
root@imx8mmevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 53350 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 952 Mbits/sec 0 585 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 585 KBytes
[ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec 0 585 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 0 648 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 648 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver

4.1 i.MX8MP with tx ring size 512
root@imx8mpevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 51892 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 594 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec 0 626 KBytes
[ 5] 2.00-3.00 sec 113 MBytes 946 Mbits/sec 0 626 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 937 Mbits/sec 0 626 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 0 626 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 940 Mbits/sec 0 626 KBytes
[ 5] 6.00-7.00 sec 113 MBytes 946 Mbits/sec 0 626 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 939 Mbits/sec 0 626 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 935 Mbits/sec 0 626 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 943 Mbits/sec 0 626 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver

4.2 i.MX8MP with tx ring size 1024
root@imx8mpevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 37922 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 951 Mbits/sec 0 608 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec 0 608 KBytes
[ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec 0 608 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 934 Mbits/sec 0 608 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 942 Mbits/sec 0 608 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 939 Mbits/sec 0 608 KBytes
[ 5] 6.00-7.00 sec 113 MBytes 949 Mbits/sec 0 608 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 608 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 936 Mbits/sec 0 608 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 0 608 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver

5.1 i.MX93 with tx ring size 512
root@imx93evk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 44216 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 115 MBytes 965 Mbits/sec 0 656 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 934 Mbits/sec 0 656 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 656 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver

5.2 i.MX93 with tx ring size 1024
root@imx93evk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 51058 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 588 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 935 Mbits/sec 0 649 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 649 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 940 Mbits/sec receiver