Re: [PATCH V3 net-next] net: fec: add XDP_TX feature support

From: Jesper Dangaard Brouer
Date: Fri Aug 04 2023 - 08:36:45 EST

On 04/08/2023 05.06, Wei Fang wrote:
The FEC of i.MX8MP-EVK has dma_coherent=false, and as I mentioned
above, I did not see an obvious difference in the performance. :(

That is surprising - given the results.

(see below, lack of perf/diff might be caused by Ethernet flow-control).

The result of the current modification.
root@imx8mpevk:~# ./xdp2 eth0
proto 17: 260180 pkt/s

These results are*significantly* better than reported in patch-1.
What happened?!?

The test environment is slightly different, in patch-1, the FEC port
was directly connected to the port of another board. But in the latest
test, the ports of the two boards were connected to a switch, so the
ports of the two boards are not directly connected.

Hmm, I've seen this kind of perf behavior of direct-connected or via switch
before. The mistake I made was, that I had not disabled Ethernet flow-control.
The xdp2 XDP_TX program will swap the mac addresses, and send the packet
back to the packet generator (running pktgen), which will get overloaded
itself and starts sending Ethernet flow-control pause frames.

Command line to disable:
# ethtool -A eth0 rx off tx off

Can I ask/get you to make sure that Ethernet flow-control is disabled (on
both generator and DUT (to be on safe-side)) and run the test again?

The flow-control was not disabled before, so according to your suggestion,
I disable the flow-control on the both boards and run the test again, the
performance is slightly improved, but still can not see a clear difference
between the two methods. Below are the results.

Something else must be stalling the CPU.
When looking at fec_main.c code, I noticed that fec_enet_txq_xmit_frame() will do a MMIO write for every xdp_frame (to trigger transmit start), which I believe will stall the CPU.
The ndo_xdp_xmit/fec_enet_xdp_xmit does bulking, and should be the function that does the MMIO write to trigger transmit start.

$ git diff
diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c
index 03ac7690b5c4..57a6a3899b80 100644
--- a/drivers/net/ethernet/freescale/fec_main.c
+++ b/drivers/net/ethernet/freescale/fec_main.c
@@ -3849,9 +3849,6 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep,

txq->bd.cur = bdp;

- /* Trigger transmission start */
- writel(0, txq->bd.reg_desc_active);
return 0;

@@ -3880,6 +3877,9 @@ static int fec_enet_xdp_xmit(struct net_device *dev,

+ /* Trigger transmission start */
+ writel(0, txq->bd.reg_desc_active);

return sent_frames;

Result: use "sync_dma_len" method
root@imx8mpevk:~# ./xdp2 eth0

The xdp2 (and xdp1) program(s) have a performance issue
(due to using

Can I ask you to test using xdp_rxq_info, like:

sudo ./xdp_rxq_info --dev mlx5p1 --action XDP_TX

proto 17: 258886 pkt/s
proto 17: 258879 pkt/s

If you provide numbers for xdp_redirect, then we could better evaluate
if changing the lock per xdp_frame, for XDP_TX also, is worth it.

And also find out of moving the MMIO write have any effect.

I also noticed driver does a MMIO write (on rxq) for every RX-packet in
fec_enet_rx_queue() napi-poll loop. This also looks like a potential
performance stall.