Re: [WITHDRAW PATCH v1 1/1] net: r8169: fix the pci setup so the Realtek RTL8111/8168/8411 ethernet speeds up

From: Heiner Kallweit
Date: Sun Apr 30 2023 - 18:49:15 EST

On 30.04.2023 01:09, Mirsad Goran Todorovac wrote:
> On 25. 04. 2023. 13:44, Mirsad Goran Todorovac wrote:
>> It was noticed that Ookla Speedtest had shown only 250 Mbps download and
>> 310 Mbps upload where Windows 10 on the same box showed 440/310 Mbps, which
>> is the link capacity.
>>
>> This article: https://www.phoronix.com/news/Intel-i219-LM-Linux-60p-Fix
>> inspired to check our speeds. (Previously I used to think it was a network
>> congestion, or reduction on our ISP, but now each time Windows 10 downlink
>> speed is 440 compared to 250 Mbps in Linuxes Linux is performing at 60% of
>> the speed.)
>>
>> The latest 6.3 kernel shows 95% speed up with this patch as compared to the
>> same commit without it:
>>
>> ::::::::::::::
>> speedtest/6.3.0-00436-g173ea743bf7a-dirty-1
>> ::::::::::::::
>> [marvin@pc-mtodorov ~]$ speedtest -s 41437
>>
>> Speedtest by Ookla
>>
>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>> ISP: Croatian Academic and Research Network
>> Idle Latency: 1.53 ms (jitter: 0.15ms, low: 1.30ms, high: 1.71ms)
>> Download: 225.13 Mbps (data used: 199.3 MB)
>> 1.65 ms (jitter: 20.15ms, low: 0.81ms, high: 418.27ms)
>> Upload: 350.00 Mbps (data used: 157.9 MB)
>> 3.35 ms (jitter: 19.46ms, low: 1.61ms, high: 474.55ms)
>> Packet Loss: 0.0%
>> Result URL: https://www.speedtest.net/result/c/a0084fd8-c275-4019-899a-a1590e49a34b
>> [marvin@pc-mtodorov ~]$ speedtest -s 41437
>>
>> Speedtest by Ookla
>>
>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>> ISP: Croatian Academic and Research Network
>> Idle Latency: 1.54 ms (jitter: 0.28ms, low: 1.17ms, high: 1.64ms)
>> Download: 222.88 Mbps (data used: 207.9 MB)
>> 10.23 ms (jitter: 31.76ms, low: 0.75ms, high: 353.79ms)
>> Upload: 349.91 Mbps (data used: 157.7 MB)
>> 3.27 ms (jitter: 13.05ms, low: 1.67ms, high: 236.76ms)
>> Packet Loss: 0.0%
>> Result URL: https://www.speedtest.net/result/c/f4c663ba-830d-44c6-8033-ce3b3b818c42
>> [marvin@pc-mtodorov ~]$
>> ::::::::::::::
>> speedtest/6.3.0-r8169-00437-g323fe5352af6-dirty-2
>> ::::::::::::::
>> [marvin@pc-mtodorov ~]$ speedtest -s 41437
>>
>> Speedtest by Ookla
>>
>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>> ISP: Croatian Academic and Research Network
>> Idle Latency: 0.84 ms (jitter: 0.05ms, low: 0.82ms, high: 0.93ms)
>> Download: 432.37 Mbps (data used: 360.5 MB)
>> 142.43 ms (jitter: 76.45ms, low: 1.02ms, high: 1105.19ms)
>> Upload: 346.29 Mbps (data used: 164.6 MB)
>> 7.72 ms (jitter: 29.80ms, low: 0.92ms, high: 283.48ms)
>> Packet Loss: 12.8%
>> Result URL: https://www.speedtest.net/result/c/e473359e-c37e-4f29-aa9f-4b008210cf7c
>> [marvin@pc-mtodorov ~]$ speedtest -s 41437
>>
>> Speedtest by Ookla
>>
>> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>> ISP: Croatian Academic and Research Network
>> Idle Latency: 0.82 ms (jitter: 0.16ms, low: 0.75ms, high: 1.05ms)
>> Download: 440.97 Mbps (data used: 427.5 MB)
>> 72.50 ms (jitter: 52.89ms, low: 0.91ms, high: 865.08ms)
>> Upload: 342.75 Mbps (data used: 166.6 MB)
>> 3.26 ms (jitter: 22.93ms, low: 1.07ms, high: 239.41ms)
>> Packet Loss: 13.4%
>> Result URL: https://www.speedtest.net/result/c/f393e149-38d4-4a34-acc4-5cf81ff13708
>>
>> 440 Mbps is the speed achieved in Windows 10, and Linux 6.3 with
>> the patch, while 225 Mbps without this patch is running at 51% of
>> the nominal speed with the same hardware and Linux kernel commit.
>>
>> Cc: David S. Miller <davem@xxxxxxxxxxxxx>
>> Cc: Eric Dumazet <edumazet@xxxxxxxxxx>
>> Cc: Jakub Kicinski <kuba@xxxxxxxxxx>
>> Cc: Paolo Abeni <pabeni@xxxxxxxxxx>
>> Cc: nic_swsd@xxxxxxxxxxx
>> Cc: netdev@xxxxxxxxxxxxxxx
>> Link: https://bugzilla.redhat.com/show_bug.cgi?id=1671958#c60
>> Suggested-by: Heiner Kallweit <hkallweit1@xxxxxxxxx>
>> Signed-off-by: Mirsad Goran Todorovac <mirsad.todorovac@xxxxxxxxxxxx>
>> ---
>> drivers/net/ethernet/realtek/r8169_main.c | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
>> index 45147a1016be..b8a04301d130 100644
>> --- a/drivers/net/ethernet/realtek/r8169_main.c
>> +++ b/drivers/net/ethernet/realtek/r8169_main.c
>> @@ -3239,6 +3239,7 @@ static void rtl_hw_start_8168h_1(struct rtl8169_private *tp)
>> r8168_mac_ocp_write(tp, 0xc094, 0x0000);
>> r8168_mac_ocp_write(tp, 0xc09e, 0x0000);
>>
>> + pci_disable_link_state(tp->pci_dev, PCIE_LINK_STATE_CLKPM);
>> rtl_hw_aspm_clkreq_enable(tp, true);
>> }
>>
>
> After some additional research, I came to the obvious realisation, reading more
> thoroughly the discussion at the link - that the above patch did not work for
> all Realtek RTL819x cards back then.
>
> My version, the RTL8168h/8111h indeed works 95% faster on the 6.3 Linux kernel,
> but I cannot speak for the people with the power management problems and
> battery life issues ... and the concerns explained here: https://github.com/KastB/r8169
>
> [root@pc-mtodorov marvin]# dmesg | grep RTL
> [ 7.304130] r8169 0000:01:00.0 eth0: RTL8168h/8111h, f4:93:9f:f0:a5:f5, XID 541, IRQ 123
>
> Currently there seem to be at least 43 revisions of the RTL816x cards and firmware,
> and I really cannot test on all of them.
>
> I will test the other Heiner's experimental patch, but it seems to disable ASPM completely,
> while for my Lenovo desktop with RTL8168h/8111h disabling CLKPM alone.
>
> However, further homework revealed that the kernel patch is unnecessary, as the same
> effect can be achieved in runtime by the sysfs parm introduced with THIS PATCH:
> https://patchwork.kernel.org/project/linux-pci/patch/b1c83f8a-9bf6-eac5-82d0-cf5b90128fbf@xxxxxxxxx/
> which was solved 3 1/2 years ago, but the default on my AlmaLinux 8.7 and Lenovo desktop
> box 10TX000VCR was the 53% of the link capacity and speed.
>
> (I don't know if the card would operate with 220 Mbps on a Gigabit link, it was
> not tested.)
>
> [marvin@pc-mtodorov ~]$ speedtest -s 41437
>
> Speedtest by Ookla
>
> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
> ISP: Croatian Academic and Research Network
> Idle Latency: 1.44 ms (jitter: 0.23ms, low: 1.20ms, high: 1.65ms)
> Download: 220.62 Mbps (data used: 214.2 MB)
> 22.01 ms (jitter: 36.04ms, low: 0.84ms, high: 817.47ms)
> Upload: 346.86 Mbps (data used: 169.1 MB)
> 3.32 ms (jitter: 12.12ms, low: 0.87ms, high: 221.69ms)
> Packet Loss: 0.6%
> Result URL: https://www.speedtest.net/result/c/20c546e7-0b8f-4a2e-a669-a597bb5aee36
> [marvin@pc-mtodorov ~]$ sudo bash
> [sudo] password for marvin:
> [root@pc-mtodorov marvin]# cat /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/link/clkpm
> 1
> [root@pc-mtodorov marvin]# echo 0 > /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/link/clkpm
> [root@pc-mtodorov marvin]# cat /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/link/clkpm
> 0
> [root@pc-mtodorov marvin]# speedtest -s 41437
>
> Speedtest by Ookla
>
> Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
> ISP: Croatian Academic and Research Network
> Idle Latency: 0.85 ms (jitter: 0.06ms, low: 0.78ms, high: 0.92ms)
> Download: 431.13 Mbps (data used: 341.0 MB)
> 157.40 ms (jitter: 68.09ms, low: 0.88ms, high: 823.19ms)
> Upload: 351.36 Mbps (data used: 158.3 MB)
> 2.88 ms (jitter: 6.24ms, low: 1.41ms, high: 209.74ms)
> Packet Loss: 13.4%
> Result URL: https://www.speedtest.net/result/c/ff695466-3ac7-405e-8cae-0a85c2c3d5cd
> [root@pc-mtodorov marvin]#
>
> The clkpm setting can be reversed back to 1, causing the RTL speed to drop again.
>
> So, the patch is withdrawn as unnecessary, even when the majority of RTL8168h/8111h
> and possibly other Realtek Gigabit cards will by default run at sub-Gigabit speeds.
>

RTL8168h doesn't need the CLKPM quirk in general. E.g. my test system runs fine w/o it
at 950Mbps. Seems that ASPM is broken on your system.
Alternatively you can test with latest linux-next, it disables ASPM during NAPI poll.

> Thank you for your time.
>
> Best regards,
> Mirsad
>
Heiner