Re: igb and bnx2: "NETDEV WATCHDOG: transmit queue timed out" whenskb has huge linear buffer

From: Zoltan Kiss
Date: Tue Feb 04 2014 - 16:33:29 EST


On 31/01/14 18:56, Wei Liu wrote:
On Thu, Jan 30, 2014 at 07:08:11PM +0000, Zoltan Kiss wrote:
Hi,

I've experienced some queue timeout problems mentioned in the
subject with igb and bnx2 cards. I haven't seen them on other cards
so far. I'm using XenServer with 3.10 Dom0 kernel (however igb were
already updated to latest version), and there are Windows guests
sending data through these cards. I noticed these problems in XenRT
test runs, and I know that they usually mean some lost interrupt
problem or other hardware error, but in my case they started to
appear more often, and they are likely connected to my netback grant
mapping patches. These patches causing skb's with huge (~64kb)
linear buffers to appear more often.
The reason for that is an old problem in the ring protocol:
originally the maximum amount of slots were linked to MAX_SKB_FRAGS,
as every slot ended up as a frag of the skb. When this value were
changed, netback had to cope with the situation by coalescing the
packets into fewer frags.
My patch series take a different approach: the leftover slots
(pages) were assigned to a new skb's frags, and that skb were
stashed to the frag_list of the first one. Then, before sending it
off to the stack it calls skb = skb_copy_expand(skb, 0, 0,
GFP_ATOMIC, __GFP_NOWARN), which basically creates a new skb and
copied all the data into it. As far as I understood, it put
everything into the linear buffer, which can amount to 64KB at most.
The original skb are freed then, and this new one were sent to the
stack.

Just my two cents, if it is this case, you can try to call
skb_copy_expand on every SKB netback receives to manually create SKBs
with ~64KB linear buffer to see how it goes...

I've tried it, and it did break everything in a similar way, so that's a strong clue that the problem lies here. I've rewrote that part of my patches to do less modification, based on Malcolm's idea: netback pulls the first frag into linear buffer, then moves a frag from the frag_list skb into the first one. That seems to help, but so far I have only one relevant test result, I'm waiting for more results.

Zoli

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/