Re: [PATCH] net: tipc: fix stall during bclink wakeup procedure

From: David Miller
Date: Wed Sep 09 2015 - 01:51:25 EST


From: Kolmakov Dmitriy <kolmakov.dmitriy@xxxxxxxxxx>
Date: Mon, 7 Sep 2015 09:05:48 +0000

> If an attempt to wake up users of broadcast link is made when there is
> no enough place in send queue than it may hang up inside the
> tipc_sk_rcv() function since the loop breaks only after the wake up
> queue becomes empty. This can lead to complete CPU stall with the
> following message generated by RCU:
...
> The issue occurs only when tipc_sk_rcv() is used to wake up postponed
> senders:
...
> After the sender thread is woke up it can gather control and perform
> an attempt to send a message. But if there is no enough place in send
> queue it will call link_schedule_user() function which puts a message
> of type SOCK_WAKEUP to the wakeup queue and put the sender to sleep.
> Thus the size of the queue actually is not changed and the while()
> loop never exits.
>
> The approach I proposed is to wake up only senders for which there is
> enough place in send queue so the described issue can't occur.
> Moreover the same approach is already used to wake up senders on
> unicast links.
>
> I have got into the issue on our product code but to reproduce the
> issue I changed a benchmark test application (from
> tipcutils/demos/benchmark) to perform the following scenario:
> 1. Run 64 instances of test application (nodes). It can be done
> on the one physical machine.
> 2. Each application connects to all other using TIPC sockets in
> RDM mode.
> 3. When setup is done all nodes start simultaneously send
> broadcast messages.
> 4. Everything hangs up.
>
> The issue is reproducible only when a congestion on broadcast link
> occurs. For example, when there are only 8 nodes it works fine since
> congestion doesn't occur. Send queue limit is 40 in my case (I use a
> critical importance level) and when 64 nodes send a message at the
> same moment a congestion occurs every time.
>
> Signed-off-by: Dmitry S Kolmakov <kolmakov.dmitriy@xxxxxxxxxx>
> Reviewed-by: Jon Maloy <jon.maloy@xxxxxxxxxxxx>
> Acked-by: Ying Xue <ying.xue@xxxxxxxxxxxxx>
> ---
> v2: Updated after comments from Jon and Ying.

Applied, thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/