Re: Thread implementations...

Richard Gooch (Richard.Gooch@atnf.CSIRO.AU)
Sun, 21 Jun 1998 01:51:31 +1000

Alex Belits writes:
> On Fri, 19 Jun 1998, David S. Miller wrote:
>
> > I look at it this way.
> >
> > If you can divide the total set of fd's logically into seperate
> > groups, one strictly to a particular thread. Do it this way.
> > The problem with one thread polling all fd's and passing event
> > notification to threads via some other mechanism has the problem that
> > this one thread becomes the bottle neck.
>
> I realize that every operation, performed indide that process/thread, if
> takes any noticeable time, will hold back everything that depends on any
> fd status change. But what if the code is optimized to reduce the time in
> loop to the absolute minimum possible? Will poll() take more time by
> itself (and indeed become a bottleneck) in one thread vs. multiple
> poll()'s made at the same time in multiple threads? If the time spent in
> the loop is minimal, is there any difference between waking up one of
> looping threads, searching through its poll array and performing some
> action, and with one thread waking up every time, searching larger array
> (IMHO not a significant time compared to time spent by system while
> processing those sockets) and then performing the same action, if that
> action takes some insignificant time, comparable with time, spent in
> buffers handling in the kernel itself? As I understand, with multiple
> threads ot not, kernel still needs a time to process file descriptors
> and choose thread to wake up even if threads already divided fds among
> themselves, so the total amount of fd lists scanning won't change.

Assuming that most FDs are inactive, the time spent scanning a list of
FDs is 2-3 us per FD. So for 1000 FDs, we are looking at milliseconds,
which is quite a bit compared to some simple datagram processing in
userspace. So the time for select(2) or poll(2) of large numbers of
FDs is significant.

Splitting this work across many threads (say 10) reduces the
probability that more than one thread needs to be woken up during any
timeslice, hence far fewer FDs need to be scanned each time (only 100
in this example).

Unfortunately splitting the work amongst many threads is not always
easy. We can improve the speed of select(2) and poll(2) by a factor of
3 by changing the way they are implemented (API remains the same, of
course:-). This will buy us more in the scalability stakes.

> > The problem, for one, with web etc. servers is the incoming connection
> > socket. If you could tell select/poll "hey, when a new conn comes in,
> > wake up one of us", poof this issue would be solved. However the
> > defined semantics for these interfaces says to wake everyone polling
> > on it up.
>
> This is why I do that in userspace -- one process is always waking up,
> connection is placed in its internal queue, its fd is added to the
> polling list, and after request is received and parsed asynchronously, fd
> is immediately passed to another process through the AF_UNIX socket. While
> main process is doing nonblocking I/O on multiple connections, there is no
> I/O in the same loop except opening new connections, reading from them and
> passing to other processes fds/data of connections that have sent their
> requests and expect the response. Kind of userspace "multithreading",
> optimized for the particular operation.

People seem to be very interested in the new connection case. This
doesn't seem all that exiting or interesting to me (just have one
thread blocking in accept(2) and store the new FD in a global
array). To me the problem of processing data on existing connections
is more interesting (and harder to solve: hence more interesting:-).
Is there something deep I'm missing here?

Regards,

Richard....

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu