Re: [PATCH v3] ummunotify: Userspace support for MMU notifications

[Roland Dreier]

 > > Second, it turns out that having the filter does cut down quite a bit on
 > > the events.  From running some Open MPI tests that Jeff provided, I saw
 > > that there were often several times as many MMU notifier events
 > > delivered in the kernel than ended up being reported to userspace.

 > So maybe multiple invalidate_page are gathered into the same range
 > event? If so, maybe it'd make sense to cache the last used rb_node in
 > ummunotify_handle_notify()? (and if multiple ranges were invalidated at
 > once, just don't cache anything, it shouldn't happen often anyway)

Well, I just meant that there were lots of events for parts of the
address space that Open MPI wasn't interested in... the fortran runtime
or whatever was freeing stuff that wasn't ever used for communication.

 > >  > 2) What happens in case of fork? If father+child keep reading from the
 > >  > previously-open /dev/ummunotify, each event will be delivered only to
 > >  > the first reader, right? Fork is always a mess in HPC, but maybe there's
 > >  > something to do here.

 > > It works just like any other file where fork results in two file
 > > descriptors in two processes... as you point out the two processes can
 > > step on each other.  (And in the ummunotify case the file remains
 > > associated with the original mm)  However this is the case for simpler
 > > stuff like sockets etc too, and I think uniformity of interface and
 > > least surprise say that ummunotify should follow the same model.

 > I was wondering if adding a special event such as "COWED" could help
 > user-space. But maybe fork already invalidates all COW'ed ranges in
 > copy_page_range() anyway?

The problem I guess is that there is only one file object (pointed to by
two file descriptors of course) after the fork.  And it is tracking
changes to the parent's mapping.  I guess in the parent when touching
pages and triggering COW, that might be interesting -- but I don't
really know how to distinguish it from any other type of invalidate
event (and I don't know how userspace would do anything different
anyway).

I haven't actually looked at what fork() does to the MMU notifiers --
but the MMU notifiers hook in at such a low level that it does seem hard
to know that what's going on is related to fork or even COW.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/