Re: HTLB mapping for drivers. Driver example

[Mel Gorman]

On Wed, Aug 19, 2009 at 09:01:17PM +1200, Alexey Korolev wrote:
> > This seems a lot of burden to put on a device driver, particularly with
> > respect to the reservations.
> 
> Thanks a lot for review you did. That is right.  I don't like this
> burden as well.
> >
> >> File operations of /dev/hpage_map do the following:
> >>
> >> In file open we  associate mappings of /dev/xxx with the file on hugetlbfs (like it is done in ipc/shm.c)
> >>       file->f_mapping = h_file->f_mapping;
> >>
> >> In get_unmapped_area we should tell about addressing constraints in case of huge pages by calling hugetlbfs procedures. (as in ipc/shm.c)
> >>       return get_unmapped_area(h_file, addr, len, pgoff, flags);
> >>
> >> We need to let hugetlbfs do architecture specific operations with mapping in mmap call. This driver does not reserve any memory for private mappings
> >> so driver requests reservation from hugetlbfs. (Actually driver can do this as well but it will make it more complex)
> >>
> >> The exit procedure:
> >> * removes memory from page cache
> >> * deletes file on hugetlbfs vfs mount
> >> *  free pages
> >>
> >> Application example is not shown here but it is very simple. It does the following: open file /dev/hpage_map, mmap a region, read/write memory, unmap file, close file.
> >>
> >
> > For the use-model you have in mind, could you look at Eric Munson's patches
> > and determine if the target application would have been happy to call the
> > following please?
> >
> > mmap(0, len, prot, MAP_ANONYMOUS|MAP_HUGETLB, 0, 0)
> >
> Hmm. But how can I at least identify which driver this call is addressed to?
> 

The example you gave was for /dev/hpage_map so in this specific case, it
would have appeared that the application didn't want hugepages belonging to
a particular driver, but huge pages in general. Furthermore, your example
driver was not populating the hugepages with data so in this case, calling
mmap(MAP_ANONYMOUS|MAP_HUGETLB) would have been sufficient. If all the data
in your target application is populated from userspace, it's worth considering
instead of a different driver.

However, lets assume you have a driver that provides the data from
somewher. The implementation for MAP_HUGETLB is basically a call to a
hugetlbfs_file_setup()-like function that is very straight-forward. It
creates a hugetlbfs file and ensures that the reservations are there which
is important and tricky to get right.

Would it be possible for your driver to do

On file open
	Create hugetlbfs-backed-file using helper similar to Eric's
		or maybe even Eric's helper for MAP_HUGETLB
	Copy get_unmapped_area handle from hugetlbfs-file so mappings
		are properly placed on mmap()

On file mmap
	Use a new helper to get a reference to each page within the
		file and populate it with driver-specific data. You
		would need a new patch for this helper because it
		doesn't exist.
	Call h_file->f_op->mmap(h_file, vma)

On file fault
	The data is already in the page cache so the normal hugetlbfs
		handlers should do the job

On file close
	Drop the hugetlbfs inode

Using the helper for getting a reference to a hugetlbfs-file, your driver
would no longer be responsible for placing the mapping, handling reservations
or handling page cache manipulations.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/