Re: [PATCH 0/3] ksm: fix incorrect count of merged pages when enabling use_zero_pages

[David Hildenbrand]

> > >    
> > > >   
> > > > > second, once a page is merged with a zero page, it's not really handled
> > > > > by KSM anymore. if you have a big allocation, of which you only touch a
> > > > > few pages, would the rest be considered "merged"? no, it's just zero
> > > > > pages, right?
> > > >
> > > > If you haven't touched memory, there is nothing populated -- no shared
> > > > zeropage.
> > > >
> > > > We only populate shared zeropages in private anonymous mappings on read
> > > > access without prior write.
> > >
> > > that's what I meant. if you read without writing, you get zero pages.
> > > you don't consider those to be "shared" from a KSM point of view
> > >
> > > does it make a difference if some pages that have been written to but
> > > now only contain zeroes are discarded and mapped back to the zero pages?
> >
> > That's a good question. When it comes to unmerging, you'd might expect
> > that whatever was deduplicated will get duplicated again -- and your
> > memory consumption will adjust accordingly. The stats might give an
> > admin an idea regarding how much memory is actually overcommited. See
> > below on the important case where we essentially never see the shared
> > zeropage.
> >
> > The motivation behind these patches would be great -- what is the KSM
> > user and what does it want to achieve with these numbers?
>
> anyone who works on big amounts of very sparse data, especially in a VM
> (as I explained above, with KSM without use_zero_pages KVM guests lose
> the zero page colouring)

I meant the patches in question here :)

> > >    
> > > >   
> > > > > this is the same, except that we take present pages with zeroes in it
> > > > > and we discard them and map them to zero pages. it's kinda like if we
> > > > > had never touched them.
> > > >
> > > > MADV_UNMERGEABLE
> > > >
> > > > "Undo  the effect of an earlier MADV_MERGEABLE operation on the
> > > > specified address range; KSM unmerges whatever pages it had merged in
> > > > the address range specified  by  addr  and length."
> > > >
> > > > Now please explain to me how not undoing a zeropage merging is correct
> > > > according to this documentation.
> > > >   
> > >
> > > because once it's discarded and replaced with a zero page, the page is
> > > not handled by KSM anymore.
> > >
> > > I understand what you mean, that KSM did an action that now cannot be
> > > undone, but how would you differentiate between zero pages that were
> > > never written to and pages that had been written to and then discarded
> > > and mapped back to a zero page because they only contained zeroes?
> >
> > An application that always properly initializes (write at least some
> > part once) all its memory will never have the shared zeropage mapped. VM
> > guest memory comes to mind, probably still the most important KSM use case.
> >
> > There are currently some remaining issues when taking a GUP R/O longterm
> > pin on such a page (e.g., vfio). In contrast to KSM pages, such pins are
> > not reliable for the shared zeropage, but I have fixes for them pending.
> > However, that is rather a corner case (it didn't work at all correctly a
> > while ago) and will be sorted out soon.
> >
> > So the question is if MADV_UNMERGEABLE etc. (stats) should be adjusted
> > to document the behavior with use_zero_pages accordingly.
>
> we can count how many times a page full of zeroes was merged with a
> zero-page, but we can't count how many time one of those pages was then
> unmerged. once it's merged it becomes a zero-page, like the others.

Right. We could special case on MADV_MERGEABLE ("how many zero pages do 
we have mapped into MADV_MERGEABLE VMAs"), but it gets tricky once we 
enable MADV_MERGEABLE on a region where the shared zeropage was already 
populated. Probably not worth the effort.


--
Thanks,

David / dhildenb