Trying to make use of hotplug memory for xen balloon driver

[Jeremy Fitzhardinge]

Hi,

I'm trying to make use of hotplug memory in the Xen balloon driver.  If 
you want to expand a domain to be larger than its initial size, it must 
add new page structures to describe the new memory.

The platform is x86-32, with CONFIG_SPARSEMEM and 
CONFIG_HOTPLUG_MEMORY.  Because the new memory is only pseudo-physical, 
the physical address within the domain is arbitrary, and I added a 
add_memory_resource() function so I could use allocate_resource() to 
find an appropriate address to put the new memory at.

When I want to expand the domain's memory, I do (error checking edited 
out for brevity):

       res = kzalloc(sizeof(*res), GFP_KERNEL);

       res->name = "Xen Balloon";
       res->flags = IORESOURCE_MEM | IORESOURCE_BUSY;

       ret = allocate_resource(&iomem_resource, res, size, 0, -1,
                               PAGE_SIZE, NULL, NULL);

       ret = add_memory_resource(0, res);

       start_pfn = res->start >> PAGE_SHIFT;
       end_pfn = (res->end + 1) >> PAGE_SHIFT;

       ret = xen_resize_phys_to_mach(end_pfn);

       for(pfn = start_pfn; pfn < end_pfn; pfn++) {
               struct page *page = pfn_to_page(pfn);

               if (PageReserved(page))
                       continue;

               set_phys_to_machine(pfn, INVALID_P2M_ENTRY);
               balloon_append(page);
       }

at this point the pages have no underlying machine (physical) memory, 
but are added to the list of potentially usable pages.  This all works fine.

However, when I actually want to use one of these pages, I do:

               page = balloon_retrieve();

               pfn = page_to_pfn(page);

               set_phys_to_machine(pfn, frame_list[i]);

               /* Relinquish the page back to the allocator. */
               online_page(page);

               /* Link back into the page tables if not highmem. */
               if (pfn < max_low_pfn) {	/* !PageHighMem(page) ? */
                       int ret;
                       ret = HYPERVISOR_update_va_mapping(
                               (unsigned long)__va(pfn << PAGE_SHIFT),
                               mfn_pte(frame_list[i], PAGE_KERNEL),
                               0);
                       BUG_ON(ret);
               }

This has two problems:

  1. the online_page() raises an error:

     Bad page state in process 'events/0'
     page:c16fa0cc flags:0x00000000 mapping:00000000 mapcount:1 count:0
     Trying to fix it up, but a reboot is needed
     Backtrace:
     Pid: 9, comm: events/0 Not tainted 2.6.25-rc7-x86-latest.git-dirty #353
      [<c015643a>] bad_page+0x55/0x82
      [<c0156be6>] free_hot_cold_page+0x60/0x1f1
      [<c0103069>] ? xen_restore_fl+0x2e/0x52
      [<c0156dae>] free_hot_page+0xa/0xc
      [<c0156dcb>] __free_pages+0x1b/0x26
      [<c0466e8c>] free_new_highpage+0x11/0x19
      [<c0466ea1>] online_page+0xd/0x1b
      [<c02809ac>] balloon_process+0x1e6/0x4d3
      [<c014671a>] ? lock_acquire+0x90/0x9d
      [<c0137720>] run_workqueue+0xbb/0x186
      [<c01376e5>] ? run_workqueue+0x80/0x186
      [<c02807c6>] ? balloon_process+0x0/0x4d3
      [<c0137fe6>] ? worker_thread+0x0/0xbe
      [<c0138099>] worker_thread+0xb3/0xbe
      [<c013a635>] ? autoremove_wake_function+0x0/0x33
      [<c013a56a>] kthread+0x3b/0x61
      [<c013a52f>] ? kthread+0x0/0x61
      [<c0108b67>] kernel_thread_helper+0x7/0x10
      =======================
         

     I can solve this by putting an explicit reset_page_mapcount(page)
     before online_page(), but I can't see any other hotplug memory
     code which does this.

  2. The new pages don't appear to be in the right zone.  When I boot a
     256M domain I get an initial setup of:

     Zone PFN ranges:
       DMA             0 ->     4096
       Normal       4096 ->    65536
       HighMem     65536 ->    65536
     Movable zone start PFN for each node
     early_node_map[1] active PFN ranges
         0:        0 ->    65536
     On node 0 totalpages: 65536
       DMA zone: 52 pages used for memmap
       DMA zone: 0 pages reserved
       DMA zone: 4044 pages, LIFO batch:0
       Normal zone: 780 pages used for memmap
       Normal zone: 60660 pages, LIFO batch:15
       HighMem zone: 0 pages used for memmap
       Movable zone: 0 pages used for memmap
         

     which presumably means that new pages above pfn 65536 should be in
     the highmem zone?  But PageHighMem() returns false for those pages.

What am I missing here?

Thanks,
   J
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/