Re: abnormal OOM killer message

[Minchan Kim]

On Wed, Aug 19, 2009 at 11:44 AM, Minchan Kim<minchan.kim@xxxxxxxxx> wrote:
> On Wed, 19 Aug 2009 10:41:51 +0900
> ììê <chungki.woo@xxxxxxxxx> wrote:
>
>> Hi all~
>> I have got a log message with OOM below. I don't know why this
>> phenomenon was happened.
>> When direct reclaim routine(try_to_free_pages) in __alloc_pages which
>> allocates kernel memory was failed,
>> one last chance is given to allocate memory before OOM routine is executed.
>> And that time, allocator uses ALLOC_WMARK_HIGH to limit watermark.
>> Then, zone_watermark_ok function test this value with current memory
>> state and decide 'can allocate' or 'cannot allocate'.
>>
>> Here is some kernel source code in __alloc_pages function to understand easily.
>> Kernel version is 2.6.18 for arm11. Memory size is 32Mbyte. And I use
>> compcache(0.5.2).
>> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>> Â Â Â Â ...
>> Â Â Â Â did_some_progress = try_to_free_pages(zonelist->zones,
>> gfp_mask); Â Â Â Â Â Â<== direct page reclaim
>>
>> Â Â Â Â p->reclaim_state = NULL;
>> Â Â Â Â p->flags &= ~PF_MEMALLOC;
>>
>> Â Â Â Â cond_resched();
>>
>> Â Â Â Â if (likely(did_some_progress)) {
>> Â Â Â Â Â Â Â Â page = get_page_from_freelist(gfp_mask, order,
>> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â zonelist, alloc_flags);
>> Â Â Â Â Â Â Â Â if (page)
>> Â Â Â Â Â Â Â Â Â Â Â Â goto got_pg;
>> Â Â Â Â } else if ((gfp_mask & __GFP_FS) && !(gfp_mask &
>> __GFP_NORETRY)) { Â Â<== when fail to reclaim
>> Â Â Â Â Â Â Â Â /*
>> Â Â Â Â Â Â Â Â Â* Go through the zonelist yet one more time, keep
>> Â Â Â Â Â Â Â Â Â* very high watermark here, this is only to catch
>> Â Â Â Â Â Â Â Â Â* a parallel oom killing, we must fail if we're still
>> Â Â Â Â Â Â Â Â Â* under heavy pressure.
>> Â Â Â Â Â Â Â Â Â*/
>> Â Â Â Â Â Â Â Â page = get_page_from_freelist(gfp_mask|__GFP_HARDWALL,
>> order, Â<== this is last chance
>> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â zonelist,
>> ALLOC_WMARK_HIGH|ALLOC_CPUSET); Â Â Â Â Â Â Â <== uses
>> ALLOC_WMARK_HIGH
>> Â Â Â Â Â Â Â Â if (page)
>> Â Â Â Â Â Â Â Â Â Â Â Â goto got_pg;
>>
>> Â Â Â Â Â Â Â Â out_of_memory(zonelist, gfp_mask, order);
>> Â Â Â Â Â Â Â Â goto restart;
>> Â Â Â Â }
>> Â Â Â Â ...
>> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> In my case, you can see free pages(6804KB) is much more higher than
>> high watermark value(1084KB) in OOM message.
>> And order of allocating is also zero.(order=0)
>> In buddy system, the number of 4kbyte page is 867.
>> So, I think OOM can't be happend.
>>
>
> Yes. I think so.
>
> In that case, even we can also avoid zone defensive algorithm.
>
>> How do you think about this?
>> Is this side effect of compcache?
>
> I don't know compcache well.
> But I doubt it. Let's Cced Nitin.
>
>> Please explain me.
>> Thanks.
>>
>> This is OOM message.
>> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>> oom-killer: gfp_mask=0x201d2, order=0 Â Â Â (==> __GFP_HIGHMEM,
>> __GFP_WAIT, __GFP_IO, __GFP_FS, __GFP_COLD)
>> [<c00246c0>] (dump_stack+0x0/0x14) from [<c006ba68>] (out_of_memory+0x38/0x1d0)
>> [<c006ba30>] (out_of_memory+0x0/0x1d0) from [<c006d4cc>]
>> (__alloc_pages+0x244/0x2c4)
>> [<c006d288>] (__alloc_pages+0x0/0x2c4) from [<c006f054>]
>> (__do_page_cache_readahead+0x12c/0x2d4)
>> [<c006ef28>] (__do_page_cache_readahead+0x0/0x2d4) from [<c006f594>]
>> (do_page_cache_readahead+0x60/0x64)
>> [<c006f534>] (do_page_cache_readahead+0x0/0x64) from [<c006ac24>]
>> (filemap_nopage+0x1b4/0x438)
>> Âr7 = C0D8C320 Âr6 = C1422000 Âr5 = 00000001 Âr4 = 00000000
>> [<c006aa70>] (filemap_nopage+0x0/0x438) from [<c0075684>]
>> (__handle_mm_fault+0x398/0xb84)
>> [<c00752ec>] (__handle_mm_fault+0x0/0xb84) from [<c0027614>]
>> (do_page_fault+0xe8/0x224)
>> [<c002752c>] (do_page_fault+0x0/0x224) from [<c0027900>]
>> (do_DataAbort+0x3c/0xa0)
>> [<c00278c4>] (do_DataAbort+0x0/0xa0) from [<c001fde0>]
>> (ret_from_exception+0x0/0x10)
>> Âr8 = BE9894B8 Âr7 = 00000078 Âr6 = 00000130 Âr5 = 00000000
>> Âr4 = FFFFFFFF
>> Mem-info:
>> DMA per-cpu:
>> cpu 0 hot: high 6, batch 1 used:0
>> cpu 0 cold: high 2, batch 1 used:1
>> DMA32 per-cpu: empty
>> Normal per-cpu: empty
>> HighMem per-cpu: empty
>> Free pages: Â Â Â Â6804kB (0kB HighMem)
>> Active:101 inactive:1527 dirty:0 writeback:0 unstable:0 free:1701
>> slab:936 mapped:972 pagetables:379
>> DMA free:6804kB min:724kB low:904kB high:1084kB active:404kB
>> inactive:6108kB present:32768kB pages_scanned:0 all_unreclaimable? no
>> lowmem_reserve[]: 0 0 0 0
>> DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
>> present:0kB pages_scanned:0 all_unreclaimable? no
>> lowmem_reserve[]: 0 0 0 0
>> Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
>> present:0kB pages_scanned:0 all_unreclaimable? no
>> lowmem_reserve[]: 0 0 0 0
>> HighMem free:0kB min:128kB low:128kB high:128kB active:0kB
>> inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
>> lowmem_reserve[]: 0 0 0 0
>> DMA: 867*4kB 273*8kB 36*16kB 2*32kB 0*64kB 0*128kB 0*256kB 1*512kB
>> 0*1024kB 0*2048kB 0*4096kB = 6804kB
>> DMA32: empty
>> Normal: empty
>> HighMem: empty
>> Swap cache: add 4597, delete 4488, find 159/299, race 0+0
>> Free swap Â= 67480kB
>> Total swap = 81916kB
>
> In addition, total swap : 79M??
>
>> Free swap: Â Â Â Â67480kB
>> 8192 pages of RAM
>> 1960 free pages
>> 978 reserved pages
>> 936 slab pages
>> 1201 pages shared
>> 109 pages swap cached
>
> free page : 6M
> page table + slab + reserved : 8M
> active + inacive : 6M
>
> Where is 12M?
>
>> Out of Memory: Kill process 47 (rc.local) score 849737 and children.
>> Out of memory: Killed process 49 (CTaskManager).
>> Killed
>> SW image is stopped..
>> script in BOOT is stopped...
>> Starting pid 348, console /dev/ttyS1: '/bin/sh'
>> -sh: id: not found
>> #
>> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> As you mentioned, your memory size is 32M and you use compcache.
> How is swap size bigger than your memory size ?
> Is the result of compression of swap pages ?
> Nitin. Could you answer the question?
>
> I can't imagine whey order 0 allocation failed although there are
> many pages in buddy.
>
> What do you mm guys think about this problem ?

I can only think that zonelists set up wrongly or freelist got damaged.
Could you print your zonelist about __GFP_HIGHMEM ?

> --
> Kind regards,
> Minchan Kim
>



-- 
Kind regards,
Minchan Kim
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/