Re: x86_64 kernel stack organization

From: Andi Kleen
Date: Fri Jul 14 2006 - 07:43:25 EST


On Friday 14 July 2006 09:00, Keith Owens wrote:
> I could not find a document that described the x86_64 kernel stack
> organization so I wrote this one. Sending to lkml for a sanity check
> before doing a patch against linux/Documentation, plus trying to get
> answers to some questions.

It contains too much implementation detail. Will likely outdate quickly.
Better shorten at least by 50%

Also I'm not sure it is that useful to describe the hardware architecture
here (like ISTs). That is well documented in the Intel/AMD manuals
and a serious kernel programmer should read these anyways. Just add
a one line reference.

> x86_64 page size (PAGE_SIZE) is 4K.
>
> Like all other architectures, x86_64 has a kernel stack for every
> active thread. These thread stacks are THREAD_SIZE (2*PAGE_SIZE) big.
> These stacks contain useful data as long as a thread is alive or a
> zombie, no matter whether the thread is in user space or in the kernel.
>
> In addition to the per thread stacks, there are specialized stacks
> associated with each cpu. These stacks are only used while the kernel
> is in control on that cpu, when a cpu returns to user space the
> specialized stacks contain no useful data. The main cpu stacks is

are

> * Interrupt stack. IRQSTACKSIZE (4*PAGE_SIZE).
>
> Used for external hardware interrupts. If this is the first external
> hardware interrupt (i.e. not a nested hardware interrupt) then the
> kernel switches from the current task to the interrupt stack. Like
> the split thread and interrupt stacks on i386 (with CONFIG_4KSTACKS),

It is the other way round - x86-64 was first, i386 copied the scheme.

> this gives more room for kernel interrupt processing without having
> to increase the size of every per thread stack.
>
> The interrupt stack is also used when processing a softirq. Unlike
> the hardware interrupt code which allows nested interrupts, softirq
> processing unconditionally switches to using the interrupt stack.
> Soft irqs cannot be nested.
>
> Switching to the kernel interrupt stack is done by software, allowing
> the kernel to decide when to switch.

Among all the irrelevant detail you left out the crucial points:
The hardware interrupt frame is still written on the original stack.
x86-64 redzones (allows compiler access stack pointer) have to be
disabled by a compiler option because of that. And ISTs are not used for
normal interrupt stacks because they don't nest.

> x86_64 also has a feature which

That should be all removed.

> The currently assigned IST stacks are :-

We call them exception stacks which they are.

> * STACKFAULT_STACK. EXCEPTION_STKSZ (PAGE_SIZE).
>
> Used for interrupt 12 - Stack Fault Exception (#SS).
>
> Question: Why use an IST for this event instead of the normal
> hardware interrupt stack? Is this an attempt to detect kernel stack
> overflow? AFAICT it is ineffective, x86_64 does not check segment
> limits in 64bit mode.

Stackfault happens for more cases than just stack segment limit checking.

> Question: The kernel code allows for DEBUG_STKSZ to be different from
> EXCEPTION_STKSZ, but currently they are the same value. This
> prevents debug interrupts from being nested.
> Hardware debug events
> cannot be nested, but software debug interrupts have the potential to
> be nested. Macro paranoid adjusts the TSS entry by EXCEPTION_STKSZ
> which only allows one level of debug interrupt. Is this intentional?
> I note that http://lkml.org/lkml/2006/5/10/22 tries to address this.

Revisiting that is on my todo list. I still maintain it's more a kprobes
bug than a real feature though.


-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/