Update on AMD Athlon/Opteron/Athlon64 Prefetch Errata

[richard . brunner]

Dear LKML,

Continuing my yearly tradition of posting just one long
novel to LKML every year, here is the literary update on the
Prefetch Errata that the early 2.6 Kernels hit on AMD Athlon
Processors.

This previously published errata can occur infrequently and
is present in all AMD Athlon processors and earlier AMD
Opteron/Athlon64 processors.  See [1] and [2].

The full details are below, but the key point is that under
certain circumstances, prefetch instructions can get memory
management faults for addresses which would fault if they
were accessed by a load or store instruction.  We plan to
revise our published errata with the new information below.

The errata requires a kernel workaround, but the good news
is that it is:

  - Harmless in most cases where it could occur. Most of the
    time the prefetch will be targeting memory that is
    accessible under the current privilege mode. So the page
    will simply be "faulted in" slightly earlier than
    needed.

  - Rare and Infrequent. AMD Athlon processors have been
    available for years running numerous Operating Systems
    and only recently have we hit this errata outside of
    code specifically designed to target the errata --
    requiring tens of thousands of iterations to cause it.

  - It can be worked around. Andi Kleen has a 2.6 and a 2.4
    Kernel patches that we have tested at AMD on a large
    number of AMD Athlon processors and AMD Opteron/Athlon64
    processors (both legacy x86 and x86-64 long mode).  It
    works just fine. (Andi will be posting them soon when he 
    wakes up ;-)

  - AMD is fixing this in future revisions of AMD
    Opteron/Athlon64 processors.

  - Andi's kernel patches will not be needed on future
    AMD processors but it is forward compatible and so
    won't break on them either.
 


The Details
===========
Software prefetch instructions are defined to ignore page
faults. Under highly specific and detailed internal
circumstances, the following conditions may cause the
PREFETCH instruction to report a page fault.

+ The target address of the PREFETCH would cause a page
  fault if the address was accessed by an actual memory load
  or store instruction under the current privilege mode.

+ The instruction is a PREFETCH or PREFETCHNTA/0/1/2
  followed in execution-order by an actual or speculative
  byte-sized load to the same address.

  In this case, the page fault exception error code bits for
  the faulting PREFETCH would be identical to that for a
  byte-sized load to the same address.

+ The instruction is a PREFETCHW followed in execution-order
  by an actual or speculative byte-sized store to the same
  address.
  
  In this case, the page fault exception error code bits for
  the faulting PREFETCHW would be identical to that for a
  byte-sized store to the same address.

Note that some misaligned accesses can be broken up by the
processor into multiple accesses where at least one of the
accesses is a byte-sized access.

If the target address of the subsequent memory load or store
is aligned and not byte-sized, this errata does not occur
and no work-around is needed.

So the net effect is that an unexpected page fault may occur
infrequently on a PREFETCH instruction.


Kernel Work-around
=================
The kernel can work around the errata by modifying the Page
Fault Handler in the following way.  This is what Andi
Kleen's patches do.  Because the actual errata is infrequent
it does not produce an excessive number of page faults that
affect system performance.
   
+ Continue to allow the page fault handler to satisfy the
  page fault.  If the faulting instruction is permitted
  access to the page, return to it as usual.

+ If the faulting instruction is not permitted access to the
  page, scan the instruction stream bytes at the faulting
  Instruction Pointer to determine if the instruction is a
  PREFETCH. 

+ If it is not a PREFETCH instruction, generate the
  appropriate memory access control violation as
  appropriate.

+ If the faulting instruction is a PREFETCH instruction,
  simply return back to it; the internal hardware conditions
  that caused the PREFETCH to fault should be removed and
  operation should continue normally.


General Work-around
===================
If the page-fault handler for a kernel can be patched as
described above, no further action by software is
required. The following general work-arounds should only be
considered for kernels where the page-fault handler can not
be patched and a PREFETCH instruction could end up targeting
an address in an "inaccessible" page. (An "inaccessible"
page is one for which memory accesses are not allowed under
the current privilege mode.)

Because the actual errata is infrequent, it does not produce
an excessive number of page faults that affect system
performance.  Therefore a page fault from a PREFETCH
instruction for an address within an "accessible" page does
not require any general work-around.  (An "accessible" page
is one for which memory accesses are allowed under the
current privilege mode once the page is resident in memory)

Software can minimize the occurrence of the errata by
issuing only one PREFETCH instruction per cache-line (a
naturally-aligned 64-byte quantity on AMD Athlon and AMD
Opteron/Athlon64) and ensuring one of the following:

+ In many cases, if a particular target address of a
  prefetch is known to encounter this errata, simply change
  the prefetch to target the next byte.

+ Avoid prefetching inaccessible memory locations, when
  possible.

+ In the general case, ensure that the address used by the
  PREFETCH is offset into the middle of an aligned quadword
  near the end of the cache-line. For example, if the
  address desired to be prefetched is "ADDR", use an offset
  of 0x33 to compute the address used by the actual PREFETCH
  instruction as: "(ADDR & ~0x3f) + 0x33"




Footnotes
=========
[1] AMD Athlon(tm) Processor Model 6 Revision Guide 24332F June 2003.
    
www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/24332.pdf


[2] Revision Guide for AMD Opteron(tm) Processors 25759 Rev. 3.07 Aug 2003

www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/25759.PDF


] -Rich ...
] AMD Fellow

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/