Re: [PATCH v3 00/31] Hardware Events Report Mecanism (HERM)

From: Borislav Petkov
Date: Fri Feb 10 2012 - 08:26:39 EST

Next message: Steven Rostedt: "Re: [PATCHv8 0/8] ftrace, perf: Adding support to use function trace"
Previous message: Wu Fengguang: "[PATCH v2] block: remove plugging at buffered write time"
In reply to: Mauro Carvalho Chehab: "[PATCH v3 06/31] edac: Add per dimm's sysfs nodes"
Next in thread: Mauro Carvalho Chehab: "Re: [PATCH v3 00/31] Hardware Events Report Mecanism (HERM)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Thu, Feb 09, 2012 at 10:00:59PM -0200, Mauro Carvalho Chehab wrote:
> The old sysfs nodes are still supported. Latter patches will allow
> disabling the old sysfs nodes.

I wouldn't remove those easily since they're documented as an EDAC
interface in <Documentation/edac.txt> and, as such, are probably used by
people.

> All errors currently generate the printk events, as before, but they'll
> also generate perf events like:

This format needs a bit more massaging:

>
> bash-1680 [001] 152.349448: mc_error: [Hardware Error]:
> mce#0: Uncorrected error FAKE ERROR on label "mc#0channel#2slot#2 "
> (channel 2 slot 2 page 0x0 offset 0x0 grain 0 for EDAC testing only)

I don't see why the process and PID are relevant to the error reported
so it should probably go. I dunno whether this can easily be done with
the current ftrace code...

> kworker/u:5-198 [006] 1341.771535: mc_error_mce: mce#0: Corrected
>error memory read error on label "CPU_SrcID#0_Channel#3_DIMM#0 "
>(channel 0 slot 0 page 0x3a2db9 offset 0x7ac grain 32 syndrome
>0x0 CPU: 0, MCGc/s: 1000c14/0, MC5: 8c00004000010090, ADDR/MISC:
>00000003a2db97ac/00000020404c4c86, RIP: 00:<0000000000000000>, TSC: 0,
>PROCESSOR: 0:206d6, TIME: 1328829250, SOCKET: 0, APIC: 0 1 error(s):
>Unknown: Err=0001:0090 socket=0 channel=2/mask=4 rank=1)

This is too much, you probably only want to say:

Corrected DRAM read error on DIMM "CPU..."

The channel, slot, page etc should be only Kconfigurable for people who
really need it.

> kworker/u:5-198 [006] 1341.792536: mc_error_mce: mce#0: Corrected
> error Can't discover the memory rank for ch addr 0x60f2a6d76 on
> label "any memory" ( page 0x0 offset 0x0 grain 32 syndrome 0x0
> CPU: 0, MCGc/s: 1000c14/0, MC5: 8c00004000010090, ADDR/MISC:
> 0000000c1e54dab6/00000020404c4c86, RIP: 00:<0000000000000000>, TSC: 0,
> PROCESSOR: 0:206d6, TIME: 1328829250, SOCKET: 0, APIC: 0 )

I guess we can report EDAC failures to map the DIMM properly like this.

> New sysfs nodes are now provided, to match the real memory architecture.

... and we need those because...?

> For example, on a Sandy Bridge-EP machine, with up to 4 channels, and up
> to 3 DIMMs per channel:
>
> /sys/devices/system/edac/mc/mc0/
> âââ ce_channel0
> âââ ce_channel0_slot0
> âââ ce_channel0_slot1
> âââ ce_channel0_slot2
> âââ ce_channel1
> âââ ce_channel1_slot0
> âââ ce_channel1_slot1
> âââ ce_channel1_slot2
> âââ ce_channel2
> âââ ce_channel2_slot0
> âââ ce_channel2_slot1
> âââ ce_channel2_slot2
> âââ ce_channel3
> âââ ce_channel3_slot0
> âââ ce_channel3_slot1
> âââ ce_channel3_slot2
> âââ ce_count
> âââ ce_noinfo_count
> âââ dimm0
> âÂÂ âââ dimm_dev_type
> âÂÂ âââ dimm_edac_mode
> âÂÂ âââ dimm_label
> âÂÂ âââ dimm_location
> âÂÂ âââ dimm_mem_type
> âÂÂ âââ dimm_size
> âââ dimm1
> âÂÂ âââ dimm_dev_type
> âÂÂ âââ dimm_edac_mode
> âÂÂ âââ dimm_label
> âÂÂ âââ dimm_location
> âÂÂ âââ dimm_mem_type
> âÂÂ âââ dimm_size
> âââ fake_inject
> âââ ue_channel0
> âââ ue_channel0_slot0
> âââ ue_channel0_slot1
> âââ ue_channel0_slot2
> âââ ue_channel1
> âââ ue_channel1_slot0
> âââ ue_channel1_slot1
> âââ ue_channel1_slot2
> âââ ue_channel2
> âââ ue_channel2_slot0
> âââ ue_channel2_slot1
> âââ ue_channel2_slot2
> âââ ue_channel3
> âââ ue_channel3_slot0
> âââ ue_channel3_slot1
> âââ ue_channel3_slot2
> âââ ue_count
> âââ ue_noinfo_count
>
> One of the above nodes allow testing the error report mechanism by
> providing a simple driver-independent way to inject errors (fake_inject).
> This node is enabled only when CONFIG_EDAC_DEBUG is enabled, and it
> is limited to test the core EDAC report mechanisms, but it helps to
> test if the tracing events are properly accredited to the right DIMMs.

What happens with the inject_* sysfs nodes which are in EDAC already?

[..]

> The memory error handling function has now the capability of reporting
> more than one dimm, when it is not possible to put the fingers into
> a single place.
>
> For example:
> # echo 1 >/sys/devices/system/edac/mc/mc0/fake_inject && dmesg |tail -1
> [ 2878.130704] EDAC MC0: CE FAKE ERROR on mc#0channel#1slot#0 mc#0channel#1slot#1 mc#0channel#1slot#2 (channel 1 page 0x0 offset 0x0 grain 0 syndrome 0x0 for EDAC testing only)
>
> All dimm memories present on channel 1 are pointed as one of them were
> responsible for the error.

I don't see how this can be of any help? I like the EDAC failure message
better: if we cannot map it properly for some reason, we tell so the
user instead of generating some misleading data.

>
> With regards to the output, the errors are now reported on a more
> user-friendly way, e. g. the EDAC core will output:
>
> - the timestamp;
> - the memory controller;
> - if the error is corrected, uncorrected or fatal;
> - the error message (driver specific, for example "read error", "scrubbing
> error", etc)
> - the affected memory labels.

"labels"? See above, if we cannot report it properly, we better say so
instead of misleading with multiple labels.
>
> Other technical details are provided, inside parenthesis, in order to
> allow hardware manufacturers, OEM, etc to have more details on it, and
> discover what DRAM has problems, if they want/need to.

Exactly, "if they want/need to" sounds like a Kconfig option to me which
can be turned on when needed.
>
> Ah, now that the memory architecture is properly represented, the DIMM
> labels are automatically filled by the mc_alloc function call, in order
> to properly represent the memory architecture.
>
> For example, in the case of Sandy Bridge, a memory can be described as:
> mc#0channel#1slot#0
>
> This matches the way the memory is known inside the technical information,
> and, hopefully, at the OEM manuals for the motherboard.

This is not always the case. You need the silkscreen labels from the
board manufacturers as they do not always match with the DIMM topology
from the hw vendor. OEM vendor BIOS should do this with a table of
silkscreen labels or something.

Thanks.

--
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Steven Rostedt: "Re: [PATCHv8 0/8] ftrace, perf: Adding support to use function trace"
Previous message: Wu Fengguang: "[PATCH v2] block: remove plugging at buffered write time"
In reply to: Mauro Carvalho Chehab: "[PATCH v3 06/31] edac: Add per dimm's sysfs nodes"
Next in thread: Mauro Carvalho Chehab: "Re: [PATCH v3 00/31] Hardware Events Report Mecanism (HERM)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]