Re: A question concerning time outs and possible lost interrupts

Linus Torvalds (torvalds@transmeta.com)
Sun, 13 Sep 1998 13:28:34 -0700 (PDT)


On Sun, 13 Sep 1998, Edward Welbon wrote:
>
> I get endless scsi timeouts at various times (usually sometime very soon
> after I switch root to drives controlled by ncr53c8xx (raid0 as root)). Do
> you have any ideas or suggestions as to how to find out if an interrupt is
> really getting lost? How do I learn to interpret the bytes provided by
> ncr53c8xx_abort?

One thing to look out for is your interrupt setup.

If you have edge-triggered IO-APIC interrupts, they happen to not work ok
in 2.1.120 and 2.1.121. They should be ok in earlier kernels and again in
pre-2.1.122-2.

More importantly, though, edge-triggered interrupts are nasty. In
particular, two separate interrupts can turn into one interrupt regardless
of what the kernel does. So the first thing you should check is whether
you can force the BIOS to use a level-triggered interrupt, assuming you
don't have one already.

The problem is that if the edge-triggered interrupt never goes away, it
will just stay active and never show another edge. Even a "good" driver
can easily have problems that are very timing-related..

In particular, a common thing for various drivers is to have a ring of
events, and traverse the whole ring of events for every interrupt. It
works 99.99% of all times, which is why people do this, but it's broken.

Let's say that you have a ring of four events: 0 1 2 3, and you have a
disk controller that raises an interrupt whenever one of the events is
active. Let's further say that event #0 and #3 become active at roughly
the same time, and you get an interrupt. The CPU sees the edge, and
everything is good. The driver starts looking at the events..

Now, the driver sees event #0, and handles it. So far so good. It looks at
status #1, and that isn't ready yet, so it looks at #2 which is the same.
Finally, it looks at #3 and handles it. However, before/while doing that,
event #1 came in and is now active.

And now you have a hung system.

The reason for the hung system is that while the interrupt dispatch
routine correctly handled event #0 and #3, it never handled #1. And we
never got another interrupt, because our interrupt was edge-triggered, and
we never saw another edge because the interrupt was always active.

The correct thing for a driver to do is to do one of two things:

- always require the interrupt to be level-triggered. This is the "good"
solution, but sadly there are setups out there that still use
edge-triggered interrupts. Don't ask me why.

One option migth be to have a way of overriding the edge/level decision
that the BIOS made for us. We could just decide that even though the
BIOS told us to use an edge interrupt, the BIOS is just wrong.

However, sometimes the BIOS might just be right. Certainly some really
stupid devices require a edge interrupt simply because they don't
de-assert their own interrupt line in any sane manner (this is true of
the timer interrupt, for example, which is just a square-wave thing and
thus would generate an endless stream of interrupts for 50% of the time
if it was level-triggered - and there may be other broken hardware out
there with the same bad behaviour)

- if there are multiple status mailboxes like the above, the driver has
to go through each and every one endlessly until it has gone one
complete round without seeing a single event (this assumes that events
don't go away once they are posted - and thus the "not seeing a single
event" guarantees that at some point in time all events were quiescent
and thus that we will get a new edge if some new event ever happens)

I don't know whether the ncr driver does this already or not, but if it
doesn't, then that may be the cause for the occasional timeouts..

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/faq.html