Hi,
On Wed, Jul 12, 2000 at 06:40:56PM +0300, Mark Mokryn wrote:
> Here's the scenario:
> 2.4.0-test3 SMP build running on a single 800MHz PIII (Dell GX-300)
> After obtaining a mapping to a high memory region (i.e. either
> PCI memory or physical memory reserved by passing mem=XXX to the kernel
> at boot), I am trying to write a raw device with data in the mapped
> region.
> This fails, with map_user_kiobuf spitting out "Mapped page missing"
> The raw write works, of course, if the mapping is to a kmalloc'ed
> buffer.
>
> I have tried the above with 2.2.14 SMP, and it works, so something in
> 2.4 is broken.
No. 2.4 is merely "changed" --- that's different!
The 2.2 mechanism was fine for all memory under 4GB --- ie. memory you
can address with an unsigned 32 bit integer. 2.4 includes large
memory support for 64GB memory on Intel, and we looked at ways of
making kiobufs work with this. The only reasonable solution was to
use "struct page *" pointers in the kiobuf in 2.4.
That is a design decision, and is deliberate. It means that we don't
have the ability to address IO aperture pages directly using the
normal struct page *'s from the mem_map array.
Eventually we want to be able to allow new struct page arrays to be
allocated at run time to describe arbitrary high-memory ranges, and
that will provide natural support for kiobufs on IO apertures. For
now, however, it just won't work, and we're aware of the missing
functionality.
> On another interesting note: The raw devices I'm writing to are Fibre
> Channel drives controlled by a Qlogic 2200 adapter (in 2.2.14 I'm using
> the Qlogic driver). When writing large sequential blocks to a single
> drive, I reached 8MB/s when the memory was mapped to the high reserved
> region, while CPU utilization was down to about 5%. When the mapping was
> to PCI space, I was able to write at only 4MB/s, and CPU utilization was
> up to 60%! This is very strange, since if the transfer rate was for some
> unknown reason lower in the case of PCI (vs. high physical memory), then
> one would expect the CPU utilization to be even lower, since the adapter
> performs DMA. But instead, the CPU is sweating...
PCI reads are very, very slow. If the CPU is contending with a large
DMA transfer for access to the bus, then you might find that the high
CPU utilisation is just coming about due to the CPU's IO or memory
accesses experiencing a massive slowdown due to bus contention. PCI
is a _lot_ faster for write than for read to PCI-mapped memory.
> So, it appears that
> there's a problem in 2.2.14 as well, when the mapping is to PCI space...
I'd need to see a bus analysis first --- this could well be
hardware...
> Additionally, the max transfer rate of 8MB/s seems rather slow - don't
> know why yet...
I have no trouble getting much more than that from raw I/O, but the
qlogics may have bandwidth problems if the IO size is small (current
raw IO for 2.2 limits the IO chunk size to 64kB).
Cheers,
Stephen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Sat Jul 15 2000 - 21:00:19 EST