Re: [ANNOUNCE 0/6] Open-iSCSI High-Performance Initiator for Linux
From: Dave Wysochanski
Date: Sat Mar 12 2005 - 11:59:11 EST
Alex Aizman wrote:
This is to announce Open-iSCSI project: High-Performance iSCSI
Initiator for
Linux.
MOTIVATION
==========
Our initial motivations for the project were: (1) implement the right
user/kernel split, and (2) design iSCSI data path for performance.
Recently
we added (3): get accepted into the mainline kernel.
As far as user/kernel, the existing iSCSI initiators bloat the kernel
with
ever-growing control plane code, including but not limited to: iSCSI
discovery, Login (Authentication and Operational), session and connection
management, connection-level error processing, iSCSI Text, Nop-Out/In,
Async
Message, iSNS, SLP, Radius... Open-iSCSI puts the entire control plane in
the user space. This control plane talks to the data plane via well
defined
interface over the netlink transport.
(Side note: prior to closing on the netlink we considered: sysfs,
ioctl, and
syscall. Because the entire control plane logic resides in the user
space,
we needed a real bi-directional transport that could support asynchronous
API to transfer iSCSI control PDUs: Login, Logout, Nop-in, Nop-Out, Text,
Async Message.
Performance.
This is the major goal and motivation for this project. As it happens,
iSCSI
has to compete with Fibre Channel, which is a more entrenched
technology in
the storage space. In addition, the "soft" iSCSI implementation have
to show
good results in presence of specialized hardware offloads.
Our today's performance numbers are:
- 450MB/sec Read on a single connection (2-way 2.4Ghz Opteron, 64KB block
size);
- 320MB/sec Write on a single connection (2-way 2.4Ghz Opteron, 64KB
block
size);
- 50,000 Read IOPS on a single connection (2-way 2.4Ghz Opteron, 4KB
block
size).
Has anyone on the list verified these #'s? I'm trying to
get open-iscsi to work but it looks like it's got a problem
in the very initial stages of lun scanning that prevents
my target from working. Open-iscsi guys I have a trace
if you want to look at it. Looks like despite the fact that
report luns is returned successfully and only 1 lun is
returned (lun 0), the initiator is still sending inquiry
commands to luns > 0, and it looks like it gets confused
when it gets a 0x3f inquiry response from the target
(for an inquiry to lun 1), tries to issue a TMF abort
task on the previous inquiry which has already completed,
and the target responds with "task not in task set", which
is understandable since the command has already completed.
I used the latest .169 code.
I don't see this problem with the latest linux-iscsi.sfnet
code and have interoperated with many other initiators,
so I'm fairly confident there's a bug in open-iscsi somewhere.
Prior to starting from-scratch the data path code we did evaluate the
sfnet
Initiator. And eventually decided against patching it. Instead, we reused
its Discovery, Login, etc. control plane code.
Technically, it was the shortest way to achieve the (1) and (2) goals
stated
above. We believe that it remains the easiest and the most practical
thing
on the larger scale of: iSCSI for Linux.
STATUS
======
There's a 100% working code that interoperates with all (count=5) iSCSI
targets we could get our hands on.
The software was tested on AMD Opteron (TM) and Intel Xeon (TM).
Code is available online via either Subversion source control database or
the latest development release (i.e., the tarball containing Open-iSCSI
sources, including user space, that will build and run on kernels
starting
2.6.10).
http://www.open-iscsi.org
Features:
- highly optimized and small-footprint data path;
- multiple outstanding R2Ts;
- thread-less receive;
- sendpage() based transmit;
- zero-copy header processing on receive;
- no data path memory allocations at runtime;
- persistent configuration database;
- SendTargets discovery;
- CHAP;
- DataSequenceInOrder=No;
- PDU header Digest;
- multiple sessions;
- MC/S (note: disabled in the patch);
- SCSI-level recovery via Abort Task and session re-open.
TODO
====
The near term plan is: test, test, and test. We need to stabilize the
existing code, after 5 months of development this seems to be the right
thing to do.
Other short-term plans include:
a) process community feedback, implement comments and apply patches;
b) cleanup user side of the iSCSI open interface; use API calls
(instead of
directly constructing events);
c) eliminate runtime control path memory allocations (for Nop-In,
Nop-Out,
etc.);
d) implement Write path optimizations (delayed because of the
self-imposed
submission deadline);
e) oProfile the data path, use the reports for further optimization;
f) complete the readme.
Comments, code reviews, patches - are greatly appreciated!
THANKS
======
Special thanks to our first reviewers: Christoph Hellwig and Mike
Christie.
Special thanks to Ming Zhang for help in testing and for insightful
questions.
Regards,
Alex Aizman & Dmitry Yusupov
=============================================
The following 6 patches alltogether represent the Open-iSCSI Initiator:
Patch 1:
SCSI LLDD consists of 3 files:
- iscsi_if.c (iSCSI open interface over netlink);
- iscsi_tcp.[ch] (iSCSI transport over TCP/IP).
Patch 2:
Common header files:
- iscsi_if.h (iSCSI open interface over netlink);
- iscsi_proto.h (RFC3720 #defines and types);
- iscsi_ifev.h (user/kernel events).
Patch 3:
drivers/scsi/Kconfig changes.
Patch 4:
drivers/scsi/Makefile changes.
Patch 5:
include/linux/netlink.h changes (added new protocol NETLINK_ISCSI)
Patch 6:
Documentation/scsi/iscsi.txt
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/