Re: Apache performance: Run queue proportional to number of connections?

Philip Gladstone (philip@raptor.com)
Mon, 28 Jun 1999 10:21:54 -0400


This is a cryptographically signed message in MIME format.

--------------ms41362BD1A3A3E472CD9BB7F8
Content-Type: multipart/mixed;
boundary="------------46585E83FDEC59C37A84029D"

This is a multi-part message in MIME format.
--------------46585E83FDEC59C37A84029D
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Various people wrote:
> > > Why you raise the finger against interrupts? Is the problem the irq
> > > lantency? It's true that we call irq handlers through two function
> > > pointers, we could remove such function pointers but I am not convinced
> > > that that's the problem. I think also NT may have something similar to
> > > allow the code to be object oriented and cleaner.
> >
> > the problem was the single lock for being in the tcp paths. we'd have N
> > apaches running on the cpus trying to transmit the data they just figured
> > out the client wants, and N interrupts arriving at the same time trying to
> > pass more data in to the waiting apaches. sync_bh was WAY at the top of
> > the profiling runs.

I made a number of minor performance changes back in the 2.0.xx days
that may still be applicable:

1) Eliminate the dummy inb instruction when acking interrupts to the
8259. The comments indicate that these are present to deal with
some old disk controller problem, and probably a flaky 8259!

2) Change the ethernet drivers to be more parsimonious in their use
of i/o instructions in their interrupt routines. There are two
changes here -- one is not to acknowledge interrupts if there
are none to be acknowledged. The second is not to check for more
working arriving while processing the first chunk of work.
[This second change should be adaptive.]

The deal is that i/o instructions are *incredibly* expensive on
modern fast processors. A simple inb takes between 500 and 1500 ns
depending on the device (on a 350Mhz PII). My suspicion is that this
is bus limited, so it will be no fast on a 550 PIII. This means that
it is worth spending significant code to avoid I/O (hundreds of
normal instructions).

Why do I bring it up? Essentially these changes can eliminate 3
i/o instructions per interrupt, at around 2 usecs per interrupt.
Will this help -- I don't know, it depends on the actual interrupt
rates in the benchmark above.

For those of you skeptics who doubt me, I attach a simple test
program that does a bunch of inb/inw/inl to the port of your
choice, and prints out the speeds. Make sure that you compile
it with -O2, otherwise you get link errors.

Philip

-- 
Philip Gladstone                           +1 781 530 2461
Axent Technologies, Waltham, MA
--------------46585E83FDEC59C37A84029D
Content-Type: application/x-unknown-content-type-cfile;
 name="ioperf.c"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
 filename="ioperf.c"

I2luY2x1ZGUgPHN0ZGlvLmg+CiNpbmNsdWRlIDxzeXMvdGltZS5oPgojaW5jbHVkZSA8YXNt L2lvLmg+CgojZGVmaW5lIExPT1BTUEVSTlMgICAgICAxMDAwCgp2b2lkCnJlcG9ydChjaGFy ICpzdHIsIGNoYXIgKnBvcnQsIHN0cnVjdCB0aW1ldmFsICp0X3N0YXJ0LCBzdHJ1Y3QgdGlt ZXZhbCAqdF9lbmQpCnsKICAgIHVuc2lnbmVkIGxvbmcgZGlmZiA9ICh0X2VuZC0+dHZfc2Vj IC0gdF9zdGFydC0+dHZfc2VjKSAqIDEwMDAwMDAgKwogICAgICAgICAgICAgICAgICAgICAg ICAodF9lbmQtPnR2X3VzZWMgLSB0X3N0YXJ0LT50dl91c2VjKTsKCiAgICBwcmludGYoIiVz KCVzKSB0YWtlcyBhcHByb3ggJWQgbnMgKCVkIFBDSSBjeWNsZXMpXG4iLCBzdHIsIHBvcnQs IGRpZmYgLyBMT09QU1BFUk5TLCAoZGlmZiAqIDMzKSAvIChMT09QU1BFUk5TICogMTAwMCkp Owp9CgojZGVmaW5lIFJVTl9URVNUKGlucykgZG8geyBcCiAgICBnZXR0aW1lb2ZkYXkoJnRf c3RhcnQsIDApOyBcCiAgICBmb3IgKGkgPSAwOyBpIDwgMTAwMCAqIExPT1BTUEVSTlM7IGkr KykgXAogICAgICAgICh2b2lkKSBpbnMocG9ydCk7IFwKICAgIGdldHRpbWVvZmRheSgmdF9l bmQsIDApOyBcCiAgICByZXBvcnQoI2lucywgYXJndlsxXSwgJnRfc3RhcnQsICZ0X2VuZCk7 IH0gd2hpbGUgKDApCgppbnQKbWFpbihpbnQgYXJnYywgY2hhciAqYXJndltdKSAKewogICAg c3RydWN0IHRpbWV2YWwgdF9zdGFydCwgdF9lbmQ7CiAgICBpbnQgcG9ydDsKICAgIGludCBp OwoKICAgIGlmIChhcmdjIDwgMikgewogICAgICAgIGZwcmludGYoc3RkZXJyLCAiTXVzdCBn aXZlIHBvcnQgbnVtYmVyXG4iKTsKICAgICAgICByZXR1cm4gMTsKICAgIH0KCiAgICBwb3J0 ID0gc3RydG9sKGFyZ3ZbMV0sIDAsIDApOwoKICAgIGlmIChwb3J0ID4gMHgzZmYgfHwgMSkg CiAgICAgICAgaW9wbCgzKTsKICAgIGVsc2UKICAgICAgICBpb3Blcm0ocG9ydCwgNCwgMSk7 CgogICAgUlVOX1RFU1QoKHZvaWQpKTsKCiAgICBSVU5fVEVTVChpbmIpOwogICAgaWYgKChw b3J0ICYgMSkgPT0gMCkKICAgICAgICBSVU5fVEVTVChpbncpOwogICAgaWYgKChwb3J0ICYg MykgPT0gMCkKICAgICAgICBSVU5fVEVTVChpbmwpOwp9Cg== --------------46585E83FDEC59C37A84029D--

--------------ms41362BD1A3A3E472CD9BB7F8 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature

MIIIaQYJKoZIhvcNAQcCoIIIWjCCCFYCAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCC Bi0wggLsMIICVaADAgECAgMArf4wDQYJKoZIhvcNAQEEBQAwgbkxCzAJBgNVBAYTAlpBMRUw EwYDVQQIEwxXZXN0ZXJuIENhcGUxFDASBgNVBAcTC0R1cmJhbnZpbGxlMRowGAYDVQQKExFU aGF3dGUgQ29uc3VsdGluZzEpMCcGA1UECxMgVGhhd3RlIFBGIFJTQSBJSyAxOTk4LjkuMTYg MTc6NTUxNjA0BgNVBAMTLVRoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBSU0EgSXNzdWVyIDE5 OTguOS4xNjAeFw05OTAzMjQxODMyNTVaFw0wMDAzMjMxODMyNTVaMGYxHzAdBgNVBAMTFlRo YXd0ZSBGcmVlbWFpbCBNZW1iZXIxIDAeBgkqhkiG9w0BCQEWEXBoaWxpcEByYXB0b3IuY29t MSEwHwYJKoZIhvcNAQkBFhJwanNnQGl4Lm5ldGNvbS5jb20wgZ8wDQYJKoZIhvcNAQEBBQAD gY0AMIGJAoGBAMQ3YtRscyyQn2dK9Z52v2lHCoX33ym6m1yOkIDaeBPVAL9BVkSMeroFO4hK p2Xi72zgGOkm+amhY/N06NfM4RcL61QlbSpRRyiMuUpU2rIdDtSLSpwEoDyzzju83iIclf4A OwFEPmY5+lbwwMUdZXnoatPZwAyAlkU+lTGPIBUxAgMBAAGjVDBSMBEGCWCGSAGG+EIBAQQE AwIFoDAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBT+PmCca4wP sNgzxsrGHliwcTi14DANBgkqhkiG9w0BAQQFAAOBgQBZemjNn+zoyQ45PhztrBoepNW2tSi4 0MdfBblRjen40gB2H9/XvPTcFRmrC2mRzzHo3vTrwYibNcqXiiAAo2yg4WVUBlQuaxSJ89Ds FoM08CbKzmfGAxJS+87cwvDU9pB857YcO355q/6rAhOgPD6BHquPjA0sr+TvvxvHDYFulDCC AzkwggKioAMCAQICAQowDQYJKoZIhvcNAQEEBQAwgdExCzAJBgNVBAYTAlpBMRUwEwYDVQQI EwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEaMBgGA1UEChMRVGhhd3RlIENv bnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2VydmljZXMgRGl2aXNpb24xJDAi BgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkGCSqGSIb3DQEJARYccGVy c29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw05ODA5MTYxNzU1MzRaFw0wMDA5MTUxNzU1 MzRaMIG5MQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBDYXBlMRQwEgYDVQQHEwtE dXJiYW52aWxsZTEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKTAnBgNVBAsTIFRoYXd0 ZSBQRiBSU0EgSUsgMTk5OC45LjE2IDE3OjU1MTYwNAYDVQQDEy1UaGF3dGUgUGVyc29uYWwg RnJlZW1haWwgUlNBIElzc3VlciAxOTk4LjkuMTYwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJ AoGBAMSl5dTU0F8IAu4HIX0kv6trjh7rIAcCFYRrj9CTJB8bne5osrksT+mTZxcQFx6h+UNB I7kwqnaXu/Pn/YHAtTGL9qZQJlTylSjrGaQelx6w4ribwQSaMtA8CWxP5DVP8Ha/ABMDT0UI YPP8tNCQAYoSyZy6f1LqKpM1Njw85DUvAgMBAAGjNzA1MBIGA1UdEwEB/wQIMAYBAf8CAQAw HwYDVR0jBBgwFoAUcknCczTGVfQLdnKBfnf0h+fGsg4wDQYJKoZIhvcNAQEEBQADgYEALMeC HwFDPgeP7mlcqWSC+MCWrZMry5tQ10CagcK6pnadPJVA3FXB4VWCeasKKabVDOFXKD6P+bvV 3w2TWKpbLYuPM+TdWBU1dnIVKb1C9FqSC3dfnSfbmi1OG4IGjtKNVruV3tsMZQXelZ4C3VMX vr78a8MaInoUK2G9wp9eeloxggIEMIICAAIBATCBwTCBuTELMAkGA1UEBhMCWkExFTATBgNV BAgTDFdlc3Rlcm4gQ2FwZTEUMBIGA1UEBxMLRHVyYmFudmlsbGUxGjAYBgNVBAoTEVRoYXd0 ZSBDb25zdWx0aW5nMSkwJwYDVQQLEyBUaGF3dGUgUEYgUlNBIElLIDE5OTguOS4xNiAxNzo1 NTE2MDQGA1UEAxMtVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIFJTQSBJc3N1ZXIgMTk5OC45 LjE2AgMArf4wCQYFKw4DAhoFAKCBmTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqG SIb3DQEJBTEPFw05OTA2MjgxNDIxNTVaMCMGCSqGSIb3DQEJBDEWBBQoK5CxpvQzbYasMgYG eMgEBb4ouDA6BgkqhkiG9w0BCQ8xLTArMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDAN BggqhkiG9w0DAgIBQDANBgkqhkiG9w0BAQEFAASBgAAkXDNJkKlnVf7NXw215SsTPKh48t2t A8qmKgbE90R7+2IfBuJm+jQJzxOPZQTD5Ds2pWC9+pkrmlV+Qlh3DRXQz+ntyEz95M13tyw/ 2zZ4oZoBsddG+0JoolSgjp0LJ59uhDF/poVQlp8m4YPSV27JgKZcwewUctUmMmJsIafG --------------ms41362BD1A3A3E472CD9BB7F8--

- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/