As the title implies we subjected 2.1.125 to a production load test
this morning and had some difficulties.
The machine in question is:
dual PII-300 Intel FX motherboard.
256 Megabyte RAM
DPT-3334UW Raid controller driving 4 x 9Gig drives (RAID5)
2 SMC Etherpower II (shared IRQ 11)
The only change to the stock 2.1.125 sources was to substitute v1.04
of Donald's driver for the Etherpower II. Alan had suggested in
private e-mail that the most current driver should prevent the wedging
that we were seeing with these cards in 2.0.35/SMP. Kernel was
statically compiled and very minimal. I can forward a specific
.config if this is necessary.
The first problem we noticed was when the networking was configured.
We received back an error message of "kmalloc too large (xxxxxxx)"
with some huge value that we didn't get copied for xxxxxx.
Performance was otherwise good as quotacheck was run and the system
initialized.
The server in question is the IMAP server for the universities
messaging system. It has moderate to heavy traffic on eth0 and
generally light traffic on eth1 unless backups are being run, which
of course was not occurring. The user home directories sit in a 20
Gigabyte ext2 filesystem which is the only filesystem that has quotas
enabled for. There was both a quota.group and a quota.user file in
/home but only user quotas were turned on.
After about 20 minutes of uptime and with low load the NIC card
supporting eth0 wedged itself. This behavior was identical to what we
have been seeing with 2.0.35 and the v1.03 driver. The secondary
interface (eth1) was fully functional. Downing and upping eth0 would
not free the interface. Downing and upping both interfaces resolved
the lock.
Operation was fine for the first hour but we did receive the following
string of debug messages:
---------------------------------------------------------------------------
Oct 14 07:00:47 pobox1 kernel: Socket destroy delayed (r=0 w=160)
Oct 14 07:11:52 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 07:12:47 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 07:13:57 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 07:19:00 pobox1 kernel: Socket destroy delayed (r=0 w=160)
Oct 14 07:32:00 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 07:38:13 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 07:39:49 pobox1 kernel: Socket destroy delayed (r=0 w=160)
Oct 14 07:43:47 pobox1 kernel: Socket destroy delayed (r=0 w=320)
Oct 14 07:47:42 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 07:48:09 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 07:51:27 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 07:52:05 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 07:55:12 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 07:55:34 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 07:56:14 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 08:01:54 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:02:03 pobox1 kernel: Socket destroy delayed (r=0 w=160)
Oct 14 08:07:47 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:12:41 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:15:00 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:15:00 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:15:52 pobox1 kernel: Socket destroy delayed (r=0 w=320)
Oct 14 08:17:02 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:18:37 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 08:21:06 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:23:35 pobox1 kernel: Socket destroy delayed (r=0 w=160)
Oct 14 08:25:07 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:26:24 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:26:40 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:29:07 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:29:09 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 08:29:58 pobox1 kernel: Socket destroy delayed (r=0 w=160)
Oct 14 08:32:23 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:35:16 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:36:15 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:37:00 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:41:07 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:41:35 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:42:01 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 08:42:18 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:42:43 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:43:22 pobox1 kernel: Socket destroy delayed (r=0 w=160)
Oct 14 08:47:41 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:48:00 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:49:03 pobox1 kernel: Socket destroy delayed (r=0 w=112)
---------------------------------------------------------------------------
At about 08:45 we experienced our first serious problem when users
began reporting that they were unable to establish IMAP connections to
the server. At this time the following string of debug messages
appeared:
---------------------------------------------------------------------------
Oct 14 08:49:06 pobox1 kernel: get_empty_dquot: pruning 128
Oct 14 08:50:02 pobox1 kernel: get_empty_dquot: pruning 128
Oct 14 08:50:05 pobox1 kernel: get_empty_dquot: pruning 136
Oct 14 08:50:17 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:50:28 pobox1 kernel: Socket destroy delayed (r=0 w=160)
Oct 14 08:52:40 pobox1 kernel: get_empty_dquot: pruning 182
Oct 14 08:52:48 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 08:53:13 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:54:11 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:54:39 pobox1 kernel: get_empty_dquot: pruning 284
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 353
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 370
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 387
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 365
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 414
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 211
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 105
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 56
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 28
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 13
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:54:41 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:54:45 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 08:54:45 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:54:45 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:54:45 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:54:46 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 08:54:46 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:54:46 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:54:46 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:55:01 pobox1 kernel: get_empty_dquot: pruning 15
Oct 14 08:55:01 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 08:55:01 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:55:01 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:55:01 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:55:01 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:55:04 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 08:55:04 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:55:04 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:55:04 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:55:15 pobox1 kernel: get_empty_dquot: pruning 9
Oct 14 08:55:15 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:55:15 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:55:15 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:55:15 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:55:20 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:55:25 pobox1 kernel: get_empty_dquot: pruning 9
Oct 14 08:55:25 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 08:55:25 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:55:25 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:55:25 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:55:28 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:55:28 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:55:28 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:55:33 pobox1 kernel: get_empty_dquot: pruning 9
Oct 14 08:55:33 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 08:55:33 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:55:33 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:55:33 pobox1 last message repeated 2 times
Oct 14 08:55:39 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 08:55:39 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:55:39 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:56:13 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:56:43 pobox1 kernel: get_empty_dquot: pruning 23
Oct 14 08:56:43 pobox1 kernel: get_empty_dquot: pruning 16
Oct 14 08:56:43 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 08:56:43 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:56:43 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:56:43 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:57:53 pobox1 kernel: get_empty_dquot: pruning 25
Oct 14 08:57:53 pobox1 kernel: get_empty_dquot: pruning 13
Oct 14 08:57:53 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 08:57:53 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:57:53 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:57:53 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:57:54 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 08:57:54 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:57:54 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:57:54 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:57:54 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:57:54 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:57:54 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:57:54 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:58:13 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 08:58:13 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:58:13 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:58:13 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:58:13 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:58:13 pobox1 last message repeated 2 times
Oct 14 08:58:14 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 08:58:14 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:58:14 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:58:14 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:58:14 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:58:14 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:58:21 pobox1 kernel: get_empty_dquot: pruning 16
Oct 14 08:58:21 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 08:58:21 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:58:21 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:58:21 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:58:22 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 08:58:22 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:58:22 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:58:22 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:58:36 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 08:58:36 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:58:36 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:58:36 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:58:39 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 08:58:39 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:58:39 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:58:39 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:58:50 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 08:58:50 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:58:50 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:58:50 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:59:06 pobox1 kernel: get_empty_dquot: pruning 9
Oct 14 08:59:06 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 08:59:06 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:59:06 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:59:06 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:59:23 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 08:59:23 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:59:23 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:59:23 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:59:27 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 08:59:28 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 08:59:28 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 08:59:28 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:59:28 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:59:28 pobox1 last message repeated 2 times
Oct 14 08:59:38 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 08:59:38 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 08:59:38 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 08:59:38 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 08:59:40 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 08:59:41 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 08:59:41 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:00:20 pobox1 kernel: get_empty_dquot: pruning 10
Oct 14 09:00:20 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 09:00:20 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:00:20 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:00:26 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 09:00:26 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:00:36 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 09:00:38 pobox1 kernel: get_empty_dquot: pruning 9
Oct 14 09:00:38 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 09:00:38 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:00:38 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:00:38 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:00:38 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:00:40 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 09:00:45 pobox1 kernel: get_empty_dquot: pruning 9
Oct 14 09:00:45 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 09:00:45 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:00:45 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:00:45 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:00:47 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 09:00:47 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:00:47 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:00:47 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:00:57 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 09:00:57 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 09:00:57 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:01:08 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 09:01:10 pobox1 kernel: get_empty_dquot: pruning 10
Oct 14 09:01:10 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 09:01:10 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:01:10 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:01:10 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:01:10 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:01:35 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 09:01:35 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:01:35 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:01:35 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:01:41 pobox1 kernel: get_empty_dquot: pruning 10
Oct 14 09:01:41 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 09:01:41 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:02:00 pobox1 kernel: Socket destroy delayed (r=0 w=160)
Oct 14 09:02:07 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 09:02:13 pobox1 kernel: get_empty_dquot: pruning 9
Oct 14 09:02:13 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 09:02:13 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:02:13 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:02:13 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:13 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:15 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 09:02:15 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:02:15 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:15 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:17 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 09:02:20 pobox1 kernel: get_empty_dquot: pruning 12
Oct 14 09:02:20 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 09:02:20 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:02:20 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:20 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:22 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 09:02:22 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:02:22 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:22 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:23 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 09:02:25 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 09:02:25 pobox1 kernel: Socket destroy delayed (r=0 w=240)
Oct 14 09:02:28 pobox1 kernel: get_empty_dquot: pruning 13
Oct 14 09:02:28 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 09:02:28 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 09:02:28 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:02:28 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:29 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 09:02:29 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:02:29 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:29 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:33 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 09:02:33 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 09:02:33 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:02:33 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:47 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 09:02:47 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 09:02:47 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:02:47 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:47 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:50 pobox1 kernel: get_empty_dquot: pruning 7
Oct 14 09:02:50 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 09:02:50 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:02:50 pobox1 last message repeated 3 times
Oct 14 09:03:01 pobox1 kernel: Socket destroy delayed (r=0 w=96)
Oct 14 09:03:01 pobox1 kernel: get_empty_dquot: pruning 8
Oct 14 09:03:01 pobox1 kernel: get_empty_dquot: pruning 5
Oct 14 09:03:01 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:03:01 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:03:03 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 09:03:03 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:03:03 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:03:03 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:03:07 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 09:03:09 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 09:03:09 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:03:09 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:03:26 pobox1 kernel: get_empty_dquot: pruning 12
Oct 14 09:03:26 pobox1 kernel: get_empty_dquot: pruning 6
Oct 14 09:03:26 pobox1 kernel: get_empty_dquot: pruning 4
Oct 14 09:03:26 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:03:26 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:03:26 pobox1 kernel: get_empty_dquot: pruning 1
Oct 14 09:03:36 pobox1 kernel: Socket destroy delayed (r=0 w=112)
Oct 14 09:04:03 pobox1 kernel: get_empty_dquot: pruning 20
Oct 14 09:04:03 pobox1 kernel: get_empty_dquot: pruning 9
Oct 14 09:04:03 pobox1 kernel: get_empty_dquot: pruning 3
Oct 14 09:04:03 pobox1 kernel: get_empty_dquot: pruning 2
Oct 14 09:04:03 pobox1 kernel: get_empty_dquot: pruning 1
---------------------------------------------------------------------------
During this time period the following syslog messages were generated.
These would seem to be of more concern:
---------------------------------------------------------------------------
Oct 14 08:54:41 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 08:56:43 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 08:58:13 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 08:59:06 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 09:00:45 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 09:01:10 pobox1 kernel: VFS: No free dquots,contact mvw@planets.elm.net
Oct 14 09:01:35 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 09:01:41 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 09:02:13 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 09:02:15 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 09:02:20 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
Oct 14 09:04:03 pobox1 kernel: VFS: No free dquots, contact mvw@planets.elm.net
---------------------------------------------------------------------------
About 5-10 minutes later we discovered that the machine was
essentially non-functional with respect to anything involving quotas.
The load average was approximately 20 and climbed to 75 before we were
forced to reboot the machine.
I am not sure that this was a true load average of 75 since the system
was still reasonably responsive from the console. There were a large
number of 'stuck' IMAP process in a state of 'dpget'. When we tried
to run a quota command on a user the quota command hung as well.
A string of error messages were emitted when the machine was rebooted
which we did not capture. Fsck reported the file systems as being
dirty suggesting that the error messages may have been secondary to a
failure to unmount the filesystems cleanly.
A fair number of the following messages were reported by fsck:
Deleted inode has zero dtime.
Unattached zero-length inode cleared.
Otherwise the filesystems were intact and everything started up just
fine under 2.0.35.
I apologize for the length of this commentary but I wanted to make
sure that I had complete information for everyone. This server
typically has around 150 simultaneous IMAP users and we hit loads of
8-10 during sendmail runs so my hopes are that this serves as a
reasonable test for a 'production' environment.
Hopefully this is not a situation of horrendous user error, in which
case I will be suitably chagrined.
I am now reasonably convinced that there are problems with the SMC
Etherpower driver in the SMP environment. We have a 233Mhz PII
UNI-2.0.35 machine with the same card hooked into the same shared hub
which has been routing about 500 megabytes of mail a day for 33 days
without as much as a peep out of it.
Thats about it. Have a pleasant remainder of the week everyone.
Greg
As always,
Dr. G.W. Wettstein Enjellic Systems Development - Specialists in
4206 N. 19th Ave. intranet based enterprise information solutions.
Fargo, ND 58102 WWW: http://www.enjellic.com
Phone: 701-281-1686 EMAIL: greg@wind.enjellic.com
------------------------------------------------------------------------------
"We restored the user's real .pinerc from backup but another of our users
must still be missing those cows."
-- Malcolm Beattie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/