The syslogs (on another machine logged over the network, so they were
logged despite the fs problems) read:
Jan 4 05:49:16 loki -- MARK --
Jan 4 05:49:33 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1914976
Jan 4 05:49:33 loki last message repeated 2 times
Jan 4 05:49:33 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1916900
Jan 4 05:49:33 loki last message repeated 2 times
Jan 4 05:49:33 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1918824
Jan 4 05:49:33 loki last message repeated 2 times
Jan 4 05:49:33 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 469769535
Jan 4 05:49:33 loki last message repeated 2 times
Jan 4 05:49:33 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1476402481
Jan 4 05:49:33 loki last message repeated 2 times
Jan 4 05:49:44 loki kernel: EXT2-fs error (device 08:0a): ext2_read_inode: bad inode number: 1275075874
Jan 4 05:49:44 loki last message repeated 2 times
Jan 4 05:49:46 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 469769527
Jan 4 05:49:47 loki last message repeated 2 times
Jan 4 05:49:54 loki kernel: EXT2-fs error (device 08:0a): ext2_read_inode: bad inode number: 1006640445
Jan 4 05:49:54 loki last message repeated 2 times
Jan 4 05:49:54 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1916692
Jan 4 05:49:54 loki last message repeated 2 times
Jan 4 05:49:54 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1914664
Jan 4 05:49:54 loki last message repeated 2 times
Jan 4 05:49:59 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1916744
Jan 4 05:50:00 loki last message repeated 2 times
Jan 4 05:50:00 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1916016
Jan 4 05:50:01 loki last message repeated 2 times
Jan 4 05:50:01 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1914352
Jan 4 05:50:01 loki last message repeated 2 times
Jan 4 05:50:01 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1917836
Jan 4 05:50:01 loki last message repeated 2 times
Jan 4 05:50:01 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1919240
Jan 4 05:50:01 loki last message repeated 2 times
Jan 4 05:50:11 loki kernel: EXT2-fs error (device 08:09): ext2_read_inode: bad inode number: 1919364
Jan 4 05:50:11 loki last message repeated 2 times
Jan 4 05:50:14 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 469769536
Jan 4 05:50:14 loki last message repeated 2 times
Jan 4 05:50:21 loki kernel: EXT2-fs error (device 08:09): ext2_read_inode: bad inode number: 1918200
Jan 4 05:50:21 loki last message repeated 2 times
Jan 4 05:50:40 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1476402500
Jan 4 05:50:40 loki last message repeated 2 times
Jan 4 05:50:40 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1476402503
Jan 4 05:50:40 loki last message repeated 2 times
Jan 4 05:50:40 loki kernel: EXT2-fs error (device 08:09): ext2_read_inode: bad inode number: 577765405
Jan 4 05:50:40 loki last message repeated 2 times
Jan 4 05:51:35 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 469769542
Jan 4 05:51:35 loki last message repeated 2 times
Jan 4 05:52:11 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 469769540
Jan 4 05:52:11 loki last message repeated 2 times
Jan 4 05:53:10 loki kernel: EXT2-fs error (device 08:09): ext2_read_inode: bad inode number: 184556867
Jan 4 05:53:10 loki last message repeated 2 times
Jan 4 05:53:40 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1917992
Jan 4 05:53:40 loki last message repeated 2 times
Jan 4 05:53:40 loki kernel: EXT2-fs error (device 08:09): ext2_read_inode: bad inode number: 1140064285
Jan 4 05:53:40 loki last message repeated 2 times
Jan 4 05:53:52 musashi -- MARK --
Jan 4 05:54:13 loki kernel: EXT2-fs error (device 08:09): ext2_read_inode: bad inode number: 1912376
Jan 4 05:54:13 loki last message repeated 2 times
Jan 4 05:55:01 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1917108
Jan 4 05:55:01 loki last message repeated 2 times
Jan 4 05:55:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_inode: bit already cleared for inode 2055
Jan 4 05:55:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_inode: bit already cleared for inode 2055
Jan 4 05:56:10 loki kernel: EXT2-fs error (device 08:09): ext2_read_inode: bad inode number: 184556866
Jan 4 05:56:10 loki last message repeated 2 times
Jan 4 05:57:13 loki kernel: EXT2-fs error (device 08:09): ext2_read_inode: bad inode number: 1907904
Jan 4 05:57:14 loki last message repeated 2 times
Jan 4 05:57:14 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 469769541
Jan 4 05:57:14 loki last message repeated 2 times
Jan 4 05:58:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_inode: bit already cleared for inode 2058
Jan 4 05:58:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_inode: bit already cleared for inode 2058
Jan 4 06:01:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_blocks: bit already cleared for block 5169
Jan 4 06:01:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_blocks: bit already cleared for block 5169
Jan 4 06:01:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_inode: bit already cleared for inode 2058
Jan 4 06:01:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_inode: bit already cleared for inode 2058
Jan 4 06:02:11 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 469769523
Jan 4 06:02:11 loki last message repeated 2 times
Jan 4 06:04:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_blocks: bit already cleared for block 5169
Jan 4 06:04:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_blocks: bit already cleared for block 5169
Jan 4 06:04:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_inode: bit already cleared for inode 2058
Jan 4 06:04:18 loki kernel: EXT2-fs warning (device 08:09): ext2_free_inode: bit already cleared for inode 2058
Henceforward it settled down to reports at semi-random intervals of
the cleared state of block 5169 and inode 2058.
08:09 (/dev/sda9) is /var; inode 2058 is no longer allocated to
anything, so likely it was /var/run/utmp, or something very similar.
When I noticed the problem and started prodding the machine the errors changed again:
Jan 5 00:55:48 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 469769531
Jan 5 00:55:48 loki last message repeated 2 times
Jan 5 00:56:05 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 491402240
Jan 5 00:56:05 loki last message repeated 2 times
Jan 5 00:56:05 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 2751470874
Jan 5 00:56:05 loki last message repeated 2 times
Jan 5 00:56:05 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 3422559546
Jan 5 00:56:05 loki last message repeated 2 times
Jan 5 00:56:05 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 3690994995
Jan 5 00:56:05 loki last message repeated 2 times
Jan 5 00:56:06 loki kernel: EXT2-fs error (device 08:06): ext2_read_inode: bad inode number: 1140858163
Jan 5 00:56:06 loki last message repeated 2 times
I think (but am not sure) that one of these corresponded to each
attempt of mine to run some binary (yielding -ENOTDIR each time).
Something odd is going on here. /var fscked as OK after a reboot, so
it's not the filesystem. Did the ext2fs driver simply confuse itself
unto death?
The relevant .config is
CONFIG_EXPERIMENTAL=y
CONFIG_MODULES=y
# CONFIG_MODVERSIONS is not set
CONFIG_KERNELD=y
# CONFIG_MATH_EMULATION is not set
CONFIG_NET=y
# CONFIG_MAX_16M is not set
CONFIG_PCI=y
CONFIG_PCI_OPTIMIZE=y
CONFIG_SYSVIPC=y
# CONFIG_BINFMT_AOUT is not set
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_JAVA=m
CONFIG_KERNEL_ELF=y
# CONFIG_M386 is not set
# CONFIG_M486 is not set
CONFIG_M586=y
# CONFIG_M686 is not set
# CONFIG_MTRR is not set
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_IDE is not set
# CONFIG_BLK_DEV_HD_ONLY is not set
CONFIG_BLK_DEV_LOOP=y
# CONFIG_BLK_DEV_MD is not set
# CONFIG_BLK_DEV_RAM is not set
# CONFIG_BLK_DEV_XD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_DEV_HD is not set
# CONFIG_FIREWALL is not set
# CONFIG_NET_ALIAS is not set
CONFIG_INET=y
# CONFIG_IP_FORWARD is not set
# CONFIG_IP_MULTICAST is not set
CONFIG_SYN_COOKIES=y
# CONFIG_IP_ACCT is not set
# CONFIG_IP_ROUTER is not set
# CONFIG_NET_IPIP is not set
# CONFIG_INET_PCTCP is not set
# CONFIG_INET_RARP is not set
# CONFIG_NO_PATH_MTU_DISCOVERY is not set
CONFIG_IP_NOSR=y
CONFIG_SKB_LARGE=y
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_AX25 is not set
# CONFIG_BRIDGE is not set
# CONFIG_NETLINK is not set
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
# CONFIG_SCSI_MULTI_LUN is not set
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AHA1740 is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_SCSI_AM53C974 is not set
# CONFIG_SCSI_MEGARAID is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA_DMA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_NCR53C7xx is not set
CONFIG_SCSI_NCR53C8XX=y
CONFIG_SCSI_NCR53C8XX_NVRAM_DETECT=y
CONFIG_SCSI_NCR53C8XX_TAGGED_QUEUE=y
# CONFIG_SCSI_NCR53C8XX_IOMAPPED is not set
CONFIG_SCSI_NCR53C8XX_MAX_TAGS=8
CONFIG_SCSI_NCR53C8XX_SYNC=10
CONFIG_SCSI_NCR53C8XX_SYMBIOS_COMPAT=y
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_SEAGATE is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_GDTH is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_EQUALIZER is not set
# CONFIG_DLCI is not set
CONFIG_PLIP=m
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_RADIO is not set
CONFIG_NET_ETHERNET=y
CONFIG_NET_VENDOR_3COM=y
# CONFIG_EL1 is not set
# CONFIG_EL2 is not set
# CONFIG_ELPLUS is not set
# CONFIG_EL16 is not set
# CONFIG_EL3 is not set
# CONFIG_3C515 is not set
CONFIG_VORTEX=y
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_PCI is not set
# CONFIG_NET_ISA is not set
# CONFIG_NET_EISA is not set
# CONFIG_NET_POCKET is not set
# CONFIG_TR is not set
# CONFIG_FDDI is not set
# CONFIG_ARCNET is not set
# CONFIG_SHAPER is not set
# CONFIG_RCPCI is not set
# CONFIG_ISDN is not set
# CONFIG_CD_NO_IDESCSI is not set
CONFIG_QUOTA=y
CONFIG_MINIX_FS=m
# CONFIG_EXT_FS is not set
CONFIG_EXT2_FS=y
# CONFIG_XIA_FS is not set
CONFIG_NLS=y
CONFIG_ISO9660_FS=m
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_UMSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=m
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
CONFIG_NLS_ISO8859_1=m
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_KOI8_R is not set
CONFIG_PROC_FS=y
CONFIG_NFS_FS=y
# CONFIG_ROOT_NFS is not set
# CONFIG_SMB_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_SERIAL=y
# CONFIG_DIGI is not set
# CONFIG_CYCLADES is not set
# CONFIG_ISI is not set
# CONFIG_STALDRV is not set
# CONFIG_RISCOM8 is not set
CONFIG_PRINTER=m
# CONFIG_SPECIALIX is not set
CONFIG_MOUSE=y
# CONFIG_ATIXL_BUSMOUSE is not set
# CONFIG_BUSMOUSE is not set
# CONFIG_MS_BUSMOUSE is not set
CONFIG_PSMOUSE=y
# CONFIG_82C710_MOUSE is not set
# CONFIG_UMISC is not set
# CONFIG_QIC02_TAPE is not set
# CONFIG_FTAPE is not set
CONFIG_APM=y
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
# CONFIG_APM_DO_ENABLE is not set
# CONFIG_APM_CPU_IDLE is not set
# CONFIG_APM_DISPLAY_BLANK is not set
CONFIG_APM_POWER_OFF=y
# CONFIG_APM_IGNORE_MULTIPLE_SUSPEND is not set
# CONFIG_WATCHDOG is not set
CONFIG_RTC=y
CONFIG_SOUND=m
# CONFIG_PAS is not set
CONFIG_SB=y
CONFIG_ADLIB=y
# CONFIG_GUS is not set
# CONFIG_MPU401 is not set
# CONFIG_PSS is not set
# CONFIG_GUS16 is not set
# CONFIG_GUSMAX is not set
# CONFIG_MSS is not set
# CONFIG_SSCAPE is not set
# CONFIG_TRIX is not set
# CONFIG_MAD16 is not set
# CONFIG_CS4232 is not set
# CONFIG_MAUI is not set
# CONFIG_OPL3SA1 is not set
# CONFIG_SOFTOSS is not set
CONFIG_YM3812=y
SBC_BASE=220
SBC_IRQ=5
SBC_DMA=1
SB_DMA2=5
SB_MPU_BASE=330
SB_MPU_IRQ=-1
CONFIG_LOWLEVEL_SOUND=y
# CONFIG_ACI_MIXER is not set
CONFIG_AWE32_SYNTH=y
# CONFIG_AEDSP16 is not set
# CONFIG_PROFILE is not set
Modules loaded would have been lm_sensors-1.3.6, the ossfree-3.8 sound
module, and the NLS cp437 module.
Any ideas?
-- `Anyone who says you can have a lot of widely dispersed people hack away on a complicated piece of code and avoid total anarchy has never managed a software project.' - Andy Tanenbaum in 1992 on comp.os.minix- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/