I haven't checked the buffer cache code lately, but in the past it
didn't deal well with having a mix of differently sized buffers.
That was precisely the hypothesis I gave as well.
It should be possible to make 1kb filesystem to have nearly all of the
performance advantages of the 4kb filesystems, while still retaining
their disk space efficiency advantages. That isn't the case today, but
hopefully we can fix that.
I've been hearing this answer ever since the question first came up quite a
while ago. Looking back to some of the earliest measurements I made comparing
1KB and 4KB file systems, it looks like there hasn't been much change in the
last two years.
Leonard