Generally, there are lots of dependencies between the driver versions
and the kernel itself. If you were to break them out, you would have
lots of problems with people trying to not grab the latest stuff, just
upgrading the drivers, etc. The kerenel needs to be distributed as a
definite version. There are enough problems with userland programs
breaking when the kernel changes something. I don't think you want to
make that worse by making it possible for drivers to have the same
problems.
Just my two cents. The linux kernel isn't that big.
Warner