That doesn't sound good. Why did you not mention this first, it's unlikely
that such a compiler produces a working kernel. When the segfaults
are not deterministic (go away when you try again) then you likely
have some hardware problem, like bad DIMMs (run memtest86 for 12+hours to
make sure)
To rule out the compiler you can use the compiler/binutils fromOK I'll try that.
ftp.suse.com:/pub/suse/x86-64/supplementary/CrossTools/8.1-i386/
That's rpms for SuSE 8.1/i386, but I suspect you install it on Debian with
rpm2cpio or somesuch. That's an older gcc 3.2 that is known to work.
Then just put /opt/cross/bin in your $PATH and compile with
CROSS_COMPILE=x86_64-linux- ARCH=x86_64