Well, what is the kernel? Obviously the zImage, but:
Loadable modules
Process using mlock()
Kernel threads, like nfsiod and kswapd
Process using/abusing real time scheduler
Swappable kernel (someone has a patch)
Process (?) running in ring 0
IRQ handler in a process (mlocked, of course!)
Process mapped with physical addresses
Process that provides a system call
Process with contiguous memory for DMA (but perhaps not identity mapped)
You could give an X server full control over VCs, and even unload the
kernel vt102/select/VGA/scrollback/keyboard support. This moves in
the microkernel direction I believe. It would _sometimes_ help the
kernel oops problem with graphics mode. It would help speed if local
X clients could use system calls for common operations. That would
be somewhat gross, but it would be secure and fast.
The current SVGAlib/X/DOSEMU/SVGATextMode/whatever mess is not even
capable of supporting video cards that require IRQ and DMA for the
accelerator functions.