KVM guest-kernel panics double fault

[Stephan Bärwolf]

This is a multi-part message in MIME format.Hello guys,

I am sorry to disturb you this short before New Year, but I think this
shouldn't wait until next year.

After experiencing crashes in virtual maschines and considering kernel /
qemu / kvm / cpu -bugs, I discovered the following (see patch) issue.
Because unpriviledged users can crash VMs, I think it is a serious one
and needs short-term attention.

The patch I wrote is against 3.2-rc7 but I always tested with linux 3.1.6.
Hopfully it solve the problems to your satisfaction.

regards and a happy new year,
    Stephan Bärwolf




Subject: [PATCH] KVM: fix missing "illegal instruction"-trap in guests
within non-64bit protected modes

On hosts without this patch, 32bit guests will crash for
example by simply executing following nasm-demo-application:

        [bits 32]
        global _start
        SECTION .text
        _start: syscall

(I am not sure if this can be exploited in more worse ways,
like breaking out of VMs in more complex szenarios?
But I tested it with win32 and linux - both always crashed)

        Disassembly of section .text:

        00000000 <_start>:
           0:   0f 05                   syscall

The reason seems a missing "invalid opcode"-trap (int6) for the
syscall opcode "0f05", which is not available on 32bit cpus.
Intel's "Intel 64 and IA-32 Architecture Software Developers
Manual" (http://www.intel.com/content/dam/doc/manual/
64-ia-32-architectures-software-developer-manual-325462.pdf)
documents on page 1804 (4-586) "syscall" is only available
in 64bit longmode. So "syscall" must trap in real- and
virtual 8086 -mode, as also in all non-64bit protected-modes.

The last ones (16 & 32bit protected mode) are not beeing checked
by kvm and so causing a missing trap as an double-fault-panic
on 32bit guests.

Also an initially not observed problem can be explained
with this bug:
On 64bit guests (x86_64) 32bit compat-programs are able to
syscall their kernel via "0f05" correctly, althought native
(not virtualized) systems would also trap!

This patch solves the described problem by extending the
checking of cpu's operational mode.

Screenshots of a i686 testing VM  before and after applying
this patch are available under:

http://matrixstorm.com/software/linux/kvm/20111229/before.jpg
http://matrixstorm.com/software/linux/kvm/20111229/after.jpg