Re: [RFC PATCH v3 00/32] x86: enable FRED for x86-64

From: Kang, Shan
Date: Fri Feb 24 2023 - 04:27:10 EST

Next message: Kang, Shan: "Re: [RFC PATCH v3 00/32] x86: enable FRED for x86-64"
Previous message: Pekka Paalanen: "Re: [PATCH v4 06/14] dma-buf/sync_file: Support (E)POLLPRI"
In reply to: Xin Li: "[RFC PATCH v3 32/32] x86/fred: disable FRED by default in its early stage"
Next in thread: Li, Xin3: "RE: [RFC PATCH v3 00/32] x86: enable FRED for x86-64"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

We tested the v3 FRED patch set on the Intel Simics® Simulator and a machine
with a 7th Intel(R) Core(TM) CPU.

Following are the LTP verion 20220121 test results on X86-64.
+--------------------------------------------+-------+-------+-------+-------+
| Config | Pass | Fail | Skip | Hang |
+--------------------------------------------+-------+-------+-------+-------+
| the 7th Intel(R) Core(TM) CPU | 1788 | 108 | 453 | 0 |
| 6.2.0-rc7+ | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| the 7th Intel(R) Core(TM) CPU | 1788 | 108 | 453 | 0 |
| 6.2.0-rc7+ w/ FRED patch set | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/o FRED model | 1797 | 111 | 440 | 1 |
| 6.2.0-rc7+ | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/o FRED model | 1797 | 111 | 440 | 1 |
| 6.2.0-rc7+ w/ FRED patch set | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/ FRED model | 1797 | 111 | 440 | 1 |
| 6.2.0-rc7+ | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/ FRED model | 1797 | 111 | 440 | 1 |
| 6.2.0-rc7+ w/ FRED patch set FRED disabled | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/ FRED model | 1797 | 111 | 440 | 1 |
| 6.2.0-rc7+ w/ FRED patch set | | | | |
+--------------------------------------------+-------+-------+-------+-------+
We were unable to identify any regression with the LTP tests.

Following are the Kselftest results on X86-64.
+--------------------------------------------+-------+-------+-------+-------+
| Config | Pass | Fail | Skip | Hang |
+--------------------------------------------+-------+-------+-------+-------+
| the 7th Intel(R) Core(TM) CPU | 3224 | 454 | 591 | 5 |
| 6.2.0-rc7+ | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| the 7th Intel(R) Core(TM) CPU | 3224 | 454 | 591 | 5 |
| 6.2.0-rc7+ w/ FRED patch set | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/o FRED model | 1853 | 267 | 2143 | 11 |
| 6.2.0-rc7+ | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/o FRED model | 1853 | 267 | 2143 | 11 |
| 6.2.0-rc7+ w/ FRED patch set | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/ FRED model | 1853 | 267 | 2143 | 11 |
| 6.2.0-rc7+ | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/ FRED model | 1853 | 267 | 2143 | 11 |
| 6.2.0-rc7+ w/ FRED patch set FRED disabled | | | | |
+--------------------------------------------+-------+-------+-------+-------+
| Intel Simics® Simulator w/ FRED model | 1847 | 272 | 2143 | 12 |
| 6.2.0-rc7+ w/ FRED patch set | | | | |
+--------------------------------------------+-------+-------+-------+-------+
No regression was found on the bare metal, while the following are the
regressions on the Simics® Simulator.
+----------------------+----------------------+----------------------+
| | Intel Simics® | Intel Simics® |
| | Simulator w/ FRED | Simulator w/ FRED |
| test name | model | model |
| | 6.2.0-rc7+ w/ FRED | 6.2.0-rc7+ w/ FRED |
| | patch set FRED | patch set |
| | disabled | |
+----------------------+----------------------+----------------------+
| kvm:hyperv_clock | PASS | FAIL |
+----------------------+----------------------+----------------------+
| kvm:hyperv_features | PASS | FAIL |
+----------------------+----------------------+----------------------+
| kvm:xen_vmcall_test | PASS | FAIL |
+----------------------+----------------------+----------------------+
| kvm:hyperv_ipi | PASS | FAIL |
+----------------------+----------------------+----------------------+
| kvm:hyperv_tlb_flush | PASS | FAIL |
+----------------------+----------------------+----------------------+
| bpf:test_progs | FAIL | HANG |
+----------------------+----------------------+----------------------+
| x86:sysret_rip_64 | PASS | FAIL |
+----------------------+----------------------+----------------------+
The test "x86:sysret_rip_64" is NOT a valid test on FRED, and there is a fix
from Ammar Faizi after we discussed it in the LKML. Other test failures are
under investigation.

FRED is a 64-bit only feature, however the 31st patch "x86/fred: allow dynamic
stack frame size" applies to X86-32, thus we ran Kselftest and LTP on X86-32 to
see if there is any regression, and we din't find any.

Following are the Kselftest results on X86-32.
+-------------------------------+---------+---------+---------+---------+
| Config | Pass | Fail | Skip | Hang |
+-------------------------------+---------+---------+---------+---------+
| the 5th Intel(R) Core(TM) CPU | 1361 | 267 | 301 | 3 |
| FRED 6.2.0-rc7+ | | | | |
+-------------------------------+---------+---------+---------+---------+
| the 5th Intel(R) Core(TM) CPU | 1361 | 267 | 301 | 3 |
| 6.2.0-rc7+ w/ FRED patch set | | | | |
+-------------------------------+---------+---------+---------+---------+

Following are the LTP test results on X86-32.
+-------------------------------+---------+---------+---------+---------+
| Config | Pass | Fail | Skip | Hang |
+-------------------------------+---------+---------+---------+---------+
| the 5th Intel(R) Core(TM) CPU | 1849 | 81 | 419 | 0 |
| 6.2.0-rc7+ w/ FRED patch set | | | | |
+-------------------------------+---------+---------+---------+---------+
| the 5th Intel(R) Core(TM) CPU | 1849 | 81 | 419 | 0 |
| FRED 6.2.0-rc7+ | | | | |
+-------------------------------+---------+---------+---------+---------+

Thanks
--Shan

On Thu, 2023-02-23 at 23:01 -0800, Xin Li wrote:
> This patch set enables FRED for x86-64, and it's based on the previous LKGS
> patch set.
>
> The Intel flexible return and event delivery (FRED) architecture defines
> simple
> new transitions that change privilege level (ring transitions). The FRED
> architecture was designed with the following goals:
> 1) Improve overall performance and response time by replacing event delivery
> through the interrupt descriptor table (IDT event delivery) and event return
> by
> the IRET instruction with lower latency transitions.
> 2) Improve software robustness by ensuring that event delivery establishes the
> full supervisor context and that event return establishes the full user
> context.
>
> The new transitions defined by the FRED architecture are FRED event delivery
> and,
> for returning from events, two FRED return instructions. FRED event delivery
> can
> effect a transition from ring 3 to ring 0, but it is used also to deliver
> events
> incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0
> to
> ring 3, while the other (ERETS) returns while remaining in ring 0.
>
> Search for the latest FRED spec in most search engines with this search
> pattern:
>
> site:intel.com FRED (flexible return and event delivery) specification
>
> As of now there is no publicly avaiable CPU supporting FRED, thus the Intel
> Simics® Simulator is used as software development and testing vehicles. And
> it can be downloaded from:
>
> https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html
>
> To enable FRED, Simics package 8112 QSP-CPU needs to be installed with CPU
> model configured as:
> $cpu_comp_class = "x86-experimental-fred"
>
> Longer term, we should refactor common code shared by FRED and IDT into common
> shared files, and contain IDT code using a new config CONFIG_X86_IDT.
>
> TODO: call external_interrupt() to reinject IRQ in KVM VMX.
>
> Changes since v2:
> * Improve comments for changes in arch/x86/include/asm/idtentry.h.
>
> Changes since v1:
> * call irqentry_nmi_{enter,exit}() in both IDT and FRED debug fault kernel
> handler (Peter Zijlstra).
> * Initialize a FRED exception handler to fred_bad_event() instead of NULL
> if no FRED handler defined for an exception vector (Peter Zijlstra).
> * Push calling irqentry_{enter,exit}() and instrumentation_{begin,end}()
> down into individual FRED exception handlers, instead of in the dispatch
> framework (Peter Zijlstra).
>
> H. Peter Anvin (Intel) (24):
> x86/traps: let common_interrupt() handle IRQ_MOVE_CLEANUP_VECTOR
> x86/traps: add a system interrupt table for system interrupt dispatch
> x86/traps: add external_interrupt() to dispatch external interrupts
> x86/cpufeature: add the cpu feature bit for FRED
> x86/opcode: add ERETU, ERETS instructions to x86-opcode-map
> x86/objtool: teach objtool about ERETU and ERETS
> x86/cpu: add X86_CR4_FRED macro
> x86/fred: add Kconfig option for FRED (CONFIG_X86_FRED)
> x86/fred: if CONFIG_X86_FRED is disabled, disable FRED support
> x86/cpu: add MSR numbers for FRED configuration
> x86/fred: header file with FRED definitions
> x86/fred: make unions for the cs and ss fields in struct pt_regs
> x86/fred: reserve space for the FRED stack frame
> x86/fred: add a page fault entry stub for FRED
> x86/fred: add a debug fault entry stub for FRED
> x86/fred: add a NMI entry stub for FRED
> x86/fred: FRED entry/exit and dispatch code
> x86/fred: FRED initialization code
> x86/fred: update MSR_IA32_FRED_RSP0 during task switch
> x86/fred: let ret_from_fork() jmp to fred_exit_user when FRED is
> enabled
> x86/fred: disallow the swapgs instruction when FRED is enabled
> x86/fred: no ESPFIX needed when FRED is enabled
> x86/fred: allow single-step trap and NMI when starting a new thread
> x86/fred: allow FRED systems to use interrupt vectors 0x10-0x1f
>
> Xin Li (8):
> x86/traps: add install_system_interrupt_handler()
> x86/traps: export external_interrupt() for VMX IRQ reinjection
> x86/fred: header file for event types
> x86/fred: add a machine check entry stub for FRED
> x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user
> x86/ia32: do not modify the DPL bits for a null selector
> x86/fred: allow dynamic stack frame size
> x86/fred: disable FRED by default in its early stage
>
> .../admin-guide/kernel-parameters.txt | 4 +
> arch/x86/Kconfig | 9 +
> arch/x86/entry/Makefile | 5 +-
> arch/x86/entry/entry_32.S | 2 +-
> arch/x86/entry/entry_64.S | 5 +
> arch/x86/entry/entry_64_fred.S | 59 +++++
> arch/x86/entry/entry_fred.c | 234 ++++++++++++++++++
> arch/x86/entry/vsyscall/vsyscall_64.c | 2 +-
> arch/x86/include/asm/cpufeatures.h | 1 +
> arch/x86/include/asm/disabled-features.h | 8 +-
> arch/x86/include/asm/entry-common.h | 3 +
> arch/x86/include/asm/event-type.h | 17 ++
> arch/x86/include/asm/extable_fixup_types.h | 4 +-
> arch/x86/include/asm/fred.h | 131 ++++++++++
> arch/x86/include/asm/idtentry.h | 76 +++++-
> arch/x86/include/asm/irq.h | 5 +
> arch/x86/include/asm/irq_vectors.h | 15 +-
> arch/x86/include/asm/msr-index.h | 13 +-
> arch/x86/include/asm/processor.h | 12 +-
> arch/x86/include/asm/ptrace.h | 36 ++-
> arch/x86/include/asm/switch_to.h | 10 +-
> arch/x86/include/asm/thread_info.h | 35 +--
> arch/x86/include/asm/traps.h | 13 +
> arch/x86/include/asm/vmx.h | 17 +-
> arch/x86/include/uapi/asm/processor-flags.h | 2 +
> arch/x86/kernel/Makefile | 1 +
> arch/x86/kernel/apic/apic.c | 11 +-
> arch/x86/kernel/apic/vector.c | 8 +-
> arch/x86/kernel/cpu/acrn.c | 7 +-
> arch/x86/kernel/cpu/common.c | 88 ++++---
> arch/x86/kernel/cpu/mce/core.c | 11 +
> arch/x86/kernel/cpu/mshyperv.c | 22 +-
> arch/x86/kernel/espfix_64.c | 8 +
> arch/x86/kernel/fred.c | 73 ++++++
> arch/x86/kernel/head_32.S | 3 +-
> arch/x86/kernel/idt.c | 6 +-
> arch/x86/kernel/irq.c | 6 +-
> arch/x86/kernel/irqinit.c | 7 +-
> arch/x86/kernel/kvm.c | 4 +-
> arch/x86/kernel/nmi.c | 28 +++
> arch/x86/kernel/process.c | 5 +
> arch/x86/kernel/process_64.c | 21 +-
> arch/x86/kernel/signal_32.c | 21 +-
> arch/x86/kernel/traps.c | 175 +++++++++++--
> arch/x86/lib/x86-opcode-map.txt | 2 +-
> arch/x86/mm/extable.c | 28 +++
> arch/x86/mm/fault.c | 20 +-
> drivers/xen/events/events_base.c | 5 +-
> kernel/fork.c | 6 +
> tools/arch/x86/include/asm/cpufeatures.h | 1 +
> .../arch/x86/include/asm/disabled-features.h | 8 +-
> tools/arch/x86/include/asm/msr-index.h | 13 +-
> tools/arch/x86/lib/x86-opcode-map.txt | 2 +-
> tools/objtool/arch/x86/decode.c | 22 +-
> 54 files changed, 1156 insertions(+), 174 deletions(-)
> create mode 100644 arch/x86/entry/entry_64_fred.S
> create mode 100644 arch/x86/entry/entry_fred.c
> create mode 100644 arch/x86/include/asm/event-type.h
> create mode 100644 arch/x86/include/asm/fred.h
> create mode 100644 arch/x86/kernel/fred.c
>

Next message: Kang, Shan: "Re: [RFC PATCH v3 00/32] x86: enable FRED for x86-64"
Previous message: Pekka Paalanen: "Re: [PATCH v4 06/14] dma-buf/sync_file: Support (E)POLLPRI"
In reply to: Xin Li: "[RFC PATCH v3 32/32] x86/fred: disable FRED by default in its early stage"
Next in thread: Li, Xin3: "RE: [RFC PATCH v3 00/32] x86: enable FRED for x86-64"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]