Re: [PATCH 2/2] perf test: Fix JSON format linter test checks

From: Arnaldo Carvalho de Melo
Date: Fri Jan 27 2023 - 07:27:52 EST

Next message: Reshetova, Elena: "RE: Linux guest kernel threat model for Confidential Computing"
Previous message: Steven Price: "[RFC PATCH 10/28] arm64: RME: Allocate/free RECs to match vCPUs"
Next in thread: Arnaldo Carvalho de Melo: "Re: [PATCH 2/2] perf test: Fix JSON format linter test checks"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Em Fri, Jan 20, 2023 at 02:40:39PM +0100, Michael Petlan escreveu:
> The test fails on CPUs with topdown metrics, where it is common to print
> two metrics per line. Since these are included in default event set for
> `perf stat -- something`, it causes the "no args" subtest to fail due to
> unexpected member count. We need to accept 7 or 9 members in that case.
>
> Coming to that, counting commas in the JSON line and consider it a number
> of elements is incorrect and misleading. There should be +1 element than
> the count of commas, while also, commas can potentially appear in the
> event names. Count " : " substrings rather, since these better fit to
> what should be actually counted.

Before I apply this first patch I can run, as root, the 'perf test' JSON
lint 100 times without problems:

[root@quaco ~]# for a in $(seq 100) ; do echo -n $a ; perf test 98 ; done
1 98: perf stat JSON output linter : Ok
2 98: perf stat JSON output linter : Ok
3 98: perf stat JSON output linter : Ok
4 98: perf stat JSON output linter : Ok
<SNIP>
96 98: perf stat JSON output linter : Ok
97 98: perf stat JSON output linter : Ok
98 98: perf stat JSON output linter : Ok
99 98: perf stat JSON output linter : Ok
100 98: perf stat JSON output linter : Ok
[root@quaco ~]#

After applying it it fails seemingly randomly, I'll remove both patches
from my tmp.perf/core branch and investigate.

[acme@quaco perf]$ git log --oneline -1
fdafd42aac3629db (HEAD) perf stat: Fix JSON metric printout for multiple metrics per line
[acme@quaco perf]$ perf -v
perf version 6.2.rc5.gfdafd42aac36
[acme@quaco perf]$
[root@quaco ~]# perf -v
perf version 6.2.rc5.gfdafd42aac36
[root@quaco ~]# for a in $(seq 100) ; do echo -n $a ; perf test 98 ; done
[root@quaco ~]# perf -v
perf version 6.2.rc5.gfdafd42aac36
[root@quaco ~]# for a in $(seq 100) ; do echo -n $a ; perf test 98 ; done
1 98: perf stat JSON output linter : Ok
2 98: perf stat JSON output linter : Ok
3 98: perf stat JSON output linter : FAILED!
4 98: perf stat JSON output linter : Ok
5 98: perf stat JSON output linter : Ok
6 98: perf stat JSON output linter : Ok
7 98: perf stat JSON output linter : FAILED!
8 98: perf stat JSON output linter : Ok
9 98: perf stat JSON output linter : Ok
10 98: perf stat JSON output linter : Ok
11 98: perf stat JSON output linter : Ok
12 98: perf stat JSON output linter : Ok
13 98: perf stat JSON output linter : Ok
14 98: perf stat JSON output linter : Ok
15 98: perf stat JSON output linter : Ok
16 98: perf stat JSON output linter : FAILED!
17 98: perf stat JSON output linter : Ok
18 98: perf stat JSON output linter : Ok
19 98: perf stat JSON output linter : Ok
20 98: perf stat JSON output linter : FAILED!
21 98: perf stat JSON output linter : Ok
22 98: perf stat JSON output linter : Ok
23 98: perf stat JSON output linter : FAILED!
24 98: perf stat JSON output linter : Ok
25 98: perf stat JSON output linter : Ok
26 98: perf stat JSON output linter : Ok
27 98: perf stat JSON output linter : Ok
28 98: perf stat JSON output linter : FAILED!
29 98: perf stat JSON output linter : Ok
30 98: perf stat JSON output linter : Ok
31 98: perf stat JSON output linter : FAILED!
32 98: perf stat JSON output linter : FAILED!
33 98: perf stat JSON output linter : FAILED!
34 98: perf stat JSON output linter : Ok
35 98: perf stat JSON output linter : Ok
36 98: perf stat JSON output linter : Ok
37 98: perf stat JSON output linter : Ok
38 98: perf stat JSON output linter : FAILED!
39 98: perf stat JSON output linter : FAILED!
40 98: perf stat JSON output linter : FAILED!
41 98: perf stat JSON output linter : Ok
42 98: perf stat JSON output linter : Ok
43 98: perf stat JSON output linter : Ok
44 98: perf stat JSON output linter : Ok
45 98: perf stat JSON output linter : Ok
46 98: perf stat JSON output linter : FAILED!
47 98: perf stat JSON output linter : Ok
48 98: perf stat JSON output linter : FAILED!
49 98: perf stat JSON output linter : FAILED!
50 98: perf stat JSON output linter : FAILED!
51 98: perf stat JSON output linter : FAILED!
52 98: perf stat JSON output linter : Ok
53 98: perf stat JSON output linter : FAILED!
54 98: perf stat JSON output linter : Ok
55 98: perf stat JSON output linter : Ok
56 98: perf stat JSON output linter : FAILED!
57 98: perf stat JSON output linter : Ok
58 98: perf stat JSON output linter : Ok
59 98: perf stat JSON output linter : FAILED!
60 98: perf stat JSON output linter : Ok
61 98: perf stat JSON output linter : FAILED!
62 98: perf stat JSON output linter : Ok
63 98: perf stat JSON output linter : Ok
64 98: perf stat JSON output linter : Ok
65 98: perf stat JSON output linter : Ok
66 98: perf stat JSON output linter : Ok
67 98: perf stat JSON output linter : Ok
68 98: perf stat JSON output linter : Ok
69 98: perf stat JSON output linter : Ok
70 98: perf stat JSON output linter : Ok
71 98: perf stat JSON output linter : FAILED!
72 98: perf stat JSON output linter : FAILED!
73 98: perf stat JSON output linter : Ok
74 98: perf stat JSON output linter : FAILED!
75 98: perf stat JSON output linter : Ok
76 98: perf stat JSON output linter : FAILED!
77 98: perf stat JSON output linter : Ok
78 98: perf stat JSON output linter : FAILED!
79 98: perf stat JSON output linter : FAILED!
80 98: perf stat JSON output linter : FAILED!
81 98: perf stat JSON output linter : Ok
82 98: perf stat JSON output linter : FAILED!
83 98: perf stat JSON output linter : FAILED!
84 98: perf stat JSON output linter : Ok
85 98: perf stat JSON output linter : Ok
86 98: perf stat JSON output linter : Ok
87 98: perf stat JSON output linter : FAILED!
88 98: perf stat JSON output linter : FAILED!
89 98: perf stat JSON output linter : Ok
90 98: perf stat JSON output linter : FAILED!
91 98: perf stat JSON output linter : FAILED!
92 98: perf stat JSON output linter : Ok
93 98: perf stat JSON output linter : FAILED!
94 98: perf stat JSON output linter : FAILED!
95 98: perf stat JSON output linter : FAILED!
96 98: perf stat JSON output linter : FAILED!
97 98: perf stat JSON output linter : FAILED!
98 98: perf stat JSON output linter : Ok
99 98: perf stat JSON output linter : Ok
100 98: perf stat JSON output linter : FAILED!
[root@quaco ~]#

> Fixes: 0c343af2a2f8 ("perf test: JSON format checking")
>
> Signed-off-by: Michael Petlan <mpetlan@xxxxxxxxxx>
> ---
> .../tests/shell/lib/perf_json_output_lint.py | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/tools/perf/tests/shell/lib/perf_json_output_lint.py b/tools/perf/tests/shell/lib/perf_json_output_lint.py
> index d90f8d102eb9..4f1bbb3f07ec 100644
> --- a/tools/perf/tests/shell/lib/perf_json_output_lint.py
> +++ b/tools/perf/tests/shell/lib/perf_json_output_lint.py
> @@ -40,17 +40,17 @@ def is_counter_value(num):
> return isfloat(num) or num == '<not counted>' or num == '<not supported>'
>
> def check_json_output(expected_items):
> - if expected_items != -1:
> + if expected_items:
> for line in Lines:
> if 'failed' not in line:
> count = 0
> - count = line.count(',')
> - if count != expected_items and count >= 1 and count <= 3 and 'metric-value' in line:
> + count = line.count(' : ')
> + if count not in expected_items and count >= 2 and count <= 4 and 'metric-value' in line:
> # Events that generate >1 metric may have isolated metric
> # values and possibly other prefixes like interval, core and
> # aggregate-number.
> continue
> - if count != expected_items:
> + if count not in expected_items:
> raise RuntimeError(f'wrong number of fields. counted {count} expected {expected_items}'
> f' in \'{line}\'')
> checks = {
> @@ -82,14 +82,14 @@ def check_json_output(expected_items):
>
> try:
> if args.no_args or args.system_wide or args.event:
> - expected_items = 6
> + expected_items = [7, 9]
> elif args.interval or args.per_thread or args.system_wide_no_aggr:
> - expected_items = 7
> + expected_items = [8, 10]
> elif args.per_core or args.per_socket or args.per_node or args.per_die:
> - expected_items = 8
> + expected_items = [9, 11]
> else:
> # If no option is specified, don't check the number of items.
> - expected_items = -1
> + expected_items = []
> check_json_output(expected_items)
> except:
> print('Test failed for input:\n' + '\n'.join(Lines))
> --
> 2.18.4
>

--

- Arnaldo

Next message: Reshetova, Elena: "RE: Linux guest kernel threat model for Confidential Computing"
Previous message: Steven Price: "[RFC PATCH 10/28] arm64: RME: Allocate/free RECs to match vCPUs"
Next in thread: Arnaldo Carvalho de Melo: "Re: [PATCH 2/2] perf test: Fix JSON format linter test checks"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]