Re: [PATCH 00/21] perf, c2c: Add new tool to analyze cachelinecontention on NUMA systems

From: Don Zickus
Date: Mon Feb 10 2014 - 17:21:12 EST

On Mon, Feb 10, 2014 at 10:29:55PM +0100, Peter Zijlstra wrote:
> On Mon, Feb 10, 2014 at 12:28:55PM -0500, Don Zickus wrote:
> > The data output is verbose and there are lots of data tables that interprit the latencies
> > and data addresses in different ways to help see where bottlenecks might be lying.
>
> Would be good to see what the output looks like.

hehe. unfortunately, my node info is causing double frees now, but
attached below (without node info)..

>
> What I haven't seen; and what I would find most useful; is using the IP
> + dwarf info to map it back to a data structure member.

Yeah, we would like that too. :-)

>
> Since you're already using the PEBS data-source fields, you can also
> have a precise IP. For many cases its possible to reconstruct the exact
> data member the instruction is modifying.
>
> At that point you can do pahole like output of data structures, showing
> which members are 'hot' on misses etc.

Yeah, Arnaldo promised to look into that. I think Stephane was doing some
research into that too.

Cheers,
Don


=================================================
Trace Event Information
=================================================
Total records : 1322047
Locked Load/Store Operations : 206317
Load Operations : 355701
Loads - uncacheable : 590
Loads - no mapping : 207
Load Fill Buffer Hit : 100214
Load L1D hit : 148454
Load L2D hit : 15170
Load LLC hit : 53872
Load Local HITM : 15388
Load Remote HITM : 26760
Load Remote HIT : 3910
Load Local DRAM : 2436
Load Remote DRAM : 3648
Load MESI State Exclusive : 2883
Load MESI State Shared : 3201
Load LLC Misses : 36754
LLC Misses to Local DRAM : 6.6%
LLC Misses to Remote DRAM : 9.9%
LLC Misses to Remote cache (HIT) : 10.6%
LLC Misses to Remote cache (HITM) : 72.8%
Store Operations : 966322
Store - uncacheable : 0
Store - no mapping : 42931
Store L1D Hit : 915696
Virt -> Phys Remap Rejects : 0
No Page Map Rejects : 0


================================================================================
Execution Latency For Loads to Non Shared Memory

Metric Overall Extremes Selected
================================================================================
Samples 301189 3454 104

Minimum 32 1006 4095
Maximum 8149 8149 8149
Threshold 0 1005 4042

Mode 34 1152 4556
Median 136 1250 5163
Mean 236 1524 5337

Std Dev 256.3 839.7 979.5
Coeff of Variation 1.086 0.551 0.184
Confid Interval 0.8 23.5 41.3




====================================================================================================================================================================
Non Shared Data Loads With Excessive Execution Latency

Load ------ Load Inst Execute Latency ------
Num %dist %cumm Count Data Address Inst Address Min Median Max Mean CV Symbol Object
====================================================================================================================================================================
-----------------------------------------------
0 57.3% 57.3% 59 0xffffffff81c57ac0
-----------------------------------------------
33.9% 0x00 0xffffffff81098c43 4169 0.0 8149 5479.0 20.5% update_cfs_shares [kernel.kallsyms]
66.1% 0x20 0xffffffff81094929 4155 0.0 7492 5286.6 16.9% update_cfs_rq_blocked_load [kernel.kallsyms]
-----------------------------------------------
1 5.8% 63.1% 6 0xffffffff818a1180
-----------------------------------------------
50.0% 0x04 0xffffffff815c4f1e 4866 0.0 8116 6745.7 25.0% _raw_spin_lock [kernel.kallsyms]
50.0% 0x04 0xffffffff815c4f47 4556 0.0 7389 6099.3 23.5% _raw_spin_lock [kernel.kallsyms]
-----------------------------------------------
2 4.9% 68.0% 5 0xffff881fbf608ac0
-----------------------------------------------
100.0% 0x24 0xffffffff810981dd 4201 0.0 6653 5429.4 17.4% task_tick_numa [kernel.kallsyms]
-----------------------------------------------
3 2.9% 70.9% 3 0xffff881fa55b0140
-----------------------------------------------
100.0% 0x38 0xffffffff81082e8a 4317 0.0 4571 4418.7 3.0% mspin_lock [kernel.kallsyms]
-----------------------------------------------
4 2.9% 73.8% 3 0xffff883fff834700
-----------------------------------------------
100.0% 0x30 0xffffffff810948e4 4906 0.0 6078 5532.3 10.7% update_cfs_rq_blocked_load [kernel.kallsyms]
-----------------------------------------------
5 2.9% 76.7% 3 0xffff885fff834700
-----------------------------------------------
100.0% 0x30 0xffffffff810948e4 4921 0.0 6703 5828.7 15.3% update_cfs_rq_blocked_load [kernel.kallsyms]
-----------------------------------------------
6 2.9% 79.6% 3 0xffff885fff8b4700
-----------------------------------------------
100.0% 0x30 0xffffffff810948e4 4101 0.0 6022 5166.7 18.9% update_cfs_rq_blocked_load [kernel.kallsyms]
-----------------------------------------------
7 2.9% 82.5% 3 0xffff885fff9d4700
-----------------------------------------------
100.0% 0x30 0xffffffff810948e4 4319 0.0 4486 4381.7 2.1% update_cfs_rq_blocked_load [kernel.kallsyms]
-----------------------------------------------
8 1.9% 84.5% 2 0xffff885fff854700
-----------------------------------------------
100.0% 0x30 0xffffffff810948e4 5434 0.0 6075 5754.5 7.9% update_cfs_rq_blocked_load [kernel.kallsyms]
-----------------------------------------------
9 1.9% 86.4% 2 0xffff885fff974700
-----------------------------------------------
100.0% 0x30 0xffffffff810948e4 5326 0.0 5589 5457.5 3.4% update_cfs_rq_blocked_load [kernel.kallsyms]
-----------------------------------------------
10 1.9% 88.3% 2 0xffffffff819bd400
-----------------------------------------------
50.0% 0x04 0xffffffff810ae886 5427 0.0 5427 5427.0 0.0% ktime_get [kernel.kallsyms]
50.0% 0x04 0xffffffff810aed07 4334 0.0 4334 4334.0 0.0% update_wall_time [kernel.kallsyms]
-----------------------------------------------
11 1.0% 89.3% 1 0xffff881fc5600040
-----------------------------------------------
100.0% 0x20 0xffffffffa162cb70 5606 0.0 5606 5606.0 0.0% mtip_irq_handler [mtip32xx]

<snip 11 records>
...




========================================================================================================
Load Access & Excute Latency Information

Count Minmum Average CV Maximum %dist
========================================================================================================
L1 Hit - Snp None 148444 32 242 1.1501 7492 39.5%
LFB Hit - Snp None 100190 32 271 0.8572 8149 29.8%
L2 Hit - Snp None 15154 32 227 0.9071 1682 3.8%
L3 Hit - Snp None 32029 32 70 1.7761 6353 2.5%
L3 Hit - Snp Miss 2489 38 373 0.7307 5306 1.0%
L3 Hit - Snp Hit - Lcl Cache 3802 32 150 0.9289 3225 0.6%
L3 Hit - Snp Hitm - Lcl Cache 15388 32 187 1.0542 8485 3.2%
L3 Miss - Snp Hit - Rmt Cache 3910 32 355 0.3318 3972 1.5%
L3 Miss - Snp Hitm - Rmt Cache 26760 32 493 0.5116 6236 14.5%
L3 Miss - Snp Hit - Lcl Dram 1029 32 400 0.5783 3578 0.5%
L3 Miss - Snp Hit - Rmt Dram 2170 32 541 0.7315 9967 1.3%
L3 Miss - Snp Miss - Lcl Dram 1406 32 431 0.9117 8116 0.7%
L3 Miss - Snp Miss - Rmt Dram 1477 34 554 0.4437 2956 0.9%
L3 Miss - Snp NA 440 0 607 0.6940 2717 0.3%
Ld UNC - Snp None 0 18446744073709551615 0 -nan 0 0.0%

=================================================
Global Shared Cache Line Event Information
=================================================
Total Shared Cache Lines : 1327
Load HITs on shared lines : 167131
Fill Buffer Hits on shared lines : 43469
L1D hits on shared lines : 50497
L2D hits on shared lines : 960
LLC hits on shared lines : 38467
Locked Access on shared lines : 100032
Store HITs on shared lines : 118659
Store L1D hits on shared lines : 113783
Total Merged records : 160807


================================================================================================================================================================================================================

Shared Data Cache Line Table

Total %All Total ---- Core Load Hit ---- -- LLC Load Hit -- ----- LLC Load Hitm ----- -- Load Dram -- LLC ---- Store Reference ----
Index Phys Adrs Records Ld Miss %hitm Loads FB L1D L2D Lcl Rmt Total Lcl Rmt Lcl Rmt Ld Miss Total L1Hit L1Miss
================================================================================================================================================================================================================
0 0xffff881fa55b0140 72006 16.97% 23.31% 43095 13591 16860 45 2651 25 9526 3288 6238 266 131 6660 28911 28098 813
1 0xffff881fba47f000 21854 5.29% 7.26% 13938 3887 6941 15 1 7 3087 1143 1944 0 0 1951 7916 7916 0
2 0xffff881fc21b9cc0 2153 1.61% 2.21% 862 32 70 0 15 1 740 148 592 0 4 597 1291 1235 56
3 0xffff881fc7d91cc0 1957 1.40% 1.92% 866 34 94 0 14 3 720 207 513 0 1 517 1091 1028 63
4 0xffff881fba539cc0 1813 1.35% 1.85% 808 33 84 3 14 1 665 170 495 8 0 504 1005 967 38
5 0xffff881fc770bcc0 1939 1.30% 1.78% 827 36 70 0 16 1 700 223 477 4 0 482 1112 1058 54
6 0xffff881fbc8adcc0 1854 1.25% 1.72% 788 21 77 1 12 2 674 215 459 0 1 462 1066 965 101
7 0xffff881fc6c03cc0 1825 1.19% 1.63% 800 20 80 1 16 3 677 240 437 1 2 443 1025 973 52
8 0xffff881fb93f1cc0 1934 1.17% 1.61% 846 42 79 2 7 2 711 280 431 0 3 436 1088 1022 66
9 0xffff881fd1391cc0 1901 1.16% 1.59% 840 35 97 0 10 0 693 267 426 0 5 431 1061 1000 61
10 0xffff881fd0771cc0 1731 1.14% 1.57% 779 19 62 1 22 9 663 244 419 1 2 431 952 890 62
11 0xffff881fc7d31cc0 1971 1.13% 1.56% 826 29 91 2 19 5 677 260 417 0 3 425 1145 1064 81
12 0xffff881fb9dcdcc0 1821 1.13% 1.55% 784 31 67 0 13 8 663 249 414 0 2 424 1037 973 64
13 0xffff881fc3febcc0 1795 1.10% 1.51% 788 33 74 1 11 4 663 258 405 1 1 411 1007 936 71
14 0xffff881fc7d29cc0 1837 1.10% 1.51% 756 30 71 0 19 1 634 229 405 1 0 407 1081 1023 58
15 0xffff881fbc365cc0 1961 1.06% 1.46% 850 23 73 0 16 0 736 345 391 2 0 393 1111 1047 64
16 0xffff881fd0259cc0 1896 1.04% 1.43% 779 28 68 1 15 0 665 282 383 1 1 385 1117 1052 65
17 0xffff881fd0589cc0 1848 1.03% 1.42% 860 26 95 0 22 2 714 334 380 0 1 383 988 921 67
18 0xffff881fd01a1cc0 1822 1.02% 1.40% 823 38 78 0 17 1 688 313 375 1 0 377 999 926 73
19 0xffff881fb7ce1cc0 1833 1.00% 1.38% 761 28 72 0 11 2 642 274 368 1 5 376 1072 975 97
20 0xffff881fb8099cc0 1846 0.94% 1.30% 779 23 52 1 11 2 684 337 347 1 5 355 1067 991 76
21 0xffff881fc7e91cc0 1751 0.91% 1.25% 792 29 79 0 20 1 662 327 335 0 1 337 959 905 54
22 0xffff881fbf608000 625 0.33% 0.45% 551 162 17 27 189 18 134 13 121 4 0 143 74 64 10

<snip 40 lines>
....


================================================================================================================================================================================================================

Shared Cache Line Distribution Pareto

---- All ---- -- Shared -- ---- HITM ---- Load Inst Execute Latency
Data Misses Data Misses Remote Local -- Store Refs --
---- cycles ---- cpu
Num %dist %cumm %dist %cumm LLCmiss LLChit L1 hit L1 Miss Data Address Pid Tid Inst Address median mean CV cnt Symbol Object
================================================================================================================================================================================================================
-----------------------------------------------------------------------------------------------
0 17.0% 17.0% 23.3% 23.3% 6238 3288 28098 813 0xffff881fa55b0140 ***
-----------------------------------------------------------------------------------------------
0.0% 0.0% 0.0% 0.0% 0x00 375 375 0xffffffffa018ff5b n/a n/a n/a 1 ext4_bio_write_page [ext4]
0.0% 0.0% 0.0% 0.0% 0x08 18156 18165 0xffffffffa018b7f9 -1 384 0.0% 1 ext4_mark_iloc_dirty [ext4]
0.2% 0.0% 0.0% 0.0% 0x10 18156 *** 0xffffffff811ca1aa -1 387 10.7% 7 __mark_inode_dirty [kernel.kallsyms]
0.7% 0.1% 15.9% 0.0% 0x18 18156 *** 0xffffffff815c15b1 -1 1241 24.0% 51 mutex_unlock [kernel.kallsyms]
0.0% 0.0% 23.2% 0.0% 0x18 18156 *** 0xffffffff815c1615 -1 684 0.0% 50 mutex_lock [kernel.kallsyms]
0.0% 0.0% 0.5% 3.1% 0x18 18156 *** 0xffffffff815c2082 n/a n/a n/a 38 __mutex_unlock_slowpath [kernel.kallsyms]
0.2% 3.3% 0.0% 0.0% 0x18 18156 *** 0xffffffff815c2139 -1 496 22.7% 31 __mutex_lock_slowpath [kernel.kallsyms]
0.2% 3.4% 5.1% 0.0% 0x18 18156 *** 0xffffffff815c2142 -1 821 13.2% 50 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x18 18156 *** 0xffffffff815c21bf -1 1203 0.0% 18 __mutex_lock_slowpath [kernel.kallsyms]
1.2% 0.1% 0.0% 0.0% 0x18 18156 *** 0xffffffff815c21ed -1 671 42.9% 37 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.1% 0.1% 0.1% 0x18 18156 *** 0xffffffff815c21f6 -1 971 16.5% 25 __mutex_lock_slowpath [kernel.kallsyms]
10.9% 6.2% 2.5% 89.3% 0x1c 18156 *** 0xffffffff815c4e4a -1 478 51.7% 50 _raw_spin_unlock [kernel.kallsyms]
11.8% 2.0% 18.4% 0.0% 0x1c 18156 *** 0xffffffff815c4f1e -1 1276 22.0% 50 _raw_spin_lock [kernel.kallsyms]
3.2% 2.6% 0.0% 0.0% 0x1c 18156 *** 0xffffffff815c4f47 -1 831 54.2% 48 _raw_spin_lock [kernel.kallsyms]
0.8% 0.4% 0.0% 0.0% 0x20 18156 *** 0xffffffff815c207f -1 669 26.3% 26 __mutex_unlock_slowpath [kernel.kallsyms]
0.0% 0.0% 0.0% 0.6% 0x28 18156 *** 0xffffffff812b7e4e n/a n/a n/a 5 __list_add [kernel.kallsyms]
0.0% 0.1% 0.0% 0.0% 0x30 18156 *** 0xffffffff81082f19 -1 738 58.4% 5 mutex_spin_on_owner [kernel.kallsyms]
0.1% 8.8% 0.0% 0.0% 0x30 18156 *** 0xffffffff81082f55 -1 730 33.9% 23 mutex_spin_on_owner [kernel.kallsyms]
0.0% 0.0% 0.2% 0.4% 0x30 18156 *** 0xffffffff815c15a6 n/a n/a n/a 27 mutex_unlock [kernel.kallsyms]
0.0% 0.0% 2.8% 6.5% 0x30 18156 *** 0xffffffff815c1628 n/a n/a n/a 50 mutex_lock [kernel.kallsyms]
0.4% 0.4% 0.0% 0.0% 0x30 18156 *** 0xffffffff815c20d6 -1 860 50.5% 17 __mutex_lock_slowpath [kernel.kallsyms]
60.3% 66.7% 0.0% 0.0% 0x30 18156 *** 0xffffffff815c211d -1 471 46.6% 50 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 0.0% 0.0% 0x30 18156 18165 0xffffffff815c2157 n/a n/a n/a 1 __mutex_lock_slowpath [kernel.kallsyms]
9.8% 5.8% 31.0% 0.0% 0x38 18156 *** 0xffffffff81082e8a -1 960 33.3% 50 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x38 18156 *** 0xffffffff81082ec4 -1 588 42.1% 18 mspin_unlock [kernel.kallsyms]

-----------------------------------------------------------------------------------------------
1 5.3% 22.3% 7.3% 30.6% 1944 1143 7916 0 0xffff881fba47f000 18156
-----------------------------------------------------------------------------------------------
100.0% 100.0% 0.0% 0.0% 0x00 18156 *** 0xffffffffa01b410e -1 401 13.5% 50 __ext4_journal_start_sb [ext4]
0.0% 0.0% 10.1% 0.0% 0x28 18156 *** 0xffffffffa0167409 n/a n/a n/a 50 start_this_handle [jbd2]
0.0% 0.0% 89.9% 0.0% 0x28 18156 *** 0xffffffff815c4be9 n/a n/a n/a 50 _raw_read_lock [kernel.kallsyms]

-----------------------------------------------------------------------------------------------
2 1.6% 23.9% 2.2% 32.8% 592 148 1235 56 0xffff881fc21b9cc0 18156
-----------------------------------------------------------------------------------------------
0.0% 0.0% 0.2% 0.0% 0x00 18156 18172 0xffffffff81082e75 n/a n/a n/a 2 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x00 18156 18172 0xffffffff81082f15 n/a n/a n/a 2 mutex_spin_on_owner [kernel.kallsyms]
0.3% 0.0% 0.0% 0.0% 0x00 18156 18172 0xffffffff81082f5a -1 449 0.3% 1 mutex_spin_on_owner [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x00 18156 18172 0xffffffff810908ac n/a n/a n/a 1 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 0.9% 0.0% 0x00 18156 18172 0xffffffff81090a73 n/a n/a n/a 5 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 2.3% 0.0% 0x00 18156 18172 0xffffffff8113524c n/a n/a n/a 7 generic_segment_checks [kernel.kallsyms]
0.0% 0.0% 1.1% 0.0% 0x00 18156 18172 0xffffffff81135f09 n/a n/a n/a 3 generic_file_buffered_write [kernel.kallsyms]
0.0% 0.0% 3.8% 0.0% 0x00 18156 18172 0xffffffff811ba059 n/a n/a n/a 6 file_update_time [kernel.kallsyms]
0.0% 0.0% 0.0% 1.8% 0x00 18156 18172 0xffffffff815c4275 n/a n/a n/a 1 schedule_preempt_disabled [kernel.kallsyms]
2.9% 0.0% 0.0% 0.0% 0x00 18156 18172 0xffffffff815c4299 -1 353 15.2% 6 schedule_preempt_disabled [kernel.kallsyms]
0.0% 0.0% 1.9% 0.0% 0x08 18156 18172 0xffffffff81135245 n/a n/a n/a 6 generic_segment_checks [kernel.kallsyms]
0.0% 0.0% 0.5% 0.0% 0x08 18156 18172 0xffffffff81135f05 n/a n/a n/a 3 generic_file_buffered_write [kernel.kallsyms]
0.0% 0.0% 10.0% 0.0% 0x08 18156 18172 0xffffffff811b9cb5 n/a n/a n/a 8 file_remove_suid [kernel.kallsyms]
0.0% 0.0% 2.3% 0.0% 0x08 18156 18172 0xffffffff811ba055 n/a n/a n/a 8 file_update_time [kernel.kallsyms]
0.0% 0.0% 2.8% 3.6% 0x08 18156 18172 0xffffffff815c2106 n/a n/a n/a 5 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 12.2% 0.0% 0x08 18156 18172 0xffffffff815c2118 n/a n/a n/a 9 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 2.9% 33.9% 0x08 18156 18172 0xffffffff815c212c n/a n/a n/a 8 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 0.0% 7.1% 0x08 18156 18172 0xffffffff815c21e8 n/a n/a n/a 3 __mutex_lock_slowpath [kernel.kallsyms]
0.3% 0.0% 0.0% 0.0% 0x08 18156 18172 0xffffffff815c429a -1 314 28.6% 2 schedule_preempt_disabled [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x10 18156 18172 0xffffffff81082e80 n/a n/a n/a 1 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.0% 17.9% 0x10 18156 *** 0xffffffff81082e9b n/a n/a n/a 8 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x10 18156 18172 0xffffffff810908a2 n/a n/a n/a 1 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 1.4% 0.0% 0x10 18156 18172 0xffffffff81137c4d n/a n/a n/a 7 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 7.4% 0.0% 0x10 18156 18172 0xffffffff81137d7b n/a n/a n/a 9 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x10 18156 18172 0xffffffff81137dbc n/a n/a n/a 2 __generic_file_aio_write [kernel.kallsyms]
0.2% 0.7% 0.0% 0.0% 0x10 18156 *** 0xffffffff812b7e3b -1 507 0.0% 2 __list_add [kernel.kallsyms]
89.0% 98.0% 0.0% 0.0% 0x18 18156 18172 0xffffffff81082ea7 -1 471 28.5% 9 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.0% 33.9% 0x18 18156 *** 0xffffffff81082eda n/a n/a n/a 15 mspin_unlock [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x18 18156 18172 0xffffffff810908a0 n/a n/a n/a 1 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 13.8% 1.8% 0x18 18156 18172 0xffffffff811360de n/a n/a n/a 9 generic_file_buffered_write [kernel.kallsyms]
0.0% 0.0% 6.6% 0.0% 0x18 18156 18172 0xffffffff81137dab n/a n/a n/a 9 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x20 18156 18172 0xffffffff8109089e n/a n/a n/a 1 try_to_wake_up [kernel.kallsyms]
7.3% 1.4% 0.0% 0.0% 0x20 18156 *** 0xffffffff815c208e -1 402 13.6% 29 __mutex_unlock_slowpath [kernel.kallsyms]
0.0% 0.0% 19.5% 0.0% 0x28 18156 18172 0xffffffff81137d63 n/a n/a n/a 9 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x30 18156 18172 0xffffffff81137c49 n/a n/a n/a 1 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 5.4% 0.0% 0x30 18156 18172 0xffffffff81137dc1 n/a n/a n/a 6 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 2.6% 0.0% 0x38 18156 18172 0xffffffff81090b4e n/a n/a n/a 7 wake_up_process [kernel.kallsyms]
0.0% 0.0% 1.3% 0.0% 0x38 18156 18172 0xffffffff81137c2c n/a n/a n/a 4 __generic_file_aio_write [kernel.kallsyms]

-----------------------------------------------------------------------------------------------
3 1.4% 25.3% 1.9% 34.7% 513 207 1028 63 0xffff881fc7d91cc0 18156
-----------------------------------------------------------------------------------------------
0.0% 0.0% 0.2% 0.0% 0x00 18156 18159 0xffffffff81082e75 n/a n/a n/a 2 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x00 18156 18159 0xffffffff81082f15 n/a n/a n/a 1 mutex_spin_on_owner [kernel.kallsyms]
0.4% 0.5% 0.0% 0.0% 0x00 18156 18159 0xffffffff81082f5a -1 446 2.5% 2 mutex_spin_on_owner [kernel.kallsyms]
0.0% 0.0% 0.3% 0.0% 0x00 18156 18159 0xffffffff810908ac n/a n/a n/a 3 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x00 18156 18159 0xffffffff810908fc n/a n/a n/a 1 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 1.6% 0.0% 0x00 18156 18159 0xffffffff81090a73 n/a n/a n/a 7 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 2.9% 0.0% 0x00 18156 18159 0xffffffff8113524c n/a n/a n/a 6 generic_segment_checks [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x00 18156 18159 0xffffffff81135f09 n/a n/a n/a 1 generic_file_buffered_write [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x00 18156 18159 0xffffffff811ba059 n/a n/a n/a 1 file_update_time [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x00 18156 18159 0xffffffff815c4275 n/a n/a n/a 1 schedule_preempt_disabled [kernel.kallsyms]
3.1% 0.0% 0.0% 0.0% 0x00 18156 18159 0xffffffff815c4299 -1 376 18.4% 5 schedule_preempt_disabled [kernel.kallsyms]
0.2% 0.0% 0.0% 0.0% 0x00 18156 18159 0xffffffff815c4f4f -1 431 0.0% 1 _raw_spin_lock [kernel.kallsyms]
0.0% 0.0% 1.9% 0.0% 0x08 18156 18159 0xffffffff81135245 n/a n/a n/a 5 generic_segment_checks [kernel.kallsyms]
0.0% 0.0% 1.6% 0.0% 0x08 18156 18159 0xffffffff811b9cb5 n/a n/a n/a 5 file_remove_suid [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x08 18156 18159 0xffffffff811ba055 n/a n/a n/a 1 file_update_time [kernel.kallsyms]
0.0% 0.0% 2.9% 3.2% 0x08 18156 18159 0xffffffff815c2106 n/a n/a n/a 5 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 17.6% 0.0% 0x08 18156 18159 0xffffffff815c2118 n/a n/a n/a 7 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 6.2% 31.7% 0x08 18156 18159 0xffffffff815c212c n/a n/a n/a 7 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x08 18156 18159 0xffffffff815c21aa n/a n/a n/a 1 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 0.0% 9.5% 0x08 18156 18159 0xffffffff815c21e8 n/a n/a n/a 5 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x08 18156 18159 0xffffffff815c2246 n/a n/a n/a 1 __mutex_lock_slowpath [kernel.kallsyms]
0.4% 0.0% 0.0% 0.0% 0x08 18156 18159 0xffffffff815c429a -1 358 35.7% 2 schedule_preempt_disabled [kernel.kallsyms]
0.0% 0.0% 0.0% 23.8% 0x10 18156 *** 0xffffffff81082e9b n/a n/a n/a 10 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x10 18156 18159 0xffffffff810908a2 n/a n/a n/a 2 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 1.3% 0.0% 0x10 18156 18159 0xffffffff81137c4d n/a n/a n/a 4 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 4.2% 0.0% 0x10 18156 18159 0xffffffff81137d7b n/a n/a n/a 6 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 2.0% 0.0% 0x10 18156 18159 0xffffffff81137dbc n/a n/a n/a 5 __generic_file_aio_write [kernel.kallsyms]
0.2% 0.0% 0.0% 0.0% 0x10 18156 18175 0xffffffff812b7e3b -1 502 0.0% 1 __list_add [kernel.kallsyms]
90.8% 92.3% 0.0% 0.0% 0x18 18156 18159 0xffffffff81082ea7 -1 481 25.2% 7 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.0% 31.7% 0x18 18156 *** 0xffffffff81082eda n/a n/a n/a 12 mspin_unlock [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x18 18156 18159 0xffffffff810908a0 n/a n/a n/a 1 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 24.2% 0.0% 0x18 18156 18159 0xffffffff811360de n/a n/a n/a 7 generic_file_buffered_write [kernel.kallsyms]
0.0% 0.0% 7.8% 0.0% 0x18 18156 18159 0xffffffff81137dab n/a n/a n/a 6 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x20 18156 18159 0xffffffff8109089e n/a n/a n/a 1 try_to_wake_up [kernel.kallsyms]
4.9% 7.2% 0.0% 0.0% 0x20 18156 *** 0xffffffff815c208e -1 381 15.7% 28 __mutex_unlock_slowpath [kernel.kallsyms]
0.0% 0.0% 15.9% 0.0% 0x28 18156 18159 0xffffffff81137d63 n/a n/a n/a 7 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x30 18156 18159 0xffffffff81090895 n/a n/a n/a 1 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x30 18156 18159 0xffffffff81137c49 n/a n/a n/a 1 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 4.3% 0.0% 0x30 18156 18159 0xffffffff81137dc1 n/a n/a n/a 7 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x30 18156 18159 0xffffffff815ce68c n/a n/a n/a 1 apic_timer_interrupt [kernel.kallsyms]
0.0% 0.0% 2.3% 0.0% 0x38 18156 18159 0xffffffff81090b4e n/a n/a n/a 5 wake_up_process [kernel.kallsyms]
0.0% 0.0% 1.3% 0.0% 0x38 18156 18159 0xffffffff81137c2c n/a n/a n/a 6 __generic_file_aio_write [kernel.kallsyms]

-----------------------------------------------------------------------------------------------
4 1.3% 26.6% 1.8% 36.6% 495 170 967 38 0xffff881fba539cc0 18156
-----------------------------------------------------------------------------------------------
0.0% 0.0% 0.2% 0.0% 0x00 18156 18169 0xffffffff81082e75 n/a n/a n/a 2 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x00 18156 18169 0xffffffff81082f15 n/a n/a n/a 2 mutex_spin_on_owner [kernel.kallsyms]
0.0% 0.6% 0.0% 0.0% 0x00 18156 18169 0xffffffff81082f5a n/a n/a n/a 1 mutex_spin_on_owner [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x00 18156 18169 0xffffffff810908ac n/a n/a n/a 1 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 1.0% 0.0% 0x00 18156 18169 0xffffffff81090a73 n/a n/a n/a 5 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 2.1% 0.0% 0x00 18156 18169 0xffffffff8113524c n/a n/a n/a 7 generic_segment_checks [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x00 18156 18169 0xffffffff81135f09 n/a n/a n/a 1 generic_file_buffered_write [kernel.kallsyms]
0.0% 0.0% 0.4% 0.0% 0x00 18156 18169 0xffffffff811ba059 n/a n/a n/a 3 file_update_time [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x00 18156 18169 0xffffffff815c4275 n/a n/a n/a 1 schedule_preempt_disabled [kernel.kallsyms]
1.2% 0.0% 0.0% 0.0% 0x00 18156 18169 0xffffffff815c4299 -1 298 8.7% 6 schedule_preempt_disabled [kernel.kallsyms]
0.2% 0.0% 0.0% 0.0% 0x00 18156 18169 0xffffffff815c4f2c -1 441 0.0% 1 _raw_spin_lock [kernel.kallsyms]
0.4% 0.0% 0.0% 0.0% 0x00 18156 18169 0xffffffff815c4f4f -1 353 8.8% 2 _raw_spin_lock [kernel.kallsyms]
0.0% 0.0% 1.2% 0.0% 0x08 18156 18169 0xffffffff81135245 n/a n/a n/a 9 generic_segment_checks [kernel.kallsyms]
0.0% 0.0% 1.3% 0.0% 0x08 18156 18169 0xffffffff811b9cb5 n/a n/a n/a 8 file_remove_suid [kernel.kallsyms]
0.0% 0.0% 0.6% 0.0% 0x08 18156 18169 0xffffffff811ba055 n/a n/a n/a 4 file_update_time [kernel.kallsyms]
0.0% 0.0% 4.2% 13.2% 0x08 18156 18169 0xffffffff815c2106 n/a n/a n/a 9 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 13.8% 0.0% 0x08 18156 18169 0xffffffff815c2118 n/a n/a n/a 12 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 5.0% 50.0% 0x08 18156 18169 0xffffffff815c212c n/a n/a n/a 11 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x08 18156 18169 0xffffffff815c21e0 n/a n/a n/a 1 __mutex_lock_slowpath [kernel.kallsyms]
0.0% 0.0% 0.1% 15.8% 0x08 18156 18169 0xffffffff815c21e8 n/a n/a n/a 5 __mutex_lock_slowpath [kernel.kallsyms]
1.0% 0.0% 0.0% 0.0% 0x08 18156 18169 0xffffffff815c429a -1 286 5.8% 3 schedule_preempt_disabled [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x10 18156 18169 0xffffffff81082e80 n/a n/a n/a 1 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x10 18156 18169 0xffffffff810908a2 n/a n/a n/a 2 try_to_wake_up [kernel.kallsyms]
0.0% 0.0% 1.2% 0.0% 0x10 18156 18169 0xffffffff81137c4d n/a n/a n/a 6 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 5.2% 0.0% 0x10 18156 18169 0xffffffff81137d7b n/a n/a n/a 12 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 1.9% 0.0% 0x10 18156 18169 0xffffffff81137dbc n/a n/a n/a 8 __generic_file_aio_write [kernel.kallsyms]
0.2% 0.0% 0.0% 0.0% 0x10 18156 18175 0xffffffff812b7e3b -1 355 0.0% 1 __list_add [kernel.kallsyms]
90.9% 93.5% 0.0% 0.0% 0x18 18156 18169 0xffffffff81082ea7 -1 613 28.9% 13 mspin_lock [kernel.kallsyms]
0.0% 0.0% 0.0% 21.1% 0x18 18156 *** 0xffffffff81082eda n/a n/a n/a 7 mspin_unlock [kernel.kallsyms]
0.0% 0.0% 26.0% 0.0% 0x18 18156 18169 0xffffffff811360de n/a n/a n/a 12 generic_file_buffered_write [kernel.kallsyms]
0.0% 0.0% 9.0% 0.0% 0x18 18156 18169 0xffffffff81137dab n/a n/a n/a 12 __generic_file_aio_write [kernel.kallsyms]
6.1% 5.9% 0.0% 0.0% 0x20 18156 *** 0xffffffff815c208e -1 377 19.3% 25 __mutex_unlock_slowpath [kernel.kallsyms]
0.0% 0.0% 18.2% 0.0% 0x28 18156 18169 0xffffffff81137d63 n/a n/a n/a 12 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 0.2% 0.0% 0x30 18156 18169 0xffffffff81137c49 n/a n/a n/a 2 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 0.1% 0.0% 0x30 18156 18169 0xffffffff81137d67 n/a n/a n/a 1 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 3.1% 0.0% 0x30 18156 18169 0xffffffff81137dc1 n/a n/a n/a 8 __generic_file_aio_write [kernel.kallsyms]
0.0% 0.0% 3.1% 0.0% 0x38 18156 18169 0xffffffff81090b4e n/a n/a n/a 11 wake_up_process [kernel.kallsyms]
0.0% 0.0% 1.1% 0.0% 0x38 18156 18169 0xffffffff81137c2c n/a n/a n/a 8 __generic_file_aio_write [kernel.kallsyms]


<snip 58 records>
....



=====================================================================================================================================
Object Name, Path & Reference Counts

Index Records Object Name Object Path
=====================================================================================================================================
0 931379 [kernel.kallsyms] [kernel.kallsyms]
1 192258 fio /home/joe/old_fio-2.0.15/fio
2 80302 [jbd2] /lib/modules/3.10.0c2c_all+/kernel/fs/jbd2/jbd2.ko
3 65392 [ext4] /lib/modules/3.10.0c2c_all+/kernel/fs/ext4/ext4.ko
4 8236 libpthread-2.17.so /usr/lib64/libpthread-2.17.so
5 19 [ip_tables] /lib/modules/3.10.0c2c_all+/kernel/net/ipv4/netfilter/ip_tables.ko
6 17 [ixgbe] /lib/modules/3.10.0c2c_all+/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
7 17 perf /home/root/git/rhel7.don/tools/perf/perf
8 13 [ipmi_si] /lib/modules/3.10.0c2c_all+/kernel/drivers/char/ipmi/ipmi_si.ko
9 11 libc-2.17.so /usr/lib64/libc-2.17.so
10 10 [megaraid_sas] /lib/modules/3.10.0c2c_all+/kernel/drivers/scsi/megaraid/megaraid_sas.ko
11 9 [mtip32xx] /lib/modules/3.10.0c2c_all+/kernel/drivers/block/mtip32xx/mtip32xx.ko
12 6 irqbalance /usr/sbin/irqbalance
13 6 libpython2.7.so.1.0 /usr/lib64/libpython2.7.so.1.0
14 5 [ip6_tables] /lib/modules/3.10.0c2c_all+/kernel/net/ipv6/netfilter/ip6_tables.ko
15 4 [nf_conntrack] /lib/modules/3.10.0c2c_all+/kernel/net/netfilter/nf_conntrack.ko
16 2 [dm_mod] /lib/modules/3.10.0c2c_all+/kernel/drivers/md/dm-mod.ko
17 2 sshd /usr/sbin/sshd
18 1 [iptable_raw] /lib/modules/3.10.0c2c_all+/kernel/net/ipv4/netfilter/iptable_raw.ko
19 1 [sb_edac] /lib/modules/3.10.0c2c_all+/kernel/drivers/edac/sb_edac.ko
20 1 [edac_core] /lib/modules/3.10.0c2c_all+/kernel/drivers/edac/edac_core.ko
21 1 libdbus-1.so.3.7.4 /usr/lib64/libdbus-1.so.3.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/