Re: Quad core CPUs loaded at only 50% when running a CPU and mmapintensive multi-threaded task

From: edwin
Date: Mon Aug 25 2008 - 07:03:45 EST

Peter Zijlstra wrote:
On Mon, 2008-08-25 at 13:22 +0300, TÃrÃk Edwin wrote:

Well, the real program (clamd) that this testprogram tries to simulate does an mmap for almost every file, and I have lots of small files.
6.5G, 114122 files, average size 57k.

I'll run latencytop again, last time it has showed 100ms - 500ms latency

Latencytop output attached.
There is 4 - 60 ms latency for mmap/munmap, and the more threads there are the total latency gets higher (latencytop says sum was ~480ms).

Running with MaxThreads 4 gets me 300-400% CPU usage, but with MaxThreads 8 CPU usage drops to around 120-250%.
Now, maxthreads 4 looks like a good choice from a CPU usage point of view, but is actually bad because it means that threads gets stuck in iowait, and the CPU won't have anything to do. MaxThreads 8 looked like a good alternative to fill the iowait gaps, but we run into the mmap_sem issue.
In a real world environment MaxThreads influences how many mails you can process in parallel with your MTA, so generally it should be as high as possible.

On 2.6.27-rc4:

MaxThreads 4 time, empty database (all cached, almost no I/O):
1m9s

MaxThreads 4 time, after echo 3>/proc/sys/vm/drop_caches:
1m29s

MaxThreads 8 time, empty database (all cached, almost no I/O):
2m16s

MaxThreads 8 time, after echo 3>/proc/sys/vm/drop_caches:
2m15s


Of course running with a full database will give different results, so I'll do some timing with that too (will take a little longer though).

for clamd, and it was about mmap, I'll provide you with the exact output.

Right - does it make sense to teach clamav about pread() ?

If it is preferred over mmap, then maybe yes.

Peter Zijlstra wrote:
OK, I'll poke a little more at is later today to see if I can spot
something

Thanks!

Best regards,
--Edwin
---------
Latency Top version : v0.1
clamd (5571, #threads: 10)
---------------------------------------------------------
se.exec_start : 700869.502286
se.vruntime : 262417.157677
se.sum_exec_runtime : 0.000000
se.avg_overlap : 0.000000
se.wait_start : 0.000000
se.sleep_start : 700869.502286
se.block_start : 0.000000
se.sleep_max : 0.000000
se.block_max : 0.000000
se.exec_max : 0.000000
se.slice_max : 0.000000
se.wait_max : 0.000000
se.wait_sum : 0.000000
se.wait_count : 0
sched_info.bkl_count : 0
se.nr_migrations : 0
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 0
se.nr_failed_migrations_hot : 0
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 0
se.nr_wakeups : 0
se.nr_wakeups_sync : 0
se.nr_wakeups_migrate : 0
se.nr_wakeups_local : 0
se.nr_wakeups_remote : 0
se.nr_wakeups_affine : 0
se.nr_wakeups_affine_attempts : 0
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.000001
avg_per_cpu : 0.000001
nr_switches : 0
nr_voluntary_switches : 0
nr_involuntary_switches : 0
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 87
__skb_recv_datagram---------
---------
Latency Top version : v0.1
16 15062 4955 futex_wait do_futex sys_futex system_call_fastpath
clamd (6072, #threads: 10)
---------------------------------------------------------
se.exec_start : 890682.642007
se.vruntime : 329161.740000
se.sum_exec_runtime : 480.929023
se.avg_overlap : 0.929824
se.wait_start : 0.000000
se.sleep_start : 890682.642007
se.block_start : 0.000000
se.sleep_max : 14582.725673
se.block_max : 9.018512
se.exec_max : 6.499281
se.slice_max : 27.999520
se.wait_max : 39.792243
se.wait_sum : 167.376729
se.wait_count : 340
sched_info.bkl_count : 0
se.nr_migrations : 11
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 255
se.nr_failed_migrations_hot : 7
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 4
se.nr_wakeups : 306
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 7
se.nr_wakeups_local : 96
se.nr_wakeups_remote : 210
se.nr_wakeups_affine : 7
se.nr_wakeups_affine_attempts : 300
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 1.466247
avg_per_cpu : 43.720820
nr_switches : 328
nr_voluntary_switches : 302
nr_involuntary_switches : 26
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 84
futex_wait---------
---------
Latency Top version : v0.1
11 10912 3779 sys_munmap system_call_fastpath
8 46486 19495 sys_mmap system_call_fastpath
6 13265 5138 do_page_fault error_exit
4 4142 2263 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
2 0 0 sys_mmap system_call_fastpath
clamd (6073, #threads: 10)
---------------------------------------------------------
se.exec_start : 891089.502763
se.vruntime : 297039.691398
se.sum_exec_runtime : 38531.939735
se.avg_overlap : 0.257175
se.wait_start : 891089.502774
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 56.856987
se.block_max : 39.998814
se.exec_max : 7.361909
se.slice_max : 32.002390
se.wait_max : 64.478851
se.wait_sum : 28485.466170
se.wait_count : 134840
sched_info.bkl_count : 0
se.nr_migrations : 807
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 6305
se.nr_failed_migrations_hot : 19174
se.nr_forced_migrations : 86
se.nr_forced2_migrations : 304
se.nr_wakeups : 118434
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 357
se.nr_wakeups_local : 26408
se.nr_wakeups_remote : 92026
se.nr_wakeups_affine : 357
se.nr_wakeups_affine_attempts : 118134
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.303195
avg_per_cpu : 47.747137
nr_switches : 127086
nr_voluntary_switches : 118139
nr_involuntary_switches : 8947
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 78
do_page_fault---------
---------
Latency Top version : v0.1
8 33623 10075 sys_mmap system_call_fastpath
4 126 126 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
10 8978 6660 sys_munmap system_call_fastpath
6 6322 5011 do_page_fault error_exit
3 1879 1879 sys_mmap system_call_fastpath
clamd (6074, #threads: 10)
---------------------------------------------------------
se.exec_start : 891094.957709
se.vruntime : 282081.330593
se.sum_exec_runtime : 37529.609679
se.avg_overlap : 0.128584
se.wait_start : 891094.957729
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 56.001313
se.block_max : 52.999336
se.exec_max : 8.328602
se.slice_max : 61.695609
se.wait_max : 78.999644
se.wait_sum : 27798.394839
se.wait_count : 133319
sched_info.bkl_count : 0
se.nr_migrations : 843
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 5888
se.nr_failed_migrations_hot : 20124
se.nr_forced_migrations : 83
se.nr_forced2_migrations : 307
se.nr_wakeups : 119224
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 362
se.nr_wakeups_local : 27409
se.nr_wakeups_remote : 91815
se.nr_wakeups_affine : 362
se.nr_wakeups_affine_attempts : 118907
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.296110
avg_per_cpu : 44.519109
nr_switches : 126742
nr_voluntary_switches : 118908
nr_involuntary_switches : 7834
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 78
0---------
---------
Latency Top version : v0.1
11 11662 6229 sys_munmap system_call_fastpath
10 48650 20169 sys_mmap system_call_fastpath
10 24384 7811 do_page_fault error_exit
clamd (6075, #threads: 10)
---------------------------------------------------------
se.exec_start : 891094.957715
se.vruntime : 325149.711380
se.sum_exec_runtime : 38581.104338
se.avg_overlap : 0.112403
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 891094.957715
se.sleep_max : 30.979799
se.block_max : 56.856984
se.exec_max : 7.517372
se.slice_max : 31.999387
se.wait_max : 60.001367
se.wait_sum : 28278.306752
se.wait_count : 133652
sched_info.bkl_count : 0
se.nr_migrations : 802
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 5425
se.nr_failed_migrations_hot : 19955
se.nr_forced_migrations : 77
se.nr_forced2_migrations : 277
se.nr_wakeups : 118420
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 322
se.nr_wakeups_local : 27118
se.nr_wakeups_remote : 91302
se.nr_wakeups_affine : 322
se.nr_wakeups_affine_attempts : 118161
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.304837
avg_per_cpu : 48.106115
nr_switches : 126563
nr_voluntary_switches : 118164
nr_involuntary_switches : 8399
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 78
do_page_fault---------
---------
Latency Top version : v0.1
10 15502 5352 sys_munmap system_call_fastpath
9 38554 20265 sys_mmap system_call_fastpath
7 9083 5138 do_page_fault error_exit
1 5011 5011 sys_mmap system_call_fastpath
4 3762 2641 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
clamd (6076, #threads: 10)
---------------------------------------------------------
se.exec_start : 891094.957721
se.vruntime : 329720.631055
se.sum_exec_runtime : 42341.424828
se.avg_overlap : 1.514241
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 891094.957721
se.sleep_max : 30.907363
se.block_max : 56.500704
se.exec_max : 7.507303
se.slice_max : 32.001234
se.wait_max : 59.986115
se.wait_sum : 29257.415262
se.wait_count : 133434
sched_info.bkl_count : 0
se.nr_migrations : 814
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 5043
se.nr_failed_migrations_hot : 20066
se.nr_forced_migrations : 94
se.nr_forced2_migrations : 231
se.nr_wakeups : 111450
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 284
se.nr_wakeups_local : 26837
se.nr_wakeups_remote : 84613
se.nr_wakeups_affine : 284
se.nr_wakeups_affine_attempts : 111188
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.344127
avg_per_cpu : 52.016492
nr_switches : 123040
nr_voluntary_switches : 111189
nr_involuntary_switches : 11851
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 83
do_page_fault---------
---------
Latency Top version : v0.1
11 34222 20131 sys_munmap system_call_fastpath
10 20073 7812 sys_mmap system_call_fastpath
2 3344 3344 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
7 9696 2999 do_page_fault error_exit
1 487 487 sys_mmap system_call_fastpath
clamd (6077, #threads: 10)
---------------------------------------------------------
se.exec_start : 891098.952122
se.vruntime : 282078.697244
se.sum_exec_runtime : 40209.531524
se.avg_overlap : 0.116129
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 28.499495
se.block_max : 51.003639
se.exec_max : 8.000313
se.slice_max : 31.999072
se.wait_max : 70.679248
se.wait_sum : 28364.256448
se.wait_count : 135183
sched_info.bkl_count : 0
se.nr_migrations : 776
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 5111
se.nr_failed_migrations_hot : 19975
se.nr_forced_migrations : 80
se.nr_forced2_migrations : 235
se.nr_wakeups : 114583
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 288
se.nr_wakeups_local : 26813
se.nr_wakeups_remote : 87770
se.nr_wakeups_affine : 288
se.nr_wakeups_affine_attempts : 114354
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.320261
avg_per_cpu : 51.816406
nr_switches : 125552
nr_voluntary_switches : 114359
nr_involuntary_switches : 11193
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 81
0---------
---------
Latency Top version : v0.1
10 8515 3778 sys_munmap system_call_fastpath
8 12692 3418 sys_mmap system_call_fastpath
2 0 0 do_page_fault error_exit shmem_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
7 9412 5506 do_page_fault error_exit
1 3239 3239 sys_mmap system_call_fastpath
1 1587 1587 sys_madvise system_call_fastpath
2 0 0 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
clamd (6078, #threads: 10)
---------------------------------------------------------
se.exec_start : 891094.957730
se.vruntime : 325213.210983
se.sum_exec_runtime : 38004.560148
se.avg_overlap : 2.244092
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 891094.957730
se.sleep_max : 40.291386
se.block_max : 59.495329
se.exec_max : 7.937506
se.slice_max : 39.969289
se.wait_max : 67.500878
se.wait_sum : 28689.272253
se.wait_count : 133749
sched_info.bkl_count : 0
se.nr_migrations : 891
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 4931
se.nr_failed_migrations_hot : 20678
se.nr_forced_migrations : 81
se.nr_forced2_migrations : 323
se.nr_wakeups : 117985
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 370
se.nr_wakeups_local : 28551
se.nr_wakeups_remote : 89434
se.nr_wakeups_affine : 370
se.nr_wakeups_affine_attempts : 117741
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.300516
avg_per_cpu : 42.653827
nr_switches : 126464
nr_voluntary_switches : 117738
nr_involuntary_switches : 8726
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 78
do_page_fault---------
---------
Latency Top version : v0.1
9 14443 5011 sys_mmap system_call_fastpath
4 8480 5136 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
10 17139 5993 sys_munmap system_call_fastpath
7 6619 2352 do_page_fault error_exit
1 0 0 sys_mmap system_call_fastpath
clamd (6079, #threads: 10)
---------------------------------------------------------
se.exec_start : 891094.957727
se.vruntime : 325200.342156
se.sum_exec_runtime : 39871.746222
se.avg_overlap : 0.700269
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 891094.957727
se.sleep_max : 29.363157
se.block_max : 58.999622
se.exec_max : 8.000502
se.slice_max : 49.687478
se.wait_max : 50.879058
se.wait_sum : 28190.321613
se.wait_count : 135340
sched_info.bkl_count : 0
se.nr_migrations : 791
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 6292
se.nr_failed_migrations_hot : 19989
se.nr_forced_migrations : 74
se.nr_forced2_migrations : 278
se.nr_wakeups : 115205
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 343
se.nr_wakeups_local : 27413
se.nr_wakeups_remote : 87792
se.nr_wakeups_affine : 343
se.nr_wakeups_affine_attempts : 114949
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.316658
avg_per_cpu : 50.406758
nr_switches : 125914
nr_voluntary_switches : 114946
nr_involuntary_switches : 10968
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 90
sys_munmap---------
---------
Latency Top version : v0.1
10 4854 1643 sys_munmap system_call_fastpath
10 31224 20031 sys_mmap system_call_fastpath
3 8985 5011 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
1 0 0 futex_wait do_futex sys_futex system_call_fastpath
5 2736 1837 do_page_fault error_exit
2 3906 3906 do_page_fault error_exit shmem_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
clamd (6080, #threads: 10)
---------------------------------------------------------
se.exec_start : 891094.957724
se.vruntime : 297044.689377
se.sum_exec_runtime : 39379.909583
se.avg_overlap : 0.264610
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 891094.957724
se.sleep_max : 31.055799
se.block_max : 58.999625
se.exec_max : 7.496151
se.slice_max : 72.000452
se.wait_max : 59.951360
se.wait_sum : 28778.446818
se.wait_count : 134612
sched_info.bkl_count : 0
se.nr_migrations : 817
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 4766
se.nr_failed_migrations_hot : 20436
se.nr_forced_migrations : 88
se.nr_forced2_migrations : 267
se.nr_wakeups : 116053
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 320
se.nr_wakeups_local : 27626
se.nr_wakeups_remote : 88427
se.nr_wakeups_affine : 320
se.nr_wakeups_affine_attempts : 115788
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.312553
avg_per_cpu : 48.200623
nr_switches : 125994
nr_voluntary_switches : 115786
nr_involuntary_switches : 10208
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 87
sys_mmap---------
Mon Aug 25 13:41:05 EEST 2008
[max 66.4msec] unmapping memory
- 4.01 msec (18.8%)
[max 58.6msec] Page fault
- 3.30 msec (59.0%)
[max 58.6msec] mmaping memory
- 4.35 msec (20.4%)
[max 27.3msec] sys_madvise system_call_fastpath - 5.40 msec (1.2%)
[max 25.3msec] Creating block layer request
- 25.27 msec (0.1%)
[max 8.8msec] mprotect() system call
- 8.79 msec (0.0%)
[max 4.5msec] Userspace lock contention
- 0.67 msec (0.3%)
[max 3.8msec] Submitting block IO
- 1.06 msec (0.0%)
[max 3.5msec] opening cdrom device
- 0.45 msec (0.0%)
[max 2.9msec] fsync() on a file
- 1.47 msec (0.0%)
[max 0.5msec] Removing directory
- 0.27 msec (0.0%)
[max 0.3msec] Closing a file
- 0.05 msec (0.0%)
[max 0.0msec] sys_brk system_call_fastpath - 0.00 msec (0.0%)
---------
Latency Top version : v0.1
clamd (5571, #threads: 19)
---------------------------------------------------------
se.exec_start : 931741.503387
se.vruntime : 366135.984717
se.sum_exec_runtime : 0.000000
se.avg_overlap : 0.000000
se.wait_start : 0.000000
se.sleep_start : 931741.503387
se.block_start : 0.000000
se.sleep_max : 0.000000
se.block_max : 0.000000
se.exec_max : 0.000000
se.slice_max : 0.000000
se.wait_max : 0.000000
se.wait_sum : 0.000000
se.wait_count : 0
sched_info.bkl_count : 0
se.nr_migrations : 0
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 0
se.nr_failed_migrations_hot : 0
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 0
se.nr_wakeups : 0
se.nr_wakeups_sync : 0
se.nr_wakeups_migrate : 0
se.nr_wakeups_local : 0
se.nr_wakeups_remote : 0
se.nr_wakeups_affine : 0
se.nr_wakeups_affine_attempts : 0
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.000001
avg_per_cpu : 0.000001
nr_switches : 0
nr_voluntary_switches : 0
nr_involuntary_switches : 0
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
__skb_recv_datagram---------
---------
Latency Top version : v0.1
clamd (6072, #threads: 19)
---------------------------------------------------------
se.exec_start : 921337.501698
se.vruntime : 357474.768314
se.sum_exec_runtime : 533.510071
se.avg_overlap : 5.630120
se.wait_start : 0.000000
se.sleep_start : 921337.501698
se.block_start : 0.000000
se.sleep_max : 16378.448918
se.block_max : 9.018512
se.exec_max : 6.499281
se.slice_max : 27.999520
se.wait_max : 39.792243
se.wait_sum : 190.393565
se.wait_count : 352
sched_info.bkl_count : 0
se.nr_migrations : 11
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 257
se.nr_failed_migrations_hot : 7
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 4
se.nr_wakeups : 317
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 7
se.nr_wakeups_local : 99
se.nr_wakeups_remote : 218
se.nr_wakeups_affine : 7
se.nr_wakeups_affine_attempts : 311
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 1.573776
avg_per_cpu : 48.500915
nr_switches : 339
nr_voluntary_switches : 313
nr_involuntary_switches : 26
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
futex_wait---------
---------
Latency Top version : v0.1
5 1944 907 futex_wait do_futex sys_futex system_call_fastpath
22 149759 57819 do_page_fault error_exit
1 0 0 sys_madvise system_call_fastpath
2 1928 964 sys_mmap system_call_fastpath
1 826 826 sys_munmap system_call_fastpath
clamd (6073, #threads: 19)
---------------------------------------------------------
se.exec_start : 941737.001727
se.vruntime : 369293.578220
se.sum_exec_runtime : 60400.545136
se.avg_overlap : 6.090918
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 157.455998
se.block_max : 59.207159
se.exec_max : 8.000502
se.slice_max : 103.917546
se.wait_max : 108.000687
se.wait_sum : 49523.405131
se.wait_count : 141028
sched_info.bkl_count : 0
se.nr_migrations : 867
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 6593
se.nr_failed_migrations_hot : 19659
se.nr_forced_migrations : 92
se.nr_forced2_migrations : 318
se.nr_wakeups : 122605
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 377
se.nr_wakeups_local : 27546
se.nr_wakeups_remote : 95059
se.nr_wakeups_affine : 377
se.nr_wakeups_affine_attempts : 122302
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.455221
avg_per_cpu : 69.666142
nr_switches : 132684
nr_voluntary_switches : 122307
nr_involuntary_switches : 10377
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 57
0---------
---------
Latency Top version : v0.1
4 4898 3906 futex_wait do_futex sys_futex system_call_fastpath
2 12205 12205 sys_mmap system_call_fastpath
19 52728 31249 do_page_fault error_exit
3 14773 7325 sys_munmap system_call_fastpath
2 4756 4391 sys_mmap system_call_fastpath
1 3904 3904 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
clamd (6074, #threads: 19)
---------------------------------------------------------
se.exec_start : 941737.501664
se.vruntime : 328090.268953
se.sum_exec_runtime : 57843.294472
se.avg_overlap : 2.129179
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 941737.501664
se.sleep_max : 139.916296
se.block_max : 52.999336
se.exec_max : 8.328602
se.slice_max : 61.695609
se.wait_max : 81.027965
se.wait_sum : 48893.020266
se.wait_count : 140200
sched_info.bkl_count : 0
se.nr_migrations : 897
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 6295
se.nr_failed_migrations_hot : 20615
se.nr_forced_migrations : 85
se.nr_forced2_migrations : 316
se.nr_wakeups : 123699
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 379
se.nr_wakeups_local : 28785
se.nr_wakeups_remote : 94914
se.nr_wakeups_affine : 379
se.nr_wakeups_affine_attempts : 123373
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.435478
avg_per_cpu : 64.485278
nr_switches : 132827
nr_voluntary_switches : 123376
nr_involuntary_switches : 9451
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
do_page_fault---------
---------
Latency Top version : v0.1
3 4896 3921 futex_wait do_futex sys_futex system_call_fastpath
3 7814 7814 sys_mmap system_call_fastpath
17 125178 58593 do_page_fault error_exit
5 11717 3907 sys_munmap system_call_fastpath
2 20017 19529 sys_mmap system_call_fastpath
1 0 0 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
clamd (6075, #threads: 19)
---------------------------------------------------------
se.exec_start : 941737.001727
se.vruntime : 369339.322708
se.sum_exec_runtime : 59325.744093
se.avg_overlap : 0.176192
se.wait_start : 941737.001727
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 297.024248
se.block_max : 59.999974
se.exec_max : 8.500719
se.slice_max : 140.654297
se.wait_max : 107.301024
se.wait_sum : 49163.396194
se.wait_count : 140275
sched_info.bkl_count : 0
se.nr_migrations : 872
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 5612
se.nr_failed_migrations_hot : 20612
se.nr_forced_migrations : 86
se.nr_forced2_migrations : 293
se.nr_wakeups : 122887
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 347
se.nr_wakeups_local : 28386
se.nr_wakeups_remote : 94501
se.nr_wakeups_affine : 347
se.nr_wakeups_affine_attempts : 122619
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.447768
avg_per_cpu : 68.034110
nr_switches : 132492
nr_voluntary_switches : 122622
nr_involuntary_switches : 9870
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
0---------
---------
Latency Top version : v0.1
1 0 0 futex_wait do_futex sys_futex system_call_fastpath
2 11718 8301 sys_mmap system_call_fastpath
28 86522 19040 do_page_fault error_exit
clamd (6076, #threads: 19)
---------------------------------------------------------
se.exec_start : 941761.501293
se.vruntime : 328405.149685
se.sum_exec_runtime : 63894.247770
se.avg_overlap : 1.548911
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 941761.501293
se.sleep_max : 143.499199
se.block_max : 56.500704
se.exec_max : 8.000810
se.slice_max : 63.943081
se.wait_max : 124.000596
se.wait_sum : 49831.782520
se.wait_count : 140785
sched_info.bkl_count : 0
se.nr_migrations : 893
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 5161
se.nr_failed_migrations_hot : 20439
se.nr_forced_migrations : 105
se.nr_forced2_migrations : 250
se.nr_wakeups : 114063
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 309
se.nr_wakeups_local : 27663
se.nr_wakeups_remote : 86400
se.nr_wakeups_affine : 309
se.nr_wakeups_affine_attempts : 113795
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.497537
avg_per_cpu : 71.550109
nr_switches : 128421
nr_voluntary_switches : 113795
nr_involuntary_switches : 14626
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 45
do_page_fault---------
---------
Latency Top version : v0.1
9 35633 19040 do_page_fault error_exit
5 51511 19511 sys_madvise system_call_fastpath
2 3620 3620 futex_wait do_futex sys_futex system_call_fastpath
4 12211 7814 sys_mmap system_call_fastpath
7 16104 3907 sys_munmap system_call_fastpath
3 23923 20505 sys_mmap system_call_fastpath
1 31249 31249 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
clamd (6077, #threads: 19)
---------------------------------------------------------
se.exec_start : 941761.502248
se.vruntime : 342292.045052
se.sum_exec_runtime : 62027.755009
se.avg_overlap : 16.009423
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 941761.502248
se.sleep_max : 282.119134
se.block_max : 51.003639
se.exec_max : 8.000708
se.slice_max : 47.973916
se.wait_max : 115.500667
se.wait_sum : 49175.485984
se.wait_count : 142922
sched_info.bkl_count : 0
se.nr_migrations : 847
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 5514
se.nr_failed_migrations_hot : 20415
se.nr_forced_migrations : 82
se.nr_forced2_migrations : 256
se.nr_wakeups : 119354
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 315
se.nr_wakeups_local : 28128
se.nr_wakeups_remote : 91226
se.nr_wakeups_affine : 315
se.nr_wakeups_affine_attempts : 119120
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.469263
avg_per_cpu : 73.232296
nr_switches : 132181
nr_voluntary_switches : 119127
nr_involuntary_switches : 13054
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
do_page_fault---------
---------
Latency Top version : v0.1
18 14999 12206 do_page_fault error_exit
12 23439 19044 sys_madvise system_call_fastpath
1 0 0 futex_wait do_futex sys_futex system_call_fastpath
clamd (6078, #threads: 19)
---------------------------------------------------------
se.exec_start : 941764.504558
se.vruntime : 342304.056898
se.sum_exec_runtime : 59142.145521
se.avg_overlap : 4.029801
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 130.026373
se.block_max : 59.495329
se.exec_max : 7.937506
se.slice_max : 47.987529
se.wait_max : 88.907165
se.wait_sum : 49416.291505
se.wait_count : 140949
sched_info.bkl_count : 0
se.nr_migrations : 960
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 5362
se.nr_failed_migrations_hot : 21172
se.nr_forced_migrations : 84
se.nr_forced2_migrations : 342
se.nr_wakeups : 122180
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 404
se.nr_wakeups_local : 29684
se.nr_wakeups_remote : 92496
se.nr_wakeups_affine : 404
se.nr_wakeups_affine_attempts : 121930
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.446133
avg_per_cpu : 61.606401
nr_switches : 132566
nr_voluntary_switches : 121927
nr_involuntary_switches : 10639
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 63
0---------
---------
Latency Top version : v0.1
1 2035 2035 do_page_fault error_exit
30 19092 3907 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
clamd (6079, #threads: 19)
---------------------------------------------------------
se.exec_start : 941761.502217
se.vruntime : 342472.045047
se.sum_exec_runtime : 61547.035391
se.avg_overlap : 15.077962
se.wait_start : 941761.502217
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 168.500075
se.block_max : 58.999622
se.exec_max : 8.000502
se.slice_max : 49.687478
se.wait_max : 92.000401
se.wait_sum : 50062.444615
se.wait_count : 141533
sched_info.bkl_count : 0
se.nr_migrations : 847
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 6707
se.nr_failed_migrations_hot : 20548
se.nr_forced_migrations : 80
se.nr_forced2_migrations : 292
se.nr_wakeups : 119240
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 358
se.nr_wakeups_local : 28569
se.nr_wakeups_remote : 90671
se.nr_wakeups_affine : 358
se.nr_wakeups_affine_attempts : 118978
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.468291
avg_per_cpu : 72.664740
nr_switches : 131429
nr_voluntary_switches : 118976
nr_involuntary_switches : 12453
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
0---------
---------
Latency Top version : v0.1
2 3906 3906 futex_wait do_futex sys_futex system_call_fastpath
5 66701 27634 sys_mmap system_call_fastpath
1 0 0 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
5 38081 19040 sys_munmap system_call_fastpath
15 114649 58593 do_page_fault error_exit
2 8243 8243 sys_mmap system_call_fastpath
1 0 0 sys_brk system_call_fastpath
clamd (6080, #threads: 19)
---------------------------------------------------------
se.exec_start : 941761.502244
se.vruntime : 372898.839430
se.sum_exec_runtime : 60636.590526
se.avg_overlap : 0.833244
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 941761.502244
se.sleep_max : 176.489855
se.block_max : 59.999483
se.exec_max : 8.000795
se.slice_max : 95.765041
se.wait_max : 110.998843
se.wait_sum : 50024.195476
se.wait_count : 141436
sched_info.bkl_count : 0
se.nr_migrations : 864
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 4924
se.nr_failed_migrations_hot : 20880
se.nr_forced_migrations : 92
se.nr_forced2_migrations : 276
se.nr_wakeups : 120249
se.nr_wakeups_sync : 2
se.nr_wakeups_migrate : 335
se.nr_wakeups_local : 28648
se.nr_wakeups_remote : 91601
se.nr_wakeups_affine : 335
se.nr_wakeups_affine_attempts : 119981
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.459723
avg_per_cpu : 70.181239
nr_switches : 131898
nr_voluntary_switches : 119979
nr_involuntary_switches : 11919
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
do_page_fault---------
---------
Latency Top version : v0.1
clamd (6488, #threads: 19)
---------------------------------------------------------
se.exec_start : 932061.002356
se.vruntime : 361680.992179
se.sum_exec_runtime : 20.193691
se.avg_overlap : 1.778925
se.wait_start : 0.000000
se.sleep_start : 932061.002356
se.block_start : 0.000000
se.sleep_max : 47.981172
se.block_max : 0.453532
se.exec_max : 3.996571
se.slice_max : 14.231404
se.wait_max : 65.459499
se.wait_sum : 242.627365
se.wait_count : 21
sched_info.bkl_count : 0
se.nr_migrations : 1
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 0
se.nr_failed_migrations_hot : 0
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 0
se.nr_wakeups : 10
se.nr_wakeups_sync : 1
se.nr_wakeups_migrate : 1
se.nr_wakeups_local : 4
se.nr_wakeups_remote : 6
se.nr_wakeups_affine : 1
se.nr_wakeups_affine_attempts : 10
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 1.009684
avg_per_cpu : 20.193691
nr_switches : 20
nr_voluntary_switches : 10
nr_involuntary_switches : 10
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
futex_wait---------
---------
Latency Top version : v0.1
12 6590 6348 sys_munmap system_call_fastpath
11 21773 15519 sys_mmap system_call_fastpath
5 47929 27559 do_page_fault error_exit
2 0 0 futex_wait do_futex sys_futex system_call_fastpath
1 0 0 sys_mmap system_call_fastpath
clamd (6489, #threads: 19)
---------------------------------------------------------
se.exec_start : 941761.502244
se.vruntime : 328631.656634
se.sum_exec_runtime : 1134.871842
se.avg_overlap : 0.000552
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 941761.502244
se.sleep_max : 46.675267
se.block_max : 59.999969
se.exec_max : 4.991282
se.slice_max : 24.002234
se.wait_max : 56.000427
se.wait_sum : 2612.148267
se.wait_count : 2598
sched_info.bkl_count : 0
se.nr_migrations : 69
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 66
se.nr_failed_migrations_hot : 78
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 37
se.nr_wakeups : 2355
se.nr_wakeups_sync : 1
se.nr_wakeups_migrate : 56
se.nr_wakeups_local : 1135
se.nr_wakeups_remote : 1220
se.nr_wakeups_affine : 56
se.nr_wakeups_affine_attempts : 2353
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.449632
avg_per_cpu : 16.447418
nr_switches : 2524
nr_voluntary_switches : 2352
nr_involuntary_switches : 172
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 63
sys_munmap---------
---------
Latency Top version : v0.1
31 150289 25449 do_page_fault error_exit
clamd (6490, #threads: 19)
---------------------------------------------------------
se.exec_start : 941761.502243
se.vruntime : 328571.656638
se.sum_exec_runtime : 2104.750095
se.avg_overlap : 0.004952
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 941761.502243
se.sleep_max : 49.174184
se.block_max : 51.989146
se.exec_max : 6.745074
se.slice_max : 30.408170
se.wait_max : 76.001241
se.wait_sum : 3630.921868
se.wait_count : 2117
sched_info.bkl_count : 0
se.nr_migrations : 53
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 77
se.nr_failed_migrations_hot : 89
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 18
se.nr_wakeups : 1466
se.nr_wakeups_sync : 1
se.nr_wakeups_migrate : 26
se.nr_wakeups_local : 700
se.nr_wakeups_remote : 766
se.nr_wakeups_affine : 26
se.nr_wakeups_affine_attempts : 1465
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 1.134024
avg_per_cpu : 39.712265
nr_switches : 1856
nr_voluntary_switches : 1463
nr_involuntary_switches : 393
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
sys_munmap---------
---------
Latency Top version : v0.1
10 27049 15616 sys_munmap system_call_fastpath
8 88470 27343 sys_madvise system_call_fastpath
5 15022 8788 sys_mmap system_call_fastpath
2 573 573 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
4 11834 11834 do_page_fault error_exit
1 4772 4772 sys_mmap system_call_fastpath
1 0 0 futex_wait do_futex sys_futex system_call_fastpath
clamd (6491, #threads: 19)
---------------------------------------------------------
se.exec_start : 941761.502246
se.vruntime : 372937.561308
se.sum_exec_runtime : 1956.099660
se.avg_overlap : 0.288143
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 941761.502246
se.sleep_max : 65.459500
se.block_max : 49.488078
se.exec_max : 4.371700
se.slice_max : 31.957703
se.wait_max : 73.001000
se.wait_sum : 3596.043882
se.wait_count : 2151
sched_info.bkl_count : 0
se.nr_migrations : 54
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 38
se.nr_failed_migrations_hot : 74
se.nr_forced_migrations : 8
se.nr_forced2_migrations : 17
se.nr_wakeups : 1445
se.nr_wakeups_sync : 1
se.nr_wakeups_migrate : 30
se.nr_wakeups_local : 583
se.nr_wakeups_remote : 862
se.nr_wakeups_affine : 30
se.nr_wakeups_affine_attempts : 1443
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 1.046602
avg_per_cpu : 36.224067
nr_switches : 1869
nr_voluntary_switches : 1441
nr_involuntary_switches : 428
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
sys_munmap---------
---------
Latency Top version : v0.1
6 4899 3921 futex_wait do_futex sys_futex system_call_fastpath
9 43003 9351 sys_mmap system_call_fastpath
4 5065 3536 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
8 28817 9175 sys_munmap system_call_fastpath
4 0 0 do_page_fault error_exit
clamd (6492, #threads: 19)
---------------------------------------------------------
se.exec_start : 941761.502203
se.vruntime : 342335.603991
se.sum_exec_runtime : 2616.754918
se.avg_overlap : 0.893570
se.wait_start : 941761.502233
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 37.463866
se.block_max : 67.997947
se.exec_max : 4.501504
se.slice_max : 29.002212
se.wait_max : 107.502983
se.wait_sum : 4099.174164
se.wait_count : 2148
sched_info.bkl_count : 0
se.nr_migrations : 39
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 30
se.nr_failed_migrations_hot : 84
se.nr_forced_migrations : 1
se.nr_forced2_migrations : 13
se.nr_wakeups : 1234
se.nr_wakeups_sync : 1
se.nr_wakeups_migrate : 21
se.nr_wakeups_local : 584
se.nr_wakeups_remote : 650
se.nr_wakeups_affine : 21
se.nr_wakeups_affine_attempts : 1230
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 1.503882
avg_per_cpu : 67.096279
nr_switches : 1740
nr_voluntary_switches : 1228
nr_involuntary_switches : 512
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
0---------
---------
Latency Top version : v0.1
1 15881 15881 sys_munmap system_call_fastpath
5 0 0 futex_wait do_futex sys_futex system_call_fastpath
25 34887 19533 do_page_fault error_exit
clamd (6493, #threads: 19)
---------------------------------------------------------
se.exec_start : 941761.502196
se.vruntime : 373078.839406
se.sum_exec_runtime : 2096.383223
se.avg_overlap : 0.076622
se.wait_start : 941761.502196
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 43.995613
se.block_max : 39.551665
se.exec_max : 4.978118
se.slice_max : 31.999444
se.wait_max : 96.010967
se.wait_sum : 3744.592128
se.wait_count : 2131
sched_info.bkl_count : 0
se.nr_migrations : 49
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 106
se.nr_failed_migrations_hot : 94
se.nr_forced_migrations : 1
se.nr_forced2_migrations : 16
se.nr_wakeups : 1511
se.nr_wakeups_sync : 1
se.nr_wakeups_migrate : 22
se.nr_wakeups_local : 728
se.nr_wakeups_remote : 783
se.nr_wakeups_affine : 22
se.nr_wakeups_affine_attempts : 1511
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 1.122260
avg_per_cpu : 42.783331
nr_switches : 1868
nr_voluntary_switches : 1506
nr_involuntary_switches : 362
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 54
0---------
---------
Latency Top version : v0.1
18 48286 12698 do_page_fault error_exit
3 4281 3795 sys_mmap system_call_fastpath
4 319 319 futex_wait do_futex sys_futex system_call_fastpath
4 17575 13181 sys_munmap system_call_fastpath
2 21 21 sys_mmap system_call_fastpath
clamd (6494, #threads: 19)
---------------------------------------------------------
se.exec_start : 941777.501631
se.vruntime : 373084.464784
se.sum_exec_runtime : 1563.026910
se.avg_overlap : 0.015511
se.wait_start : 941777.501631
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 27.054878
se.block_max : 59.999974
se.exec_max : 5.465053
se.slice_max : 20.613659
se.wait_max : 72.565150
se.wait_sum : 3116.116243
se.wait_count : 2314
sched_info.bkl_count : 0
se.nr_migrations : 50
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 103
se.nr_failed_migrations_hot : 87
se.nr_forced_migrations : 2
se.nr_forced2_migrations : 25
se.nr_wakeups : 1860
se.nr_wakeups_sync : 1
se.nr_wakeups_migrate : 36
se.nr_wakeups_local : 760
se.nr_wakeups_remote : 1100
se.nr_wakeups_affine : 36
se.nr_wakeups_affine_attempts : 1859
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.728683
avg_per_cpu : 31.260538
nr_switches : 2145
nr_voluntary_switches : 1852
nr_involuntary_switches : 293
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 63
0---------
---------
Latency Top version : v0.1
4 488 488 futex_wait do_futex sys_futex system_call_fastpath
10 97064 21850 sys_mmap system_call_fastpath
4 3496 2032 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
10 1683 1325 sys_munmap system_call_fastpath
2 16167 15609 do_page_fault error_exit
1 0 0 sys_mmap system_call_fastpath
clamd (6495, #threads: 19)
---------------------------------------------------------
se.exec_start : 941777.501641
se.vruntime : 342351.048003
se.sum_exec_runtime : 2022.330325
se.avg_overlap : 0.724541
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 0.000000
se.sleep_max : 67.497250
se.block_max : 59.994716
se.exec_max : 4.481811
se.slice_max : 31.998247
se.wait_max : 76.998763
se.wait_sum : 3130.901319
se.wait_count : 2311
sched_info.bkl_count : 0
se.nr_migrations : 50
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 66
se.nr_failed_migrations_hot : 48
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 22
se.nr_wakeups : 1713
se.nr_wakeups_sync : 1
se.nr_wakeups_migrate : 28
se.nr_wakeups_local : 851
se.nr_wakeups_remote : 862
se.nr_wakeups_affine : 28
se.nr_wakeups_affine_attempts : 1713
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.976971
avg_per_cpu : 40.446606
nr_switches : 2070
nr_voluntary_switches : 1703
nr_involuntary_switches : 367
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 63
0---------
---------
Latency Top version : v0.1
10 71007 31248 sys_mmap system_call_fastpath
3 28046 27559 do_page_fault error_exit generic_file_aio_read xfs_read xfs_file_aio_read do_sync_read vfs_read sys_read system_call_fastpath
11 63370 19505 sys_munmap system_call_fastpath
6 20488 19511 do_page_fault error_exit
1 19040 19040 sys_mmap system_call_fastpath
clamd (6496, #threads: 19)
---------------------------------------------------------
se.exec_start : 941777.501635
se.vruntime : 372958.839431
se.sum_exec_runtime : 1607.801008
se.avg_overlap : 0.238204
se.wait_start : 0.000000
se.sleep_start : 0.000000
se.block_start : 941777.501635
se.sleep_max : 40.001752
se.block_max : 51.326701
se.exec_max : 4.015841
se.slice_max : 29.956924
se.wait_max : 83.982555
se.wait_sum : 3023.210300
se.wait_count : 2241
sched_info.bkl_count : 0
se.nr_migrations : 66
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 32
se.nr_failed_migrations_hot : 79
se.nr_forced_migrations : 2
se.nr_forced2_migrations : 31
se.nr_wakeups : 1742
se.nr_wakeups_sync : 1
se.nr_wakeups_migrate : 47
se.nr_wakeups_local : 874
se.nr_wakeups_remote : 868
se.nr_wakeups_affine : 47
se.nr_wakeups_affine_attempts : 1739
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.779350
avg_per_cpu : 24.360621
nr_switches : 2063
nr_voluntary_switches : 1730
nr_involuntary_switches : 333
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 60
do_page_fault---------
Mon Aug 25 13:41:56 EEST 2008