[2.4.16] VM: Interesting copy behavior

From: Simon Kirby (sim@netnation.com)
Date: Thu Dec 06 2001 - 17:14:24 EST


While copying a filesystem from one hard drive to another on stock
2.4.16, I noticed some weird read starvation:

[sroot@r1:/root]# vmstat 1
   procs memory swap io system cpu
 r b w swpd free buff cache si so bi bo in cs us sy id
 0 1 2 0 3808 4908 97328 0 0 467 516 1228 213 4 15 81
 1 0 0 0 3824 5012 97216 0 0 1764 0 2935 191 1 27 72
 1 0 1 0 3696 5192 97156 0 0 8028 0 489 902 6 57 37
 1 0 0 0 3740 5436 96404 0 0 3360 0 632 1596 18 33 49
 1 0 0 0 3816 5772 95812 0 0 5228 0 688 1345 3 50 47
 2 0 0 0 3816 6048 95356 0 0 4020 0 637 1189 9 26 65
 1 0 1 0 3596 6304 95188 0 0 2800 0 607 854 2 22 76
 1 0 0 0 3732 6520 95576 0 0 2568 0 540 757 0 30 70
 1 0 0 0 3800 6636 95372 0 0 3504 0 826 1667 16 40 44
 2 0 0 0 3812 6732 95224 0 0 2956 0 950 2061 21 33 46
 1 0 0 0 3780 6820 94740 0 0 3548 0 1102 2648 29 57 14
 0 1 1 0 3820 6844 94248 0 0 1196 8544 2125 1326 13 34 53
 0 1 2 0 3812 6844 94256 0 0 4 476 6018 15 1 42 57
 0 1 2 0 3812 6844 94256 0 0 0 3092 7890 12 1 48 51
 0 1 2 0 3812 6844 94256 0 0 0 3236 6801 14 2 40 58
 0 1 2 0 3812 6844 94256 0 0 0 3784 5617 12 0 35 65
 0 1 2 0 3812 6844 94256 0 0 0 3424 5579 12 1 39 60
 0 1 2 0 3812 6844 94256 0 0 0 3736 6691 12 0 42 58
 0 1 2 0 3812 6844 94256 0 0 0 3612 6839 14 0 42 58
 0 1 2 0 3808 6848 94256 0 0 4 3684 6865 12 0 38 62
 0 1 2 0 3808 6848 94256 0 0 0 0 5760 8 0 40 60
 0 1 2 0 3788 6848 94268 0 0 8 3404 4345 23 1 29 70
 0 1 2 0 3808 6852 94228 0 0 48 1408 3376 80 2 20 78
 0 1 2 0 3780 6852 94256 0 0 12 2052 5278 34 0 28 72
 0 1 2 0 3780 6852 94256 0 0 0 3592 5863 12 0 28 72
 0 1 2 0 3776 6856 94256 0 0 4 3820 7082 12 0 40 60
 0 1 1 0 3804 6860 94204 0 0 44 2200 4350 70 1 29 70
 1 0 1 0 3816 6864 94172 0 0 52 0 3600 86 1 22 77
 0 1 1 0 3796 6864 94172 0 0 64 0 3537 106 0 21 79
 1 0 1 0 3832 6892 94088 0 0 116 0 3279 151 1 18 81
 1 0 0 0 3736 7104 93544 0 0 1412 0 984 1315 12 32 56
 2 0 0 0 3784 7188 92464 0 0 2620 0 1159 2811 40 41 19
 2 0 0 0 3812 7248 91720 0 0 1864 0 890 1955 19 42 39
 2 0 0 0 3748 7308 91068 0 0 1780 0 950 2161 23 35 42
   procs memory swap io system cpu
 r b w swpd free buff cache si so bi bo in cs us sy id
 3 0 0 0 3752 7368 90368 0 0 1764 0 969 2150 25 41 34
 1 0 1 0 3792 7676 89720 0 0 1144 0 1086 1160 12 25 64
 1 0 0 0 3728 7768 89384 0 0 2100 0 643 1236 8 25 67
 1 1 0 0 3800 7892 88848 0 0 2376 0 640 1291 14 25 61
 2 0 0 0 3800 7916 88468 0 0 1888 0 652 1339 13 21 67
 2 0 0 0 3784 7976 87948 0 0 3448 0 926 1903 19 37 44
 0 1 1 0 3748 8172 88140 0 0 6172 0 1017 1951 9 65 26
 1 0 0 0 3728 8328 87868 0 0 980 0 689 705 7 18 75
 1 0 0 0 3792 8416 87404 0 0 2528 0 728 1440 12 32 56
 1 1 0 0 3712 8576 87032 0 0 5156 0 837 1737 15 42 43
 1 0 0 0 3768 8720 86488 0 0 3660 0 749 1528 10 34 56
 2 0 0 0 3752 8800 85980 0 0 3264 0 849 1720 16 38 46
 2 0 1 0 3728 8948 85628 0 0 1576 0 751 892 6 22 73
 2 0 0 0 3724 9048 85208 0 0 1864 0 674 1226 7 27 66
 1 0 0 0 3744 9172 84724 0 0 2556 0 723 1398 12 33 55
 0 1 1 0 3812 9204 84516 0 0 932 6424 3303 478 4 25 71
 0 1 1 0 3812 9204 84516 0 0 0 3648 5929 11 0 39 61
 0 1 2 0 3812 9204 84516 0 0 0 3664 5492 13 0 30 70
 0 1 2 0 3812 9204 84516 0 0 0 3432 7031 12 0 38 62
 1 0 2 0 3808 9208 84516 0 0 4 3560 5324 13 0 36 64
 0 1 2 0 3808 9208 84516 0 0 0 0 4766 9 0 28 72
 0 1 2 0 3808 9208 84516 0 0 0 3924 6504 11 0 41 59
 0 1 2 0 3804 9212 84516 0 0 4 3896 6120 15 0 37 63
 0 1 2 0 3800 9216 84516 0 0 4 2400 5184 12 0 36 64
 0 1 2 0 3800 9216 84516 0 0 0 3804 7945 13 0 52 48
 0 1 2 0 3800 9216 84516 0 0 0 3724 6473 11 0 43 57
 0 1 2 0 3784 9224 84516 0 0 8 3408 6470 20 0 39 61
 0 1 2 0 3772 9228 84524 0 0 8 1312 5903 14 0 32 68
 0 1 2 0 3772 9228 84524 0 0 0 2884 5464 13 1 32 67
 0 1 2 0 3760 9228 84532 0 0 4 3524 5184 14 0 40 60
 0 1 2 0 3756 9228 84532 0 0 0 164 4897 14 1 24 75
 0 1 2 0 3712 9236 84548 0 0 80 108 3071 114 1 20 79
 0 1 1 0 3788 9240 84440 0 0 80 0 3147 119 2 18 80
 1 0 0 0 3824 9308 84144 0 0 624 0 2018 654 7 25 69
   procs memory swap io system cpu
 r b w swpd free buff cache si so bi bo in cs us sy id
 2 0 0 0 3748 9388 84424 0 0 2208 0 630 1248 10 29 61
 1 0 0 0 3792 9444 84068 0 0 2696 0 685 1345 13 26 61
 1 0 0 0 3732 9504 83816 0 0 1284 0 557 1080 7 19 75
 2 0 0 0 3772 9588 83248 0 0 1976 0 809 1812 18 33 49
 1 1 1 0 3780 9652 82656 0 0 2180 0 957 1983 17 48 35
 2 0 0 0 3772 9872 81980 0 0 2560 0 1078 1739 23 36 41
 2 0 0 0 3792 9948 81216 0 0 2916 0 982 2219 19 49 32
 1 0 0 0 3768 10032 79808 0 0 1684 0 1026 2628 30 65 5
 2 0 0 0 3716 10112 78932 0 0 1880 0 974 2237 27 59 14
 2 0 0 0 3800 10212 78152 0 0 2888 0 874 2019 13 42 45
 0 1 0 0 3764 10600 77436 0 0 3804 0 1367 1441 11 49 40
 1 0 0 0 3796 10732 76984 0 0 4796 0 788 1585 12 47 41
 1 0 0 0 3772 10760 76608 0 0 1716 0 668 1368 10 25 66
 2 0 1 0 5308 10652 76340 0 0 2708 0 655 1353 6 35 59
 2 0 0 0 3816 10768 79596 0 0 1744 0 701 1411 10 34 56
 1 0 0 0 3764 10732 79676 0 0 1672 8 910 1145 9 28 63
 2 0 0 0 3724 10464 79924 0 0 3544 0 926 1886 18 43 39
 1 0 1 0 3700 10492 79464 0 0 3412 0 1082 2242 18 48 34
 1 0 0 0 3704 10204 79564 0 0 3012 384 912 1210 3 29 68
 1 0 1 0 3820 10192 79460 0 0 4 7796 4594 16 2 26 72
 0 1 2 0 3820 10192 79460 0 0 0 3280 5765 16 1 43 56
 0 1 2 0 3820 10192 79460 0 0 0 0 4876 8 1 33 66
 0 1 2 0 3820 10192 79460 0 0 0 3832 5544 12 1 27 72
 0 1 2 0 3820 10192 79460 0 0 0 3688 7937 12 0 52 48
 0 1 2 0 3820 10192 79460 0 0 0 3712 6304 14 0 39 61
 0 1 2 0 3820 10192 79460 0 0 0 3824 6924 12 1 48 51
 0 1 2 0 3816 10196 79460 0 0 4 3644 5834 12 0 45 55
 0 1 2 0 3816 10196 79460 0 0 0 3640 5748 12 1 42 57
 0 1 2 0 3816 10196 79460 0 0 0 3984 6107 10 0 45 55
 0 1 2 0 3812 10200 79460 0 0 4 132 6522 14 1 48 51
 0 1 2 0 3796 10204 79472 0 0 12 5376 6622 23 0 38 62
 0 1 2 0 3740 10204 79524 0 0 24 804 5191 28 0 33 67
 0 1 2 0 3740 10204 79524 0 0 0 3676 5710 11 0 44 56
 0 1 2 0 3804 10192 79468 0 0 32 2160 4472 29 1 26 73
   procs memory swap io system cpu
 r b w swpd free buff cache si so bi bo in cs us sy id
 0 1 1 0 3808 10184 79456 0 0 120 0 3530 85 1 15 84
 0 1 1 0 3740 10168 79532 0 0 96 0 3888 83 2 20 78
 1 0 1 0 3816 10180 79436 0 0 292 0 3088 97 1 20 79
 2 0 0 0 3768 9952 79488 0 0 3248 0 814 1297 9 33 58
 2 0 0 0 3792 9924 79008 0 0 3556 0 1014 2322 19 45 36
 1 0 0 0 3780 9908 78532 0 0 3388 0 1062 2435 18 47 35
 1 2 0 0 3804 9980 79224 0 0 2848 0 893 1948 12 45 43
 1 0 0 0 3732 10012 78996 0 0 912 0 469 883 7 11 82
 1 1 0 0 3716 10248 78672 0 0 588 0 725 547 9 10 81
 1 0 0 0 3824 10292 78352 0 0 656 0 420 787 11 6 83
 1 2 0 0 3768 10316 78204 0 0 628 0 398 711 14 7 79
 2 0 0 0 3796 10348 77824 0 0 468 32 325 418 7 10 83
 0 1 0 0 3760 10356 77192 0 0 748 0 382 552 17 14 70
 1 0 1 0 3740 10460 76992 0 0 912 0 492 714 4 7 89
 2 0 0 0 3820 10476 76708 0 0 2004 0 631 1044 11 25 65
 1 0 0 0 3744 10448 76584 0 0 5056 0 714 1409 10 45 45
 2 0 0 0 3716 10500 76276 0 0 3360 0 784 1573 7 38 55
 1 0 0 0 3748 10556 75872 0 0 3296 0 843 1713 14 36 50
 2 0 0 0 3796 10540 75376 0 0 2776 0 991 2052 14 49 37
 2 0 0 0 3708 10576 74464 0 0 1716 0 815 888 19 23 59
 1 0 0 0 3736 10496 75000 0 0 1828 0 638 1245 25 33 42
 2 0 0 0 3816 10384 75916 0 0 2732 0 986 2048 19 48 33
 1 0 0 0 3760 10140 75744 0 0 2700 8 1024 2127 18 51 31
 2 0 0 0 3788 9920 75536 0 0 1736 0 817 1617 22 33 45
 1 0 1 0 3784 9988 75224 0 0 1468 0 952 1115 12 27 61
 2 0 0 0 3760 9692 75092 0 0 3144 0 894 1798 13 45 42
 0 1 1 0 3816 9392 75116 0 0 964 5728 3777 770 9 32 59
 2 1 1 0 4344 9336 74300 0 0 24 3384 4723 90 5 28 67
 0 1 1 0 4744 9320 74236 0 0 40 104 5081 40 11 33 56
 0 1 1 0 4568 9328 74380 0 0 80 0 3580 119 1 22 77
 1 0 0 0 3724 9396 75072 0 0 412 0 2515 328 4 20 76
 0 1 0 0 3768 9372 74604 0 0 1900 0 747 1497 9 35 56
 2 0 0 0 3792 9324 74364 0 0 764 0 498 889 11 18 72
 1 0 0 0 3760 9312 74172 0 0 680 0 456 801 9 16 75
   procs memory swap io system cpu
 r b w swpd free buff cache si so bi bo in cs us sy id
 1 1 0 0 3732 9312 73908 0 0 848 0 525 988 11 17 73
 1 1 1 0 3820 9380 73532 0 0 536 0 517 702 5 12 83
 0 1 0 0 3804 9440 73460 0 0 156 4096 3045 174 0 24 76
 0 1 0 0 3752 9440 73488 0 0 80 0 4798 121 2 24 75
 1 0 0 0 3708 9464 73224 0 0 412 0 1223 715 11 11 78
 1 0 0 0 5156 9444 72980 0 0 1160 0 539 1010 12 23 66
 0 1 0 0 3792 9440 74348 0 0 2952 512 1169 1250 11 31 58
 0 1 2 0 3456 9408 74704 0 0 416 856 1277 247 4 12 84
 1 1 2 0 3236 9376 74944 0 0 240 376 1417 236 2 16 82
 0 1 1 0 3516 9264 74764 0 0 308 1424 1433 235 3 14 83
 0 1 1 0 3736 9236 74564 0 0 332 1632 1793 189 1 10 89
 0 1 1 0 3816 9240 74476 0 0 164 820 3355 107 0 15 85
 1 0 1 0 3828 9244 74456 0 0 112 476 3084 85 0 18 82
 1 0 1 0 3776 9300 74448 0 0 256 572 1925 141 2 15 83
 1 0 1 0 3740 9292 74492 0 0 148 1624 803 72 1 5 94
 1 0 1 0 3736 9272 74488 0 0 448 1680 1189 265 3 13 84
 1 0 2 0 3776 9236 74476 0 0 156 876 3606 67 0 22 78
 1 0 2 0 3020 9248 75180 0 0 356 144 1278 229 1 7 92
 0 1 2 0 2828 9232 75360 0 0 136 836 1952 144 0 15 85
 0 1 2 0 3504 9180 74736 0 0 0 3496 3602 22 0 25 75
 0 1 2 0 3504 9180 74736 0 0 0 6060 4801 18 1 26 73
 1 1 1 0 4776 9152 73484 0 0 0 2896 4467 57 21 44 35
 0 1 1 0 4776 9152 73484 0 0 0 0 5704 11 0 40 60

It looks like maybe pages of "cp" itself might be getting thrown out and
then it isn't able to run because the writes are starving the reads.

I did a "ps -eo pid,args,wchan" during the last few times when "cp" was
stuck: the wchan appeared to be "get_request_wait".

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ sim@stormix.com ][ sim@netnation.com ]
[ Opinions expressed are not necessarily those of my employers. ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/



This archive was generated by hypermail 2b29 : Fri Dec 07 2001 - 21:00:35 EST