Re: 2.6.0-test4 parallel seek & read problem

From: Magnus Andersson
Date: Mon Sep 29 2003 - 09:57:23 EST

Next message: Jörn Engel: "Re: [PATCH] check headers for complete includes, etc."
Previous message: Leonard Milcin Jr.: "Re: Can't X be elemenated?"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hello again!

I have some more information for you all regarding this issue.

The problem with my server only able to handle 75 connections/sec was
solved by the 'as-initial-io-noantic' patch from Nick Piggin,
attached to this email if you want to try it out.
Looks like 2.6.0-test6 contains a modified version of this patch,
this modified version doesn't work well, the first patch was better.

With the original patch applied to -test4 or -test5, my server is
only able to handle half the amount of connections/sec as it should
be able to handle.

It doesn't matter which elevator I run. as-, deadline-, and noop-
all have the same problem.

The reason for this performance loss is that my read requests get
splitted by the kernel into those 4096 bytes chunks. Later, these
chunks are never merged by the kernel. In other words: 1 seek&read
issued by me, goes to the disk as 2 separate seek&read. Between these
2 seek&read are a lot of other seeks&reads issued by other threads.

I put this printk() after the call to elv_next_request(),
inside scsi_request_fn(), in scsi_lib.c, to see what is going on.

if(req) // 2 lines added by me
printk("SEC %d\n", req->sector);

The output from dmesg is attached to this email, in the file
printk.output. There you can see that the requests were splitted and
not merged. The splitted chunks are not even sent following each
other, there are a lot of other requests in between, resulting in a
lot of extra head movements.

I tried to disable readahead with 'blockdev --setra 0 /dev/sda',
but still the same problem, and I tried to raise readahead, but
no luck there too.

If the file I read from is opened with the flag O_DIRECT, everything
is good. All 3 elevators performs at a maximum rate.

Attached to this email is also a test program, called thread_seek,
for you all to try out.

The test program keeps nthreads (64 default) running at the same time.
The threads do 1 seek(), a READSIZE (4096 default) bytes read(), and
exits. The file it is operating on is "./bigfile"

Important is the choise of filesystem, choose one with extends. JFS
and XFS is two I know about.

I don't know if the raid system is part of the problem, but I don't
think so. Feel free to try on a different kind of disk setup.

Make sure the bigfile is much bigger than your memory to avoid cache hits.
Do a umount & mount between tests to clear the memory cache if you
feel that it is needed.

compiling: gcc -o thread_seek thread_seek.c -lpthread
running: ./thread_seek [nthreads]

Here are the commands I use to see what is going on.

2.4.22:
'sar -c 1 0'
'sar -b 1 0'

2.6.0-test4:
'sar -c 1 0'
'iostat -x 1'

What you should see on 2.4 is that the output from the two sar should
be quite equal. With kernel 2.4 for example, I see 500
processes/second created, and 500 reads/second going to the disk.
With 2.6 I see 260 processes/second created, and 500 reads/second
going to the disk. The values you will see depends on your setup. But
the ratio between processes/second & reads/second should be 1. On 2.4
it is 1, and on 2.6 it is 0.52, wich is not good. Should be 1 here
too.

Setting READSIZE to 1 byte gives no performance loss. The server on 2.6
is able to handle 500 proc/second & 500 reads/second.

I hope someone is able to reproduce this behavior. Feel free
to ask for more information if I forgot to say something
you need to know.

I also found strange stuff inside elevator_noop_add_request()
list_add_tail(&rq->queuelist, &q->queue_head);
should probably be changed to:
list_add_tail(&rq->queuelist, insert);

/Magnus

On Tue, Aug 26, 2003 at 07:32:41PM +0200, Magnus Andersson wrote:
> Hello everyone!
>
> I have run into some strange behavior with kernel 2.6.0-test4.
>
> I'm doing a lot of parallel random seeks and reads in a 128 Gigabyte
> file. The file is located on a JFS partition on a 3ware Storage
> Controller containing four IDE disks with an average seek time of
> 8.5 ms. The 3ware controller is configured as RAID0 with stripe size 1MB.
>
> The operating system is Debian GNU/Linux, version testing.
>
> Running on this system is a program listening for connections. When a
> connection is accepted, a thread is created. This thread will get a
> free file descriptor from a pool of already opened file desriptors.
> The thread will do a random seek on the fd, and a small read, around 3 Kb.
> This data is then sent to the client and the connection is closed.
>
> My program is linked agains libpthread.a from the libc6-dev package,
> version 2.3.1-16, in Debian.
>
> If I'm running kernel 2.4.21 or 2.4.22, I can connect to the program
> 500 times per second. Using iostat-4.1.5 to monitor the system I see
> 500 reads/second issued to the disk. So far so fine.
>
> If I change to 2.6.0-test4 I can only connect to the program around 75
> times per second. During this time iostat reports around 120
> reads/second issued to the disk.
>
> If I change the elevator to 'deadline' or 'noop' I can connect to the
> program around 300 times/second. Iostat reports around 520
> reads/second issued to the disk.
>
> I also have a modified version of my program using nonblocking network
> code, and aio. This time I'm using the same fd for all aio
> requests. The file is opened with the flag O_DIRECT, bypassing the
> buffer cache. This time I'm able to reach close to 500 requests/second.
>
> 2.6.0-test1-ac3, and 2.6.0-test3-mm1 are behaving the same as 2.6.0-test4.
>
> Someone have any idea what is going on? Seems like something is not
> what it should be. My hope is to be able to reach the same amounts of
> connections/second as with the 2.4 kernels.
>
> I can't provide the source to the program, because it is owned by
> someone else, but I hope someone can point me in the right direction
> for further investigation of this problem.
>
> Below are the output of dmesg and the kernel config file for 2.6.0-test4.
>
> Best regards
>
> /Magnus

LocalWords: nthreads READSIZE
--- linux-2.6/drivers/block/as-iosched.c.orig 2003-08-27 19:15:24.000000000 +1000
+++ linux-2.6/drivers/block/as-iosched.c 2003-08-27 19:18:13.000000000 +1000
@@ -709,6 +709,14 @@ static int as_can_break_anticipation(str
return 1;
}

+ if (aic->seek_samples == 0 || aic->ttime_samples == 0) {
+ /*
+ * Process has just started IO so default to not anticipate.
+ * Maybe should be smarter.
+ */
+ return 1;
+ }
+
if (aic->ttime_mean > ad->antic_expire) {
/* the process thinks too much between requests */
return 1;
SEC 172051559
SEC 13813847
SEC 51237615
SEC 257744159
SEC 53686847
SEC 137177695
SEC 257744167
SEC 83426135
SEC 205555351
SEC 135267503
SEC 23309303
SEC 83426143
SEC 169104455
SEC 61499679
SEC 148370231
SEC 92443095
SEC 131900239
SEC 82034119
SEC 272031151
SEC 230777247
SEC 100578959
SEC 233801887
SEC 82034127
SEC 66901983
SEC 256493039
SEC 22051087
SEC 272031159
SEC 273658927
SEC 105083095
SEC 94483935
SEC 263437351
SEC 68318911
SEC 144796511
SEC 22051095
SEC 222388159
SEC 270802071
SEC 103099303
SEC 76072159
SEC 263437359
SEC 239927103
SEC 246605751
SEC 44445911
SEC 147363839
SEC 120025063
SEC 94075375
SEC 222388167
SEC 20386855
SEC 127551663
SEC 270802079
SEC 119170007
SEC 135267511
SEC 196171671
SEC 13913719
SEC 216117327
SEC 66066511
SEC 104923519
SEC 240880119
SEC 173821935
SEC 41347055
SEC 239927111
SEC 207128775
SEC 58956727
SEC 60025927
SEC 94075383
SEC 119170015
SEC 184290999
SEC 230666983
SEC 66066519
SEC 138100855
SEC 255878895
SEC 234213607
SEC 131900247
SEC 58956735
SEC 126403215
SEC 15054079
SEC 24817471
SEC 204953343
SEC 236541455
SEC 43407967
SEC 180961311
SEC 54968479
SEC 120829735
SEC 43407975
SEC 9785215
SEC 254416295
SEC 40044159
SEC 232982767
SEC 260532927
SEC 246605759
SEC 242248263
SEC 14644367
SEC 255878903
SEC 95238295
SEC 15054087
SEC 270508199
SEC 20386863
SEC 247261103
SEC 95238303
SEC 161708527
SEC 235662823
SEC 41347063
SEC 59500591
SEC 161708535
SEC 147308535
SEC 9785223
SEC 230391735
SEC 54968487
SEC 182184839
SEC 147308543
SEC 104923527
SEC 183982231
SEC 107056799
SEC 13486631
SEC 13913727
SEC 230391743
SEC 174778119
SEC 44445919
SEC 183982239
SEC 97193383
SEC 204899303
SEC 126403223
SEC 246319231
SEC 24817479
SEC 271034303
SEC 60025935
SEC 243920143
SEC 61499687
SEC 138100863
SEC 66901991
SEC 180961319
SEC 91056623
SEC 68318919
SEC 40044167
SEC 13327079
SEC 15026039
SEC 183837607
SEC 209808183
SEC 94483943
SEC 13327087
SEC 221087743
SEC 137177703
SEC 195626039
SEC 72230487
SEC 144796519
SEC 164200583
SEC 228672047
SEC 147363847
SEC 172051567
SEC 170040023
SEC 73980887
SEC 191188727
SEC 148370239
SEC 8945447
SEC 169104463
SEC 84774735
SEC 170040031
SEC 240748527
SEC 174479871
SEC 207128783
SEC 73980895
SEC 204953351
SEC 164200591
SEC 8945455
SEC 108952063
SEC 189060039
SEC 174778127
SEC 144553111
SEC 233801895
SEC 174479879
SEC 218810311
SEC 107952551
SEC 240880127
SEC 100385047
SEC 183837615
SEC 108952071
SEC 151077735
SEC 54418487
SEC 209965591
SEC 107952559
SEC 217406503
SEC 195626047
SEC 11099095
SEC 260532935
SEC 204899311
SEC 202259247
SEC 54418495
SEC 126854047
SEC 218810319
SEC 48845551
SEC 273658935
SEC 217406511
SEC 13486639
SEC 247189359
SEC 211302215
SEC 15038279
SEC 228672055
SEC 11099103
SEC 238555119
SEC 38489039
SEC 240748535
SEC 59500599
SEC 119050591
SEC 119448951
SEC 202259255
SEC 188781391
SEC 86228223
SEC 239900503
SEC 65671023
SEC 169764335
SEC 195576463
SEC 145282799
SEC 247261111
SEC 86228231
SEC 175043159
SEC 93345735
SEC 84774743
SEC 210166407
SEC 91056631
SEC 108048151
SEC 270508207
SEC 97193391
SEC 107056807
SEC 271034311
SEC 120829743
SEC 144553119
SEC 143163055
SEC 242758527
SEC 15026047
SEC 60004495
SEC 190250447
SEC 143163063
SEC 97230359
SEC 189060047
SEC 121631215
SEC 60004503
SEC 72230495
SEC 182184847
SEC 134023007
SEC 209965599
SEC 191188735
SEC 190250455
SEC 259782111
SEC 100385055
SEC 144318647
SEC 138620911
SEC 97230367
SEC 49217207
SEC 57249663
SEC 192911959
SEC 209808191
SEC 49217215
SEC 221087751
SEC 119448959
SEC 15038287
SEC 126854055
SEC 192911967
SEC 166672791
SEC 165928935
SEC 217907079
SEC 48845559
SEC 151077743
SEC 166672799
SEC 235662831
SEC 165928943
SEC 93345743
SEC 24636871
SEC 76173815
SEC 119050599
SEC 217907087
SEC 252238983
SEC 126251575
SEC 157247391
SEC 243920151
SEC 252238991
SEC 53107823
SEC 117011503
SEC 169764343
SEC 246319239
SEC 121631223
SEC 247189367
SEC 93171375
SEC 53107831
SEC 271004903
SEC 190849287
SEC 222187279
SEC 64662383
SEC 145282807
SEC 195576471
SEC 105185631
SEC 175043167
SEC 139463215
SEC 173512887
SEC 188781399
SEC 24636879
SEC 165636287
SEC 210166415
SEC 173512895
SEC 61488783
SEC 71309455
SEC 38489047
SEC 110441303
SEC 211302223
SEC 269598351
SEC 57249671
SEC 61488791
SEC 225573535
SEC 161166039
SEC 239900511
SEC 64662391
SEC 242758535
SEC 71309463
SEC 43103079
SEC 65671031
SEC 259782119
SEC 124095087
SEC 269598359
SEC 262065111
SEC 154509991
SEC 271004911
SEC 225573543
SEC 269534607
SEC 38520175
SEC 35937271
SEC 239653151
SEC 93171383
SEC 133366335
SEC 109896895
SEC 35937279
SEC 67189407
SEC 110789831
SEC 67189415
SEC 158897095
SEC 39137751
SEC 102212719
SEC 243041263
SEC 108048159
SEC 40070927
SEC 138620919
SEC 43103087
SEC 247942095
SEC 139463223
SEC 189677711
SEC 120256959
SEC 105185639
SEC 133415759
SEC 76173823
SEC 120256967
SEC 167801095
SEC 109896903
SEC 125278847
SEC 110789839
SEC 125278855
SEC 81100167
SEC 142417903
SEC 59901551
SEC 164916039
SEC 222187287
SEC 249394783
SEC 81100175
SEC 110441311
SEC 117011511
SEC 133366343
SEC 142417911
SEC 140703607
SEC 220075063
SEC 82654663
SEC 176750447
SEC 134023015
SEC 176750455
SEC 69511239
SEC 213379951
SEC 238555127
SEC 156818855
SEC 126251583
SEC 144318655
SEC 242775487
SEC 12593943
SEC 156818863
SEC 248105967
SEC 135122143
SEC 158897103
SEC 267063343
SEC 154509999
SEC 161166047
SEC 167801103
SEC 130798751
SEC 157247399
SEC 275665679
SEC 269397391
SEC 209151967
SEC 65434487
SEC 190849295
SEC 40070935
SEC 60524287
SEC 217823271
SEC 55138759
SEC 255785615
SEC 239653159
SEC 75091975
SEC 195973959
SEC 235601903
SEC 217823279
SEC 269534615
SEC 45535367
SEC 69511247
SEC 212791751
SEC 60561175
SEC 55138767
SEC 38520183
SEC 60404503
SEC 128105607
SEC 59901559
SEC 124095095
SEC 255785623
SEC 60524295
SEC 39137759
SEC 135122151
SEC 235601911
SEC 267204439
SEC 194999175
SEC 219305271
SEC 102212727
SEC 231063359
SEC 45535375
SEC 188652527
SEC 128105615
SEC 20506983
SEC 268983959
SEC 114516943
SEC 143827151
SEC 188652535
SEC 60404511
SEC 251567439
SEC 60561183
SEC 164916047
SEC 65434495
SEC 231063367
SEC 35044399
SEC 165636295
SEC 98201191
SEC 133415767
SEC 268983967
SEC 67384351
SEC 75091983
SEC 48976863
SEC 213379959
SEC 138640623
SEC 251567447
SEC 82654671
SEC 260765967
SEC 176792423
SEC 35044407
SEC 129848655
SEC 232184407
SEC 101497207
SEC 65834431
SEC 138640631
SEC 258446559
SEC 186815119
SEC 108658575
SEC 260765975
SEC 199440007
SEC 27162639
SEC 242758551
SEC 247942103
SEC 248105975
SEC 130798759
SEC 189677719
SEC 101497215
SEC 140703615
SEC 262065119
SEC 76138895
SEC 271538815
SEC 269397399
SEC 209151975
SEC 152811639
SEC 232184415
SEC 172743607
SEC 243041271
SEC 194999183
SEC 12593951
SEC 63671687
SEC 195973967
SEC 108658583
SEC 108601319
SEC 20506991
SEC 212791759
SEC 175397975
SEC 144056399
SEC 152811647
SEC 172743615
SEC 87102575
SEC 256861831
SEC 219305279
SEC 251200503
SEC 220075071
SEC 65834439
SEC 271538823
SEC 48186863
SEC 26160711
SEC 49759319
SEC 27162647
SEC 75929615
SEC 76138903
SEC 239738239
SEC 242775495
SEC 108601327
SEC 249394791
SEC 114516951
SEC 98201199
SEC 49202247
SEC 135146735
SEC 242742111
SEC 205215527
SEC 175397983
SEC 165023359
SEC 267063351
SEC 89558175
SEC 143827159
SEC 267204447
SEC 75929623
SEC 170588167
SEC 202746255
SEC 236218031
SEC 144056407
SEC 275665687
SEC 115031175
SEC 239738247
SEC 81488087
SEC 176951839
SEC 190974631
SEC 176792431
SEC 115031183
SEC 23128919
SEC 121435983
SEC 170588175
SEC 194908175
SEC 17597087
SEC 234808871
SEC 153930847
SEC 82738591
SEC 265632535
SEC 253193887
SEC 276456703
SEC 48976871
SEC 205702735
SEC 82738599
SEC 251200511
SEC 93751279
SEC 53040463
SEC 272470279
SEC 256861839
SEC 140850623
SEC 265632543
SEC 58910407
SEC 250529055
SEC 211170719
SEC 67384359
SEC 17597095
SEC 208947071
SEC 48186871
SEC 113554967
SEC 95508247
SEC 49202255
SEC 200871647
SEC 87102583
SEC 49759327
SEC 23128927
SEC 58910415
SEC 129848663
SEC 26160719
SEC 184452663
SEC 93751287
SEC 63671695
SEC 216894655
SEC 232399095
SEC 71789735
SEC 186815127
SEC 58973287
SEC 88509967
SEC 202937135
SEC 216745727
SEC 84655239
SEC 58973295
SEC 81488095
SEC 199440015
SEC 89558183
SEC 84655247
SEC 101585639
SEC 45626551
SEC 106101439
SEC 147925407
SEC 205215535
SEC 106101447
SEC 210120855
SEC 184452671
SEC 271466231
SEC 135146743
SEC 205702743
SEC 153930855
SEC 201810199
SEC 271466239
SEC 120949551
SEC 141212623
SEC 208947079
SEC 242742119
SEC 165023367
SEC 141212631
SEC 241730447
SEC 216745735
SEC 242758559
SEC 190974639
SEC 121124703
SEC 200871655
SEC 232399103
SEC 202746263
SEC 234808879
SEC 236505823
SEC 258446567
SEC 216894663
SEC 236218039
SEC 19118735
SEC 253193895
SEC 250529063
SEC 90794815
SEC 19118743
SEC 104516335
SEC 178977367
SEC 104452679
SEC 112676663
SEC 45626559
SEC 259307255
SEC 178977375
SEC 136701799
SEC 93659759
SEC 123873487
SEC 276456711
SEC 53040471
SEC 71789743
SEC 181178239
SEC 56431959
SEC 114157287
SEC 51378815
SEC 274711519
SEC 95508255
SEC 88509975
SEC 51378823
SEC 220852079
SEC 52807527
SEC 101585647
SEC 119318751
SEC 85109583
SEC 113554975
SEC 90794823
SEC 225582591
SEC 104516343
SEC 121124711
SEC 93659767
SEC 121435991
SEC 225582599
SEC 112676671
SEC 34691327
SEC 106607655
SEC 196873975
SEC 119318759
SEC 114157295
SEC 34691335
SEC 275217983
SEC 120949559
SEC 140850631
SEC 123873495
SEC 136701807
SEC 236907215
SEC 176951847
SEC 110377903
SEC 275217991
SEC 77873071
SEC 110377911
SEC 256461911
SEC 147925415
SEC 201810207
SEC 194908183
SEC 77873079
SEC 58790607
SEC 203247767
SEC 202126631
SEC 229539327
SEC 255926639
SEC 196582031
SEC 171528399
SEC 229539335
SEC 202937143
SEC 46584079
SEC 181178247
SEC 196582039
SEC 81709495
SEC 16203895
SEC 250269335
SEC 167129847
SEC 206498959
SEC 157103487
SEC 210120863
SEC 33941655
SEC 61832119
SEC 211170727
SEC 51080359
SEC 20052727
SEC 241730455
SEC 61832127
SEC 205165335
SEC 214632423
SEC 122824343
SEC 20052735
SEC 140191023
SEC 72637647
SEC 81807879
SEC 220852087
SEC 274711527
SEC 72637655
SEC 128485151
SEC 272470287
SEC 236505831
SEC 26748951
SEC 65279047
SEC 117772527
SEC 243127655
SEC 33941663
SEC 16203903
SEC 46584087
SEC 52807535
SEC 51080367
SEC 63747023
SEC 9727031
SEC 154125591
SEC 56431967
SEC 248621495
SEC 81709503
SEC 58790615
SEC 122824351
SEC 81807887
SEC 85109591
SEC 128485159
SEC 28894271
SEC 212288703
SEC 244553023
SEC 240170887
SEC 104452687
SEC 20195519
SEC 140191031
SEC 106607663
SEC 87504743
SEC 60974015
SEC 154125599
SEC 70560551
SEC 106535679
SEC 243848543
SEC 183553015
SEC 106535687
SEC 84613951
SEC 214632431
SEC 20081295
SEC 230841647
SEC 84613959
SEC 224588031
SEC 214736271
SEC 185091447
SEC 157103495
SEC 230841655
SEC 86633039
SEC 214736279
SEC 165450671
SEC 255926647
SEC 62035695
SEC 167129855
SEC 155598055
SEC 88750831
SEC 171528407
SEC 145660055
SEC 155598063
SEC 196873983
SEC 80230447
SEC 88750839
SEC 157307751
SEC 206498967
SEC 70322287
SEC 132340807
SEC 28894279
SEC 145660063
SEC 108340719
SEC 37795543
SEC 188785751
SEC 202126639
SEC 148319719
SEC 80230455
SEC 31851487
SEC 236907223
SEC 203247775
SEC 60974023
SEC 205165343
SEC 250269343
SEC 132340815
SEC 233859647
SEC 270652639
SEC 225418159
SEC 35692287
SEC 76047119
SEC 256461919
SEC 109379583
SEC 59689919
SEC 9727039
SEC 225418167
SEC 149748295
SEC 259307263
SEC 20081303
SEC 103313047
SEC 20195527
SEC 82232647
SEC 26748959
SEC 35692295
SEC 150942231
SEC 185091455
SEC 37795551
SEC 254427015
SEC 119379759
SEC 188785759
SEC 49133999
SEC 149748303
SEC 176409007
SEC 212288711
SEC 59689927
SEC 108707647
SEC 53238151
SEC 70322295
SEC 254427023
SEC 70560559
SEC 61420463
SEC 41722847
SEC 257685167
SEC 62035703
SEC 49134007
SEC 270024007
SEC 63747031
SEC 11128215
SEC 150493111
SEC 53238159
SEC 148918551
SEC 86633047
SEC 65279055
SEC 31851495
SEC 108707655
SEC 79049759
SEC 18028527
SEC 197659967
SEC 87504751
SEC 148319727
SEC 102963759
SEC 231653911
SEC 270024015
SEC 178557551
SEC 95239991
SEC 76047127
SEC 183553023
SEC 61784455
SEC 117772535
SEC 102963767
SEC 190999695
SEC 82232655
SEC 25147095
SEC 95239999
SEC 8835991
SEC 22143575
SEC 252103959
SEC 233859655
SEC 108340727
SEC 157307759
SEC 141336751
SEC 109379591
SEC 165450679
SEC 248621503
SEC 173172303
SEC 176583135
SEC 37392095
SEC 162253743
SEC 22143583
SEC 251391223
SEC 223742871
SEC 193250671
SEC 134560935
SEC 209745527
SEC 30787143
SEC 41722855
SEC 134560943
SEC 108068639
SEC 61784463
SEC 11128223
SEC 224588039
SEC 108068647
SEC 28391807
SEC 200923263
SEC 270504303
SEC 71219791
SEC 240170895
SEC 185298527
SEC 28391815
SEC 209693343
SEC 243127663
SEC 200923271
SEC 37392103
SEC 243848551
SEC 86238191
SEC 267178911
SEC 71219799
SEC 157982879
SEC 21306431
SEC 245068151
SEC 222179039
SEC 244553031
SEC 96866095
SEC 83571471
SEC 153955399
SEC 49938543
SEC 141336759
SEC 21306439
SEC 261117143
SEC 103313055
SEC 216100911
SEC 270652647
SEC 173172311
SEC 111395999
SEC 162253751
SEC 193250679
SEC 176409015
SEC 8835999
SEC 182721087
SEC 112948215
SEC 18028535
SEC 185298535
SEC 182721095
SEC 92534743
SEC 231653919
SEC 25147103
SEC 190999703
SEC 237512511
SEC 197659975
SEC 45690991
SEC 61420471
SEC 195375095
SEC 30787151
SEC 209693351
SEC 16479719
SEC 223742879
SEC 78461455
SEC 59330903
SEC 264401663
SEC 91499119
SEC 49938551
SEC 79049767
SEC 78461463
SEC 18776463
SEC 47245743
SEC 83571479
SEC 251391231
SEC 86238199
SEC 59756255
SEC 252103967
SEC 50773639
SEC 213929823
SEC 119379767
SEC 157982887
SEC 50773647
SEC 170143759
SEC 215533183
SEC 148918559
SEC 124077087
SEC 216210495
SEC 150493119
SEC 18776471
SEC 216210503
SEC 87799663
SEC 47245751
SEC 150942239
SEC 219983815
SEC 59756263
SEC 103870319
#define _FILE_OFFSET_BITS 64
#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdint.h>
#include <error.h>
#include <errno.h>
#include <assert.h>
#include <pthread.h>
#include <semaphore.h>

#define ERR(args...) fprintf(stderr, args);

#define READSIZE 4096
#define FILENAME "./bigfile"

int nthreads = 64;

extern int errno;
int fds[1024];
pthread_mutex_t locks[1024];
uint64_t filesize;
sem_t sem;

void * thread(void *arg)
{
char buf[READSIZE];
off_t offset;
int index, fd, ret;

fd = 0;
do {
for(index = 0; index < nthreads; index++) {
if(pthread_mutex_trylock(&locks[index]) == 0) {
// Found a free file descriptor
fd = fds[index];
break;
}
}
} while(fd == 0);

offset = (((uint64_t)random() << 32) | random()) % (filesize - READSIZE);

assert(offset < filesize - READSIZE);

if(lseek(fd, offset, SEEK_SET) == -1) {
ERR("thread(): lseek(): %s\n", strerror(errno));
goto done;
}

if((ret = read(fd, buf, READSIZE)) != READSIZE) {
if(ret == -1) {
ERR("thread(): read(): %s\n", strerror(errno));
} else if(ret == 0) {
ERR("thread(): read(): EOF\n");
} else {
ERR("thread(): read(): Only read %d of %d bytes\n", ret, READSIZE);
}
}

done:

pthread_mutex_unlock(&locks[index]);

sem_post(&sem);
}

int main(int argc, char **argv)
{
pthread_attr_t attr;
pthread_t child;
struct stat fstat;
int i;

if(argc == 2) {
nthreads = atoi(argv[1]);
assert(nthreads < 1024);
}

if(pthread_attr_init(&attr) != 0) {
ERR("main(): pthread_attr_init(): failed\n");
exit(1);
}

if(pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED) != 0) {
ERR("main(): pthread_attr_setdetachstate(): failed\n");
exit(1);
}

if(stat(FILENAME, &fstat) == -1) {
ERR("main(): stat(): %s\n", strerror(errno));
exit(1);
}

filesize = fstat.st_size;

for(i = 0; i < nthreads; i++) {
if((fds[i] = open(FILENAME, O_RDONLY | O_LARGEFILE)) == -1) {
ERR("main(): open(): %s\n", strerror(errno));
exit(1);
}
if(pthread_mutex_init(&locks[i], NULL) != 0) {
ERR("main(): pthread_mutex_init(): %s\n", strerror(errno));
exit(1);
}
}

if(sem_init(&sem, 0, nthreads) == -1) {
ERR("main(): sem_init(): %s\n", strerror(errno));
exit(1);
}

while(1) {
if(sem_wait(&sem) == 0) {
if(pthread_create(&child, &attr, thread, NULL) != 0) {
ERR("main(): pthread_create(): failed\n");
exit(1);
}
}
}
}

Next message: Jörn Engel: "Re: [PATCH] check headers for complete includes, etc."
Previous message: Leonard Milcin Jr.: "Re: Can't X be elemenated?"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]