fadvise+sync_file_range Vs O_DIRECT

From: Ameya Mitragotri
Date: Tue Apr 03 2007 - 06:51:39 EST

Hello,
I am doing some benchmarking on writes to the hdd from a simple
application (attached).
Im benchmarking the writes using the following
- write with O_DIRECT
- write with fadvise + O_SYNC
- write with fadvise + sync_file_range

Ive tested the writes in chunks of 4/8/16 KB and 1MB

The results show fadvise + sync_file_range is on par or better than
O_DIRECT. Detailed results are attached.

Now are these results along the expected lines? AFAIK O_DIRECT should
be faster because it bypasses the cache. Does it make sense for apps
to use fadvise + sync_file_range rather than use O_DIRECT?

I havent tried benchmarking using tools like Iozone as yet.

Hardware/OS details
1) AMD Athlon with 512 MB RAM running plain vanilla 2.6.20 (ext3 partition)
2) Intel P4 with 512 MB RAM running plain vanilla 2.6.20 (ext3 partition)

Thanks
Ameya

Attachment: appl.tar.gz
Description: GNU Zip compressed data

O_DIRECT

4K 8K 16K 64K 1M

75626.34 53137.48 49263.58 48017.88 43199

44787 25452 17050 10442 8775
39895 97939 34024 18844 30244
111063 27507 27139 28850 44359
48343 29046 28858 82900 40925
44384 30985 16371 45679 35577
36919 28481 56743 37928 34170
44343 28836 27617 35426 35844
45148 28865 76324 37369 34806
44453 28861 25919 32887 29888
45639 28868 36694 43376 38589
46829 28862 47014 37133 42702
44831 27146 37178 37187 33847
44939 32808 37180 37113 35865
37180 27183 37363 37190 34063
36842 28822 20563 37157 35801
40963 28863 136910 35663 35898
42155 28867 13272 28322 35811
44859 28859 33302 47811 39756
44783 28867 27944 9038 42671
40812 26264 12536 63794 33871
49601 39187 72871 38604 35870
42811 39700 12673 36756 34080
50571 27129 51175 37164 35806
394993 28862 79338 37159 35817
38668 28863 12577 15465 18760
42864 28864 32429 49628 46786
44958 28868 28855 37082 40984
44564 29047 28861 43121 34065
37733 29420 28862 39499 8481
42304 27211 28867 37140 78052
44020 28793 28858 37153 8286
39300 28877 27134 35645 27535
38236 28831 28863 15717 69280
46225 28866 28858 65864 8290
44965 28859 28862 64842 29335
125025 109304 105308 9526 54581
124800 113417 18856 88966 102495
123426 27753 181516 20588 8686
123296 108058 18963 175393 8634
123286 106323 178076 21466 16750
122005 107265 103631 175275 174088
120922 106893 18759 11106 8464
123411 108072 13790 175869 85829
123988 106730 104150 9409 14094
121964 107051 20478 88562 179919
123960 114400 13551 104760 103707
122677 114721 104940 152458 8400
123159 112336 20458 10075 85527
124142 113325 187358 12336 8480
124276 108468 104261 10157 85407

Fadvise + O_SYNC

4K 8K 16K 64K 1M

376290.86 202126.52 131960.98 76494.46 59176.52

330257 284465 100156 24233 19054
323958 193652 142911 74914 30224
357719 183639 149487 43475 90392
336065 166881 137875 114947 20033
340743 192979 89507 78008 117245
317783 181974 98894 98492 48205
351636 162300 109036 19969 51045
347116 170515 107965 81623 89149
333348 172022 133903 133210 12201
320617 180446 132839 63198 92913
339150 148410 107109 31272 12311
338398 168031 109812 102671 85124
329884 172748 121741 29860 12899
342271 150067 133982 94922 84807
330628 183596 123372 62554 12467
558715 156611 89231 121640 93566
365960 173794 103880 66925 12245
353970 157820 131674 76353 87231
376809 156348 97552 41378 12704
349713 169535 111219 117543 85127
389939 162914 121444 48571 12266
366682 166365 110327 70402 85495
329085 176377 136800 80525 12314
322519 160928 117013 79167 76546
371126 170565 124932 65822 12200
340151 165020 116263 29476 84982
369965 165502 121148 156296 12822
363818 178192 114615 73100 85061
358747 163731 110885 62335 12211
320117 167126 133125 42280 85532
332665 430943 187494 75522 12221
381596 261888 115885 59814 85586
343245 202842 122189 49196 12262
349435 169774 143609 119339 86675
435420 194419 104202 63832 12278
422798 207130 127684 94907 77211
405447 172173 97506 56524 12332
412660 393660 100924 92678 102327
409800 218263 201179 131964 16074
412765 250095 179226 106675 99372
468265 240041 163801 107430 12725
444476 245848 161119 23130 179834
460215 246600 167639 19080 12702
393892 229683 197418 124864 179599
450509 238987 141143 116350 12737
410514 250762 173352 110995 178979
418630 221949 185076 111781 12910
427651 253373 150761 22674 180690
421891 241654 154951 19971 109743
435780 233689 184194 132836 14198


Fadvise

4K 8K 16K 64K 1M

68731.44 54785.04 43424.16 50675.74 53662.22

54267 31361 18784 12546 12600
46225 30065 61485 72450 66269
45311 28853 15009 11509 11709
45133 28851 29856 15609 21572
46757 27137 28841 53980 44241
44940 28860 29225 73881 42943
45983 28864 58814 11475 69733
44601 29347 14753 111475 70690
39152 26284 24452 13650 11700
39065 36044 28854 28861 26130
43345 29162 27136 100312 70684
45889 28555 28857 11279 11676
45584 28862 28865 14421 24567
47868 28861 20303 80027 66358
43100 28860 17709 22290 11953
39529 24043 61646 63433 27332
46386 40337 56458 11468 69087
46438 28826 14683 16734 11696
37284 28835 15559 26140 42117
47199 28851 30653 105319 31073
46785 27219 116569 25272 91205
46335 28772 14246 75217 33579
45141 28877 69241 43902 37290
50516 23573 27784 44080 12316
46348 61561 14939 13472 35430
45591 42320 54400 17236 47532
41470 28726 15312 104052 40950
47506 28432 66203 12444 35604
39612 28978 15029 14713 12324
47629 40811 30374 24188 35886
38803 29123 28852 95727 68921
54245 28685 28854 11336 11665
45689 29030 29799 50497 24019
40043 28560 29637 59630 73122
48807 29543 28744 11790 11750
46114 27973 28852 12989 24374
136032 159174 114181 107551 102757
132853 117766 109321 18466 11755
129386 116853 25764 11455 181876
128712 112223 15968 11372 98634
124886 110876 24331 179743 16890
137826 116510 14934 17423 174706
130776 112289 20925 179588 17769
127226 112761 185525 11327 174125
126002 113973 20798 182007 11837
128900 112909 185024 109730 174311
126636 113482 22853 17607 18292
131218 113734 182927 184193 173647
125746 112420 23183 18421 11748
125683 111241 14697 11500 174667