[rrd-developers] patch/tuning for very large RRD systems (was "Re: SQL Backend request")
Henrik Stoerner
henrik at hswn.dk
Thu May 31 00:04:48 CEST 2007
On Tue, May 29, 2007 at 05:00:18PM -0500, Dave Plonka wrote:
> I wrote:
> > However, the general I/O
> > activity did not go down - in fact, it increased by about 15-20%,
> > as measured by vmstat.
>
> With some more detail about what you saw, we could find what's
> really going on.
I've been doing some more detailed measurements today. The initial
result still holds, but it was kind of obscured by the fact that
prior to changes in the application, the disk system on the box
was saturated and couldn't keep up with the load - this of course
makes comparisons impossible.
So today I've done some measurements while running a version of the
application that batches updates, which by itself has kept the
I/O utilisation well below the system capacity. I've used vmstat
to track overall CPU utilisation, and iostat to monitor I/O.
The system used for testing has:
* 40.000 RRD files with 100.000 datasets. Each has 4 RRA with a
5 minute/30 minute/4 hours/24 hours averaging function. This
is currently a very stable set of files, new files haven't
been added during the tests.
* Update frequency is once per 5 minutes (as with MRTG)
* 2 CPUs, 1 GB RAM, 2 72 GB SCSI disks (10K rpm) with hardware RAID-1.
* Local filesystem formatted with ReiserFS, mounted with
"notail,noatime,nodiratime" options.
I've tried the following scenarios, all with rrdtool 1.0.50 as the
base version:
1) Stock rrdtool 1.0.49
2) rrdtool 1.0.50 just with the
fadvise(fileno(*in_file), 0, 0, POSIX_FADV_RANDOM)
in rrd_open.c, i.e. Dave original patch.
3) Same as 2), but with the first "0" changed to "4096",
i.e. all access after the first 4 KB is random.
4) As 3), but with the limit set at 1 KB instead of 4
5) As 2) but patched with the change that is in the SVN version
of rrdtool 1.2, where some additional POSIX_FADV_DONTNEED
calls have been inserted for rrd_fetch, rrd_create etc.
All tests ran for at least 1 hour. The system is dedicated for this
purpose, so nothing else would disturb measurements.
The attached image shows the vmstat behaviour. The attached iostat*
files are the iostat data collected. I've grabbed some rough numbers
from this:
Test 1) ran from 00:00 - 11:00, and shows an average CPU utilisation
of 2-4% (the peak just after midnight and around 06:40 are daily cron jobs).
iostat data is in iostat-noadvise.txt, showing a utilisation of 6-8% of
capacity, with 10 reads/sec and 60 writes/sec.
Test 2) ran from 11:00 - 13:00. Avg. CPU is higher, 6-8%.
iostat-fadvise_randomonly.txt shows 15% utilisation, with 25 reads/sec
and 60 writes/sec.
Test 3) ran from 13:00 - 14:00. Avg. CPU is a bit higher than 1), but
less than 2) - probably 4-7% (I should have run this a bit longer).
iostat-fadvise_randomPost4K.txt is roughly like 2).
Test 4) ran from 14:30 - 22:00. Avg. CPU is lower than 2) and 3),
around 4-5%. iostat-fadvise_randomPost1K.txt shows utilisation of
10%, with 20 reads/sec and 50 writes/sec.
Test 5) ran from 22:30 - 00:00. Avg. CPU is the highest, 12-15%.
iostat-fadvise_randomanddontneed.txt shows utilisation is about
25%, with 150 reads/sec and 60 writes/sec.
Based on this, the fadvise() additions do not seem to be universally
good. At the very least, one should use it so that read-ahead is done
on the first 1K of the RRD file, and the rest is flagged with FADV_RANDOM
(test 3). But the default rrdtool - with no fadvise() - still has
fewer reads, and less cpu+disk utilisation.
It is indisputable that the DONTNEED additions in test 5) result in the
buffer cache occupying much less memory. This could be an advantage, if
there are applications running that require lots of memory.
I can only speculate about why I see these results which differ somewhat
from David's. One obvious difference between our systems is the memory
size; but we do share the fact that our set of RRD files does not fit in
the buffer cache.
Another - perhaps more important? difference - is that David uses SAN
storage, whereas my system has local disks only. I suspect the SAN
drivers might do read-ahead on their own and buffer the data, but that
is just me guessing.
Anyway, it's been interesting to test - and I've ended up with a much
better performing system. So I'm happy :-)
Regards,
Henrik
-------------- next part --------------
$ iostat -dx 300 10
Linux 2.6.18-4-686 (hobbit-wrk2) 2007-05-30
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,39 30,82 102,79 156,49 282,96 96,74 1,46 0,23 0,88 1,76 45,53
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,06 22,58 11,89 63,78 958,52 674,71 21,59 0,97 12,81 1,14 8,64
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,03 22,01 10,48 63,06 839,08 666,55 20,47 1,04 14,16 1,18 8,70
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,04 21,60 9,85 60,16 825,02 638,53 20,91 0,85 12,16 0,90 6,29
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,02 20,10 9,95 54,22 804,21 580,55 21,58 0,70 10,92 1,05 6,76
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,02 19,83 10,45 57,09 860,28 600,01 21,62 0,80 11,80 1,01 6,79
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,04 20,96 9,03 57,47 765,59 613,03 20,73 0,73 10,92 0,97 6,43
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,02 32,68 9,46 74,36 826,73 840,59 19,89 1,21 14,45 0,92 7,70
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,03 28,58 10,67 64,62 887,28 731,19 21,50 1,12 14,85 1,07 8,06
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,09 29,21 12,13 67,41 932,78 757,99 21,26 1,09 13,67 1,08 8,60
-------------- next part --------------
$ iostat -dx 300 10
Linux 2.6.18-4-686 (hobbit-wrk2) 2007-05-30
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,39 30,81 102,63 156,22 284,66 98,59 1,48 0,23 0,90 1,76 45,44
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,03 30,82 72,39 67,62 580,21 771,41 9,65 1,46 10,45 1,10 15,40
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,02 26,71 64,58 61,37 516,59 689,76 9,58 1,44 11,46 1,15 14,47
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,01 25,80 69,04 60,63 552,26 677,12 9,48 1,44 11,08 1,14 14,76
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 23,73 65,47 57,33 523,72 633,49 9,42 1,64 13,35 1,20 14,75
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,04 24,40 65,46 60,25 525,22 661,50 9,44 1,22 9,71 1,22 15,33
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,01 27,08 83,35 66,97 666,78 737,31 9,34 1,81 12,05 1,25 18,76
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 22,40 55,71 60,17 445,66 645,74 9,42 1,26 10,88 1,29 14,95
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,01 21,36 50,28 57,85 402,21 619,10 9,45 1,33 12,31 1,24 13,40
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 20,59 52,15 57,22 417,22 608,60 9,38 1,45 13,25 1,29 14,15
-------------- next part --------------
Linux 2.6.18-4-686 (hobbit-wrk2) 2007-05-30
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,39 30,80 102,56 156,04 285,16 99,71 1,49 0,24 0,92 1,75 45,38
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 19,45 44,56 53,02 356,49 564,96 9,44 1,15 11,77 1,03 10,06
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 19,25 38,82 50,70 310,50 544,39 9,55 0,82 9,12 0,99 8,85
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,03 21,08 48,24 51,72 386,52 567,74 9,55 1,14 11,40 1,03 10,34
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,01 29,75 58,01 66,05 464,02 751,17 9,80 1,43 11,50 0,99 12,30
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 27,79 52,88 62,04 423,07 704,27 9,81 1,29 11,23 1,07 12,29
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,03 29,71 85,21 69,44 682,14 778,67 9,45 2,00 12,95 1,29 19,95
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,08 26,91 67,81 60,48 552,38 683,36 9,63 1,27 9,94 1,17 15,07
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,19 78,44 63,20 59,99 512,82 743,87 10,20 1,30 10,57 1,11 13,69
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 2,23 29,21 66,80 57,70 614,49 678,75 10,39 1,17 9,37 1,09 13,53
-------------- next part --------------
Linux 2.6.18-4-686 (hobbit-wrk2) 2007-05-30
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,39 30,79 102,50 155,87 285,60 100,66 1,49 0,24 0,94 1,75 45,32
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,01 18,54 49,69 48,62 397,58 522,40 9,36 0,75 7,58 1,03 10,14
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 18,59 44,08 49,05 352,65 526,41 9,44 1,00 10,72 1,08 10,10
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 18,51 43,68 48,88 349,37 525,09 9,45 0,93 10,08 0,98 9,07
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 19,02 44,32 55,31 354,55 580,39 9,38 1,04 10,48 1,05 10,47
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,01 18,32 42,56 52,07 340,53 548,79 9,40 0,77 8,12 0,96 9,12
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,08 20,58 81,77 62,57 655,52 651,39 9,05 1,63 11,32 1,26 18,17
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,18 18,28 58,97 51,09 480,56 540,94 9,28 0,88 8,04 1,03 11,33
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 19,07 52,26 50,42 418,03 539,35 9,32 0,85 8,25 1,10 11,35
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,01 20,92 57,55 52,38 460,39 572,40 9,39 1,03 9,33 1,04 11,39
-------------- next part --------------
Linux 2.6.18-4-686 (hobbit-wrk2) 2007-05-30
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,38 30,69 102,13 154,96 287,54 105,06 1,53 0,25 0,97 1,75 45,03
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,01 17,68 136,79 60,96 1094,22 613,72 8,64 1,16 5,89 1,22 24,19
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 17,43 133,53 58,63 1068,28 594,29 8,65 0,68 3,54 1,24 23,78
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 17,49 130,58 58,08 1044,60 589,89 8,66 0,79 4,17 1,26 23,83
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 17,85 147,26 64,74 1178,07 645,65 8,60 0,85 3,99 1,25 26,54
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 20,88 194,88 88,18 1559,02 855,89 8,53 1,46 5,15 1,32 37,47
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,01 18,35 148,81 67,49 1191,55 669,90 8,61 0,86 3,98 1,25 27,07
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 17,92 144,76 65,09 1158,02 649,45 8,61 1,00 4,75 1,28 26,79
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 18,32 146,62 64,56 1172,97 646,57 8,62 0,89 4,20 1,26 26,67
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0,00 17,67 137,51 59,92 1100,09 605,27 8,64 0,67 3,41 1,28 25,32
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rrdexperiments.png
Type: image/png
Size: 12549 bytes
Desc: not available
Url : http://lists.oetiker.ch/pipermail/rrd-developers/attachments/20070531/1b5ee1a8/attachment-0001.png
More information about the rrd-developers
mailing list