[rrd-users] rrdtool memory usage and the Raspberry Pi

Rafal Gwizdala rafal.gwizdala at gmail.com
Mon Oct 19 08:28:40 CEST 2015 

Maybe the file size growth is expected to be parabolic - i think every RRA
is 'global' - it covers all the data sources. So when you're adding DS and
RRA pair you're actually adding N RRAs where N is number of data sources.
At least this is how i understood the concept of RRAs.
Best regards
Rafal

On Mon, Oct 19, 2015 at 12:51 AM, Jared Henley <jared.henley at pelena.com.au>
wrote:

>
>
> On 16/10/15 17:19, Tobias Oetiker wrote:
>
> Hi Jared,
>
> Today Jared Henley wrote:
>
>
> Hi,
>
> I've been working on a logging application that uses rrdtool.  It works
> brilliantly on my PC, not so good on the Raspberry Pi model B+.  But before I
> go there, there's something interesting I noticed while working on the PC.
>
> I'm creating a database with the following command (in rrdpython)
>
> rrdtool.create(filepath,
>     '--step', '1s',
>     'DS:progreset:GAUGE:1000s:0:1',
>     'RRA:MAX:0.1:5s:5000m',
>
> <snip - there are 29 definitions all up, all the same>
>
> This creates a 386MB file.
>
> I didn't think too hard about it, until I did a graph to a CSV file.  Of
> course the CSV would be less space-efficient than the binary rrd file, right?
> But the generated CSV turns out to be only 15MB?
>
> rrdtool seems to use 64-bit integers for everything, so I figure the rrd file
> above should use:
> 8 bytes per RRA entry * 29 RRAs + 1 timestamp * 60,000 locations in the
> round-robin (5000 minutes / 5 seconds) = 14.4MB.  I'm confused. Why is the rrd
> file 30 times bigger than I'd expect?  I did experiment with a step size of 5
> seconds, but the created file was the same size.  But I can live with largish
> files.
>
> when I create this database on my intel box it looks like this:
>
> oetiker>./rrdtool create /tmp/demo.rrd --step 1s DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m
> oetiker>ls -l /tmp/demo.rrd
> -rw-r--r-- 1 oetiker oep 480584 Oct 16 08:17 /tmp/demo.rrd
>
> so something looks rather odd here ... are you using 1.5.4 on your
> raspy ?
>
>
> cheers
> tobi
>
>
> Hi Tobi,
>
> I can't actually create the database on the rpi - the process gets
> killed.  Results mentioned are on my AMD 64-bit box, as are all tests below.
>
> I just ran your test:
>
> [08:32:39 pe at hope] $ rrdtool create /tmp/demo.rrd --step 1s
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m
>  0  ~
> [08:32:49 pe at hope] $ ls -l /tmp/demo.rrd
> -rw-rw-r-- 1 pe pe 480584 Oct 19 08:32 /tmp/demo.rrd
>
> Results are identical and obviously what I would expect.
>
> I'm having another issue with rrdpython, (but my mail to Hye-Shik Chang is
> bouncing), so I just tried my database creation directly in rrdtool with
> the following command (complete):
>
> [08:32:57 pe at hope] $ rrdtool create test.rrd \
> > --step 1s \
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> > DS:voltcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> > DS:freqcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> > DS:fault:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:uff:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:off:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:uvf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:ovf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:svopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:svclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:bvopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:bvclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:syncen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:busconnen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:wmfull:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:wmmed:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:wmlow:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:vred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:exciter:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:ired:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:iwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:iblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:f:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:isp:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:dlred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:dlwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:dlblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:svpos:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m
>  0  ~
> [08:45:55 pe at hope] $ ls -l test.rrd
> -rw-rw-r-- 1 pe pe 403757864 Oct 19 08:45 test.rrd
>
> Obviously something funny is going on between the one datasource/RRA test
> and the 29 datasource/RRA test, so I tried with just two DS/RRAs:
>
> [08:47:32 pe at hope] $ rrdtool create test.rrd \
> > --step 1s \
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m
>  0  ~
> [08:47:52 pe at hope] $ ls -l test.rrd
> -rw-rw-r-- 1 pe pe 1921184 Oct 19 08:47 test.rrd
>
> This file also seems too large.  1921184/480584 = 3.997... so it naively
> looks like the filesize is following a parabolic function against DS/RRA
> pairs rather than linear.  I then created the following two files with
> almost no data, so I could look at them in a hex editor easily.
>
> [08:55:59 pe at hope] $ rrdtool create test.rrd \
> > --step 1s \
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s
>  0  ~
> [08:56:01 pe at hope] $ ls -l test.rrd
> -rw-rw-r-- 1 pe pe 632 Oct 19 08:56 test.rrd
>  0  ~
> [08:56:03 pe at hope] $ rrdtool create test2.rrd \
> > --step 1s \
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s \
> > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:30s
>  0  ~
> [08:58:23 pe at hope] $ ls -l test2.rrd
> -rw-rw-r-- 1 pe pe 1376 Oct 19 08:58 test2.rrd
>
> I really don't understand the structure of an rrd file, but there's a
> section at the end populated with "00 00 00 00 00 00 F8 FF" multiple times
> over.  I assume this is where the RRA data is stored, and that the first
> section is the definitions and most-recently-updated values.  Working on
> that understanding, the RRA data section of test.rrd is:
>
> 0000:0240 |                           00 00 00 00  00 00 F8 FF |         ......øÿ
> 0000:0250 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:0260 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:0270 | 00 00 00 00  00 00 F8 FF                           | ......øÿ
>
>     (as expected, 6 64-bit values)
>
>
> And the RRA data section of test2.rrd is:
>
> 0000:04A0 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:04B0 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:04C0 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:04D0 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:04E0 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:04F0 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:0500 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:0510 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:0520 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:0530 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:0540 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:0550 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
>
> (I would expect 12 64-bit values, there are 24 here.)
>
> I did some updates to these files.  test.rrd made some sense to me.
> Updates were 0.1, 0.2, 0.3, 0.4, 0.5 in 5 second intervals.
>
> 0000:0240 |                           9A 99 99 99  99 99 D9 3F |         ......Ù?
> 0000:0250 | 00 00 00 00  00 00 E0 3F  33 33 33 33  33 33 E3 3F | ......à?333333ã?
> 0000:0260 | 9A 99 99 99  99 99 B9 3F  9A 99 99 99  99 99 C9 3F | ......¹?......É?
> 0000:0270 | 33 33 33 33  33 33 D3 3F                           | 333333Ó?
>
> and again, with updates 0.7, 0.7, 0.8, 0.9, 1:
>
> 0000:0240 |                           CD CC CC CC  CC CC EC 3F |         ÍÌÌÌÌÌì?
> 0000:0250 | 00 00 00 00  00 00 F0 3F  33 33 33 33  33 33 E3 3F | ......ð?333333ã?
> 0000:0260 | 66 66 66 66  66 66 E6 3F  66 66 66 66  66 66 E6 3F | ffffffæ?ffffffæ?
> 0000:0270 | 9A 99 99 99  99 99 E9 3F                           | ......é?
>
>
> Updates to test2.rrd were 0.1:0.2, 0.3:0.4, 0.5:0.6, 0.7:0.9, 0.9:1 in 5
> second intervals.
>
> 0000:04A0 | 66 66 66 66  66 66 E6 3F  CD CC CC CC  CC CC EC 3F | ffffffæ?ÍÌÌÌÌÌì?
> 0000:04B0 | CD CC CC CC  CC CC EC 3F  00 00 00 00  00 00 F0 3F | ÍÌÌÌÌÌì?......ð?
> 0000:04C0 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
> 0000:04D0 | 9A 99 99 99  99 99 B9 3F  9A 99 99 99  99 99 C9 3F | ......¹?......É?
> 0000:04E0 | 33 33 33 33  33 33 D3 3F  9A 99 99 99  99 99 D9 3F | 333333Ó?......Ù?
> 0000:04F0 | 00 00 00 00  00 00 E0 3F  33 33 33 33  33 33 E3 3F | ......à?333333ã?
> 0000:0500 | 9A 99 99 99  99 99 B9 3F  9A 99 99 99  99 99 C9 3F | ......¹?......É?
> 0000:0510 | 33 33 33 33  33 33 D3 3F  9A 99 99 99  99 99 D9 3F | 333333Ó?......Ù?
> 0000:0520 | 00 00 00 00  00 00 E0 3F  33 33 33 33  33 33 E3 3F | ......à?333333ã?
> 0000:0530 | 66 66 66 66  66 66 E6 3F  CD CC CC CC  CC CC EC 3F | ffffffæ?ÍÌÌÌÌÌì?
> 0000:0540 | CD CC CC CC  CC CC EC 3F  00 00 00 00  00 00 F0 3F | ÍÌÌÌÌÌì?......ð?
> 0000:0550 | 00 00 00 00  00 00 F8 FF  00 00 00 00  00 00 F8 FF | ......øÿ......øÿ
>
> Using the info from test.rrd's hex dump, it looks like the values in
> test2.rrd are:
>
> 0.7      0.9
> 0.9      1
> UNKNOWN  UNKNOWN
> 0.4      0.5
> 0.6      0.1
> 0.2      0.3
> 0.4      0.5
> 0.6      0.1
> 0.2      0.3
> 0.7      0.9
> 0.9      1
> UNKNOWN  UNKNOWN
>
> It seems like the file test2.rrd contains two sets of RRA data like this:
>
> first-half data
> second-half data
> second-half data
> first-half data
>
> Jared.
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> rrd-users mailing list
> rrd-users at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.oetiker.ch/pipermail/rrd-users/attachments/20151019/566c8d1e/attachment-0001.html>

More information about the rrd-users mailing list