[rrd-users] rrdtool memory usage and the Raspberry Pi
Jared Henley
jared.henley at pelena.com.au
Mon Oct 19 00:51:30 CEST 2015
On 16/10/15 17:19, Tobias Oetiker wrote:
> Hi Jared,
>
> Today Jared Henley wrote:
>
>> Hi,
>>
>> I've been working on a logging application that uses rrdtool. It works
>> brilliantly on my PC, not so good on the Raspberry Pi model B+. But before I
>> go there, there's something interesting I noticed while working on the PC.
>>
>> I'm creating a database with the following command (in rrdpython)
>>
>> rrdtool.create(filepath,
>> '--step', '1s',
>> 'DS:progreset:GAUGE:1000s:0:1',
>> 'RRA:MAX:0.1:5s:5000m',
>>
>> <snip - there are 29 definitions all up, all the same>
>>
>> This creates a 386MB file.
>>
>> I didn't think too hard about it, until I did a graph to a CSV file. Of
>> course the CSV would be less space-efficient than the binary rrd file, right?
>> But the generated CSV turns out to be only 15MB?
>>
>> rrdtool seems to use 64-bit integers for everything, so I figure the rrd file
>> above should use:
>> 8 bytes per RRA entry * 29 RRAs + 1 timestamp * 60,000 locations in the
>> round-robin (5000 minutes / 5 seconds) = 14.4MB. I'm confused. Why is the rrd
>> file 30 times bigger than I'd expect? I did experiment with a step size of 5
>> seconds, but the created file was the same size. But I can live with largish
>> files.
> when I create this database on my intel box it looks like this:
>
> oetiker>./rrdtool create /tmp/demo.rrd --step 1s DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m
> oetiker>ls -l /tmp/demo.rrd
> -rw-r--r-- 1 oetiker oep 480584 Oct 16 08:17 /tmp/demo.rrd
>
> so something looks rather odd here ... are you using 1.5.4 on your
> raspy ?
>
>
> cheers
> tobi
Hi Tobi,
I can't actually create the database on the rpi - the process gets
killed. Results mentioned are on my AMD 64-bit box, as are all tests below.
I just ran your test:
[08:32:39 pe at hope] $ rrdtool create /tmp/demo.rrd --step 1s
DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m
0 ~
[08:32:49 pe at hope] $ ls -l /tmp/demo.rrd
-rw-rw-r-- 1 pe pe 480584 Oct 19 08:32 /tmp/demo.rrd
Results are identical and obviously what I would expect.
I'm having another issue with rrdpython, (but my mail to Hye-Shik Chang
is bouncing), so I just tried my database creation directly in rrdtool
with the following command (complete):
[08:32:57 pe at hope] $ rrdtool create test.rrd \
> --step 1s \
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> DS:voltcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> DS:freqcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> DS:fault:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> DS:uff:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> DS:off:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> DS:uvf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> DS:ovf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> DS:svopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> DS:svclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> DS:bvopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> DS:bvclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> DS:syncen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> DS:busconnen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> DS:wmfull:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> DS:wmmed:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> DS:wmlow:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> DS:vred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:exciter:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:ired:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:iwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:iblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:f:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:isp:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:dlred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:dlwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:dlblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> DS:svpos:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m
0 ~
[08:45:55 pe at hope] $ ls -l test.rrd
-rw-rw-r-- 1 pe pe 403757864 Oct 19 08:45 test.rrd
Obviously something funny is going on between the one datasource/RRA
test and the 29 datasource/RRA test, so I tried with just two DS/RRAs:
[08:47:32 pe at hope] $ rrdtool create test.rrd \
> --step 1s \
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m
0 ~
[08:47:52 pe at hope] $ ls -l test.rrd
-rw-rw-r-- 1 pe pe 1921184 Oct 19 08:47 test.rrd
This file also seems too large. 1921184/480584 = 3.997... so it naively
looks like the filesize is following a parabolic function against DS/RRA
pairs rather than linear. I then created the following two files with
almost no data, so I could look at them in a hex editor easily.
[08:55:59 pe at hope] $ rrdtool create test.rrd \
> --step 1s \
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s
0 ~
[08:56:01 pe at hope] $ ls -l test.rrd
-rw-rw-r-- 1 pe pe 632 Oct 19 08:56 test.rrd
0 ~
[08:56:03 pe at hope] $ rrdtool create test2.rrd \
> --step 1s \
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s \
> DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:30s
0 ~
[08:58:23 pe at hope] $ ls -l test2.rrd
-rw-rw-r-- 1 pe pe 1376 Oct 19 08:58 test2.rrd
I really don't understand the structure of an rrd file, but there's a
section at the end populated with "00 00 00 00 00 00 F8 FF" multiple
times over. I assume this is where the RRA data is stored, and that the
first section is the definitions and most-recently-updated values.
Working on that understanding, the RRA data section of test.rrd is:
0000:0240 | 00 00 00 00 00 00 F8 FF | ......øÿ
0000:0250 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0260 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0270 | 00 00 00 00 00 00 F8 FF | ......øÿ
(as expected, 6 64-bit values)
And the RRA data section of test2.rrd is:
0000:04A0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04B0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04C0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04D0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04E0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04F0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0500 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0510 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0520 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0530 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0540 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:0550 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
(I would expect 12 64-bit values, there are 24 here.)
I did some updates to these files. test.rrd made some sense to me.
Updates were 0.1, 0.2, 0.3, 0.4, 0.5 in 5 second intervals.
0000:0240 | 9A 99 99 99 99 99 D9 3F | ......Ù?
0000:0250 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F | ......à?333333ã?
0000:0260 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F | ......¹?......É?
0000:0270 | 33 33 33 33 33 33 D3 3F | 333333Ó?
and again, with updates 0.7, 0.7, 0.8, 0.9, 1:
0000:0240 | CD CC CC CC CC CC EC 3F | ÍÌÌÌÌÌì?
0000:0250 | 00 00 00 00 00 00 F0 3F 33 33 33 33 33 33 E3 3F | ......ð?333333ã?
0000:0260 | 66 66 66 66 66 66 E6 3F 66 66 66 66 66 66 E6 3F | ffffffæ?ffffffæ?
0000:0270 | 9A 99 99 99 99 99 E9 3F | ......é?
Updates to test2.rrd were 0.1:0.2, 0.3:0.4, 0.5:0.6, 0.7:0.9, 0.9:1 in 5
second intervals.
0000:04A0 | 66 66 66 66 66 66 E6 3F CD CC CC CC CC CC EC 3F | ffffffæ?ÍÌÌÌÌÌì?
0000:04B0 | CD CC CC CC CC CC EC 3F 00 00 00 00 00 00 F0 3F | ÍÌÌÌÌÌì?......ð?
0000:04C0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
0000:04D0 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F | ......¹?......É?
0000:04E0 | 33 33 33 33 33 33 D3 3F 9A 99 99 99 99 99 D9 3F | 333333Ó?......Ù?
0000:04F0 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F | ......à?333333ã?
0000:0500 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F | ......¹?......É?
0000:0510 | 33 33 33 33 33 33 D3 3F 9A 99 99 99 99 99 D9 3F | 333333Ó?......Ù?
0000:0520 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F | ......à?333333ã?
0000:0530 | 66 66 66 66 66 66 E6 3F CD CC CC CC CC CC EC 3F | ffffffæ?ÍÌÌÌÌÌì?
0000:0540 | CD CC CC CC CC CC EC 3F 00 00 00 00 00 00 F0 3F | ÍÌÌÌÌÌì?......ð?
0000:0550 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | ......øÿ......øÿ
Using the info from test.rrd's hex dump, it looks like the values in
test2.rrd are:
0.7 0.9
0.9 1
UNKNOWN UNKNOWN
0.4 0.5
0.6 0.1
0.2 0.3
0.4 0.5
0.6 0.1
0.2 0.3
0.7 0.9
0.9 1
UNKNOWN UNKNOWN
It seems like the file test2.rrd contains two sets of RRA data like this:
first-half data
second-half data
second-half data
first-half data
Jared.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.oetiker.ch/pipermail/rrd-users/attachments/20151019/27f1724b/attachment.html>
More information about the rrd-users
mailing list