[rrd-users] rrdtool memory usage and the Raspberry Pi
Tobias Oetiker
tobi at oetiker.ch
Mon Oct 19 08:33:22 CEST 2015
Hi Jared,
are you using 1.5.4 ?
cheers
tobi
Today Jared Henley wrote:
>
>
> On 16/10/15 17:19, Tobias Oetiker wrote:
> > Hi Jared,
> >
> > Today Jared Henley wrote:
> >
> > > Hi,
> > >
> > > I've been working on a logging application that uses rrdtool. It works
> > > brilliantly on my PC, not so good on the Raspberry Pi model B+. But
> > > before I
> > > go there, there's something interesting I noticed while working on the PC.
> > >
> > > I'm creating a database with the following command (in rrdpython)
> > >
> > > rrdtool.create(filepath,
> > > '--step', '1s',
> > > 'DS:progreset:GAUGE:1000s:0:1',
> > > 'RRA:MAX:0.1:5s:5000m',
> > >
> > > <snip - there are 29 definitions all up, all the same>
> > >
> > > This creates a 386MB file.
> > >
> > > I didn't think too hard about it, until I did a graph to a CSV file. Of
> > > course the CSV would be less space-efficient than the binary rrd file,
> > > right?
> > > But the generated CSV turns out to be only 15MB?
> > >
> > > rrdtool seems to use 64-bit integers for everything, so I figure the rrd
> > > file
> > > above should use:
> > > 8 bytes per RRA entry * 29 RRAs + 1 timestamp * 60,000 locations in the
> > > round-robin (5000 minutes / 5 seconds) = 14.4MB. I'm confused. Why is the
> > > rrd
> > > file 30 times bigger than I'd expect? I did experiment with a step size
> > > of 5
> > > seconds, but the created file was the same size. But I can live with
> > > largish
> > > files.
> > when I create this database on my intel box it looks like this:
> >
> > oetiker>./rrdtool create /tmp/demo.rrd --step 1s
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m
> > oetiker>ls -l /tmp/demo.rrd
> > -rw-r--r-- 1 oetiker oep 480584 Oct 16 08:17 /tmp/demo.rrd
> >
> > so something looks rather odd here ... are you using 1.5.4 on your
> > raspy ?
> >
> >
> > cheers
> > tobi
>
> Hi Tobi,
>
> I can't actually create the database on the rpi - the process gets killed.
> Results mentioned are on my AMD 64-bit box, as are all tests below.
>
> I just ran your test:
>
> [08:32:39 pe at hope] $ rrdtool create /tmp/demo.rrd --step 1s
> DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m
> 0 ~
> [08:32:49 pe at hope] $ ls -l /tmp/demo.rrd
> -rw-rw-r-- 1 pe pe 480584 Oct 19 08:32 /tmp/demo.rrd
>
> Results are identical and obviously what I would expect.
>
> I'm having another issue with rrdpython, (but my mail to Hye-Shik Chang is
> bouncing), so I just tried my database creation directly in rrdtool with the
> following command (complete):
>
> [08:32:57 pe at hope] $ rrdtool create test.rrd \
> > --step 1s \
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> > DS:voltcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> > DS:freqcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> > DS:fault:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:uff:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:off:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:uvf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:ovf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:svopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:svclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:bvopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:bvclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:syncen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:busconnen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:wmfull:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:wmmed:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:wmlow:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:vred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:exciter:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:ired:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:iwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:iblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:f:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:isp:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:dlred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:dlwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:dlblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:svpos:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m
> 0 ~
> [08:45:55 pe at hope] $ ls -l test.rrd
> -rw-rw-r-- 1 pe pe 403757864 Oct 19 08:45 test.rrd
>
> Obviously something funny is going on between the one datasource/RRA test and
> the 29 datasource/RRA test, so I tried with just two DS/RRAs:
>
> [08:47:32 pe at hope] $ rrdtool create test.rrd \
> > --step 1s \
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m
> 0 ~
> [08:47:52 pe at hope] $ ls -l test.rrd
> -rw-rw-r-- 1 pe pe 1921184 Oct 19 08:47 test.rrd
>
> This file also seems too large. 1921184/480584 = 3.997... so it naively looks
> like the filesize is following a parabolic function against DS/RRA pairs
> rather than linear. I then created the following two files with almost no
> data, so I could look at them in a hex editor easily.
>
> [08:55:59 pe at hope] $ rrdtool create test.rrd \
> > --step 1s \
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s
> 0 ~
> [08:56:01 pe at hope] $ ls -l test.rrd
> -rw-rw-r-- 1 pe pe 632 Oct 19 08:56 test.rrd
> 0 ~
> [08:56:03 pe at hope] $ rrdtool create test2.rrd \
> > --step 1s \
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s \
> > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:30s
> 0 ~
> [08:58:23 pe at hope] $ ls -l test2.rrd
> -rw-rw-r-- 1 pe pe 1376 Oct 19 08:58 test2.rrd
>
> I really don't understand the structure of an rrd file, but there's a section
> at the end populated with "00 00 00 00 00 00 F8 FF" multiple times over. I
> assume this is where the RRA data is stored, and that the first section is the
> definitions and most-recently-updated values. Working on that understanding,
> the RRA data section of test.rrd is:
>
> 0000:0240 | 00 00 00 00 00 00 F8 FF |
> ......øÿ
> 0000:0250 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:0260 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:0270 | 00 00 00 00 00 00 F8 FF | ......øÿ
>
>
> (as expected, 6 64-bit values)
>
>
> And the RRA data section of test2.rrd is:
>
> 0000:04A0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:04B0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:04C0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:04D0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:04E0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:04F0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:0500 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:0510 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:0520 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:0530 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:0540 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:0550 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
>
> (I would expect 12 64-bit values, there are 24 here.)
>
> I did some updates to these files. test.rrd made some sense to me. Updates
> were 0.1, 0.2, 0.3, 0.4, 0.5 in 5 second intervals.
>
> 0000:0240 | 9A 99 99 99 99 99 D9 3F |
> ......Ù?
> 0000:0250 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F |
> ......à?333333ã?
> 0000:0260 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F |
> ......¹?......É?
> 0000:0270 | 33 33 33 33 33 33 D3 3F | 333333Ó?
>
> and again, with updates 0.7, 0.7, 0.8, 0.9, 1:
>
> 0000:0240 | CD CC CC CC CC CC EC 3F |
> ÍÌÌÌÌÌì?
> 0000:0250 | 00 00 00 00 00 00 F0 3F 33 33 33 33 33 33 E3 3F |
> ......ð?333333ã?
> 0000:0260 | 66 66 66 66 66 66 E6 3F 66 66 66 66 66 66 E6 3F |
> ffffffæ?ffffffæ?
> 0000:0270 | 9A 99 99 99 99 99 E9 3F | ......é?
>
>
> Updates to test2.rrd were 0.1:0.2, 0.3:0.4, 0.5:0.6, 0.7:0.9, 0.9:1 in 5
> second intervals.
>
> 0000:04A0 | 66 66 66 66 66 66 E6 3F CD CC CC CC CC CC EC 3F |
> ffffffæ?ÍÌÌÌÌÌì?
> 0000:04B0 | CD CC CC CC CC CC EC 3F 00 00 00 00 00 00 F0 3F |
> ÍÌÌÌÌÌì?......ð?
> 0000:04C0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
> 0000:04D0 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F |
> ......¹?......É?
> 0000:04E0 | 33 33 33 33 33 33 D3 3F 9A 99 99 99 99 99 D9 3F |
> 333333Ó?......Ù?
> 0000:04F0 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F |
> ......à?333333ã?
> 0000:0500 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F |
> ......¹?......É?
> 0000:0510 | 33 33 33 33 33 33 D3 3F 9A 99 99 99 99 99 D9 3F |
> 333333Ó?......Ù?
> 0000:0520 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F |
> ......à?333333ã?
> 0000:0530 | 66 66 66 66 66 66 E6 3F CD CC CC CC CC CC EC 3F |
> ffffffæ?ÍÌÌÌÌÌì?
> 0000:0540 | CD CC CC CC CC CC EC 3F 00 00 00 00 00 00 F0 3F |
> ÍÌÌÌÌÌì?......ð?
> 0000:0550 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF |
> ......øÿ......øÿ
>
> Using the info from test.rrd's hex dump, it looks like the values in test2.rrd
> are:
>
> 0.7 0.9
> 0.9 1
> UNKNOWN UNKNOWN
> 0.4 0.5
> 0.6 0.1
> 0.2 0.3
> 0.4 0.5
> 0.6 0.1
> 0.2 0.3
> 0.7 0.9
> 0.9 1
> UNKNOWN UNKNOWN
>
> It seems like the file test2.rrd contains two sets of RRA data like this:
>
> first-half data
> second-half data
> second-half data
> first-half data
>
> Jared.
>
>
>
>
>
>
>
>
>
>
>
>
>
--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch tobi at oetiker.ch +41 62 775 9902
More information about the rrd-users
mailing list