[rrd-developers] [PATCH,RFC] optional mmap based file I/O

[Bernhard Fischer]

On Wed, May 30, 2007 at 09:20:20AM +0200, Tobi Oetiker wrote:
>Hi Bernhard,
>
>> Well, I was not sure about this. rrd_info does close the fd early on, so
>> we cannot blindly close the fd in rrd_close. If it is ok for you, we can
>> change rrd_close to close the fd (and adjust the callers accordingly).
>
>yes I think it this makes sense as rrd_cloes realy imples close ...
>or do you see a risk in this ?

No risk, Sounds fine for me. Will you do this, or should i add it to my
TODO (i'd prefer the former, ATM)?
>
>> >* why do you flag the first TWO pages ? rrd files with headers
>> >  takeing up two pages should be pretty rare ...
>>
>> Fair enough. My small test rrd's all had about 4192 bytes of headers
>> (IIRC), so i thought that using 2 pages should be a good starting point.
>> Easy enough to change, if you are confident that 1 page is enough for
>> the majority.
>
>hmmm well at the moment your strategy with madvise is not entirely
>transparent to me ... I think we need the following:
>
>a) tell the OS that we do RANDOM access to prevent readahead while
>   accessing the header portion

ok.
>
>b) set sequential access for stuff like rrd_fetch, rrd_resize, rrd_dump

ok.
>
>c) set DONTNEED (after reading/wrting) for all blocks except the
>   header and the 'hot' RRA blocks.

I think i have a note from you about the 'hot' RRA blocks. I'll dig it
out and see if i understand what you mean.
>
>> >* for figuring the pagesize you should use getpagesize is
>> >  available.
>>
>> So i vote for using sysconf, and adding checks for
>> _SC_PAGESIZE, _SC_PAGE_SIZE, PAGESIZE, PAGE_SIZE
>>
>> Ok?
>
>ok convinced ... sunos3 (or whatever os changed the page size)
>compatibility is probably not such an issue ...

ok.
>
>> >* if mmap is not used, then it would be cool if posix fadvise was
>> >  still called (have not checked the code, just saw that you
>> >  removed the check from configure)
>>
>> fadvise is still called for the FD path. Wrapping the check for fadvise
>> in $enable_mmap just makes sure that we do not call fadvise for mmap,
>> but only check for it and eventually use it for the non-mmap path.

I'm just seeing that there is something wrong (performance-wise) with
e.g.:
rrdtool update v.rrd --template traffic_out:traffic_in 1180512000:2013364287020:558938738885

I'm looking at this now. Sounds like it was not a great idea to use
updatev for benchmarking and i should have concentrated more on update ;)