[rrd-developers] Re: Improving RRD tool scalability

[Sasha Mikheev]

> -----Original Message-----
> From: Peter Stamfest [mailto:peter at stamfest.at]
> Sent: eai u?e 03 ioo 2003 21:56
> To: Sasha Mikheev
> Cc: rrd-developers at list.ee.ethz.ch
> Subject: Re: [rrd-developers] Improving RRD tool scalability
>
>
>
> Hi,
>
> Just a quick note: I am currently writing a tool to do
> high-volume data
> gathering using  SNMP with the possibility to  store data to
> RRDs. It is
> multithreaded and it can actually write to multiple RRDs in
> parallel (if
> this turns out to work for 17k RRDs remains to be tested -
> disk seek time
> and concurrency might become a problem here.
seek time is the _major_ issue here. You will get into the disk bottleneck
before there is a need to move to high performance SNMP.

I thought it would be the other way around and even started
working on high speed SNMP client. This is a bit off topic but are you
planning to use ucd-snmp
in that collector of yours ? My idea was to make single threaded snmp client
which has all snmp
packets prebuilt and just pumps it out then uses poll to get all responses
back. In theory i will
be limited by only by my network pipe and how fast device can process snmp
request.

>
> Does throwing in more RAM improve the situation? The OS might
> cache much
> more data then. Given 20K RRDs, each sized 100KB this would
> mean ~ 2GB of
> RAM to keep everything in RAM.
Our rrds are 4M sized (lots of high resolution data for drill downs) so
keeping everything in RAM is not an option.

>
> Isn't this on the TODO list (make the thin mmap-ed), but OTOH
> this might
> still not speed up things, as disk seeks are essentially the same...
It will make things much worse for update and i think it will not matter
that
much for rrd_fetch(). In linux msync is working by 4K pages while write
operation uses 512bytes
sector. So write() will have to transfer only 1/8 of data to disk.


--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-developers
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi