[rrd-developers] Re: Improving RRD tool scalability

[Hans Jørgen Jakobsen]

On Mon, 3 Mar 2003, Sasha Mikheev wrote:

>
>
> Recently we have encountered some "intresting" problems while using
> RRDtool derivative for large scale data collection.

[One more testemonial]
At the ISP where I work we have grown from the early MRTG where gifs we
made every time a measurement were made to homegrown system with
2 frontend machines doing the snmp collection
a central DB machine having an "interface" table
2 backend machines doing updates in the rrdfiles.

Some data
The are made around a million datacollections an hour on about 1/4
millon interfaces.
Many interfaces are polled 3 times an hour.
Some are polled every 5 min  and some are polled at 1 and 3 min
interval due to 32 bit counters and high speed.
The data collected depends on the type of interface. Bytes,packets,crc
ignore,drops...
The backends have 1/2TB of raid0+1 disks.
sw:ucd/net-snmp via perl,mysql,rrd,FreeBSD

Why:
At a time in the evolution we realised that we had to
break the the loop. ie
get data by snmp
update interfacetable
update rrd

The idea by breaking the loop was to be able to make multiple updates of
one rrdfile. The first update of an rrd takes 50-100 ms. The next are
free due to caching. The efficiency of the backend will raise when the
backlog is high.
It would not be possible to take advantage of file system caching due to
the size of the random acces to the many files spread all over.
The data from the fronends are spread to the right backend and sorted
in order of rrdfile. Depending on how long time the last update took
there might be more than one update to each rrd. There are priority in
the updates so that backbone interfaces will get faster updated than
luser interfaces.
The backends are io-bound. We have to throtle the update processes when
doing backup to get the backup done in a fair amount of time...

/hjj

--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-developers
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi