[rrd-users] Re: Updating 1000's of rrd files

[Todd Caine]

I worked with a guy who tried solving a similar problem.  He spent some time analyzing the data he would be storing and realized that 65% of the data was zeroes.  Since his data collection process was I/O bound spent opening and closing a few thousand RRD files he decided to write a server using Perl which had 2 queues, a high priority queue (the "interesting" queue), and a low priority queue (the "uninteresting" queue) for queuing up RRD update requests.  Upon startup the server would create an RRDp instance (forks off an rrdtool process and sets up a pipe to it), then begins listening on a TCP port number for client connections from our data collection processes.  It was the responsibility of the data collectors to determine if the data was interesting or uninteresting and to send the request to the RRD udpater to place the update request into the appropriate queue.  The clients did this by tagging each RRD update request with a H or L to designate the intended queue.  

Now here's where you can save some I/O time.  The queues used by the RRD server are first in first out.  Both the high and low priority queues use a special promotion process for new RRD update requests being added to either queue.  When an update request gets enqueued for an RRD file that is already in the queue, the new update request gets tagged onto the earliest update request for the same RRD file still in the queue.  The queue is basically queuing up linked lists of update requests.  It does this so that when an RRD file is opened by rrdtool it can send it as many updates that were queued up for the RRD file instead of always performing one RRD update each time an RRD files is opened.

Update requests were always dequeued from the high priority queue until empty before dequeuing from the low priority queue.   This allowed our "interesting" data to be available for our reporting and graphing needs while the updates with unintersting data could wait for longer periods of time in the queue so that we can hopefully write 15-30 minutes worth of zeroes to the RRD file once it finally gets dequeued.

On (Sun, Jul 04 14:22), Seth Hettich wrote:
> So, I have a problem:
> 
> I need to collect some data from ~2,000 to 3,000 devices (perhaps more)
> and put it into rrd files.  about 100 files per device.  The code to
> collect all the data can run in ~2-3 min.  Also, code to update one
> rrd file ~5,000 times can run in ~20 sec.
> 
> But, code to collect all the data and update the rrd files take over an hour
> 
> I assume this is due to the time to update so many files (in the test case
> it's just one file over and over, so it's very fast).
> 
> 
> I assume it would go much faster if I could put all of the data
> sources for each device in one file, but new data sources will get
> added all the time, and I see no way to add a DS to an rrd file.
> 
> Anyone have any ideas for how to make this work?
> 
> -Seth

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi