[rrd-users] RRDTool + libdbi + mysql + speed

[Adam Jacob Muller]

Hi Brandon,
I've tested variants of the mysql version with both _use_ and _store_ with no appreciable difference.

the dbi version appears heavily CPU-bound, with minimal memory usage (my trivial test cases don't actually -keep- the data anywhere).

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     27941 90.4  0.1  42556 17784 pts/2    S+   23:55   0:16 ./dbi


-Adam

On Jul 3, 2011, at 10:47 PM, Brandon Phelps wrote:

> Just a wild guess here but have you checked memory usage during the 
> job?  Maybe it has something to do with libdbi using mysql_store_result 
> for such a large number of rows?  Try switching to mysql_use_result and 
> see if the problem persists?
> 
> Just a thought, could be way off.
> 
> On 7/3/2011 10:14 PM, Adam Jacob Muller wrote:
>> Hi,
>> I'm curious if anyone knows of any specific issues with regards to libdbi and MySQL with RRDTool. I have a specific dataset where I pull a significant number of rows from MySQL to draw graphs (think, in the 100,000's of rows). And its extremely slow.
>> 
>> I've specifically isolated this not to query execution time but to libdbi.
>> 
>> to completely isolate the issue, this is a small program I wrote that uses libdbi to read a months worth of samples (~2*60*24*30 = 86400 rows):
>> 
>>  # ./dbi
>> dbi_conn_queryf took 0.5319
>> dbi_result_next_row took 15.0172
>> looped through 71891 rows
>>  #
>> 
>> 
>> This is the same thing, but using the native mysql c-bindings:
>>  # ./mysql
>> mysql_query took 0.0021
>> mysql_fetch_row took 0.5352
>> looped through 71891 rows
>>  #
>> 
>> 
>> 
>> This probably ultimately seems like a libdbi issue, but I thought i'd bring it up here because it seems to have an extraordinary impact upon rrdtool performance and i'm curious if anyone here has seen it.
>> 
>> 
>> -Adam
>> _______________________________________________
>> rrd-users mailing list
>> rrd-users at lists.oetiker.ch
>> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
> 
> _______________________________________________
> rrd-users mailing list
> rrd-users at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users