[rrd-developers] implementing portable format - change format?

[Sfiligoi Igor]

Hi Tobi.

I created a small python script (attached), and ran it on my laptop
(Dell Latitude D810, a Pentium M 2GHz class machine).

For starters, the DB file stays constant, independently how many updates
one makes (tried with 1M);
4096 bytes for 100 rows
25600 bytes for 2000 rows.

PS: I created the DB files with
>>> import test1
>>> test1.create_db('t8',100)
>>> test1.create_db('t9',2000)

Running a simple open/update/close loop, I get ~9 updates per second:
>>> import test1
>>> test1.benchmark('t8',1000)
For 1000 loops, 110.665 seconds
>>> test1.benchmark('t8',1000)
For 1000 loops, 109.262 seconds
>>> test1.benchmark('t9',1000)
For 1000 loops, 121.620 seconds
>>> test1.benchmark('t9',1000)
For 1000 loops, 119.479 seconds

The overhead seems to be due to the transaction management; updating 1,
10 or 100 rows per transactions gives almost the same timing:
>>> import test1
>>> test1.benchmark_multi('t8',1000,10)
For 10000 updates (1000 loops 10 each), 123.724 seconds
>>> test1.benchmark_multi('t8',1000,10)
For 10000 updates (1000 loops 10 each), 108.126 seconds
>>> test1.benchmark_multi('t8',1000,100)
For 100000 updates (1000 loops 100 each), 111.148 seconds
>>> test1.benchmark_multi('t8',1000,100)
For 100000 updates (1000 loops 100 each), 111.108 seconds
>>> test1.benchmark_multi('t9',1000,100)
For 100000 updates (1000 loops 100 each), 124.660 seconds
>>> test1.benchmark_multi('t9',1000,100)
For 100000 updates (1000 loops 100 each), 124.104 seconds

Hope this results can help you.

Cheers,
  Igor

PS: I also ran the tests in autocommit mode (which effectively doubles
the number of transactions), and the run times were much longer, as
expected:
>>> import test1
>>> test1.benchmark_notran('t8',1000)
For 1000 loops, 210.374 seconds
>>> test1.benchmark_notran('t8',1000)
For 1000 loops, 218.828 seconds
>>> test1.benchmark_notran('t9',1000)
For 1000 loops, 233.482 seconds


Tobias Oetiker wrote:
> Igor,
> 
> my understanding is that even in the UPDATE case it is not a simple
> value replacement that happens ... but I would be most interested
> in your tests, stability and performance of such a solution ... you
> can do this with mockup configs without actually re-writing
> rrdtool.
> 
> cheers
> tobi
> 
> Today Sfiligoi Igor wrote:
> 
>> Tobias Oetiker wrote:
>>> Hi Igor,
>>>
>>> Yesterday Igor Sfiligoi wrote:
>>>
>>>> Hi Kevin and Tobi.
>>>>
>>>> Since you are planning to radically change the format, have you considered
>>>> going a more "standard" way?
>>>>
>>>> Instead of having a completely RRDTool specific format, use something
>>>> other tools could easily read?
>>>>
>>>> What about SQLite?
>>>> It is already portable.
>>>> And it is a database ;)
>>>>
>>>> It should be pretty easy to define a schema that serves RRDTool well.
>>>>
>>>> Would it be worth a consideration?
>>>>
>>>> Igor
>>> yes the thought has crossed my mind, the problem with SQL databases
>>> is that they are not realy good at updating round robin archives,
>>> du to transactional consistancy constraints they will create
>>> internal fragmentation which will cause performance to suffer
>>> dramatically.
>>>
>>> BUT it would be an interesting thought to see, if we can abstract
>>> the interface enough so that writing a 'plugin' for an sqlite
>>> storage backend becomes easy ...
>>>
>>> cheers
>>> tobi
>> What if the table had a structure like this:
>> index,date,value(1,value2,...)
>>
>> with index the table key.
>> I.e. very similar to what you have now in the RRD file.
>>
>> One would populate all the rows with consecutive indexes and
>> date=NULL,valueX=NULL at creation time.
>> The updates would be SQL UPDATEs on the existing row, i.e. replacing
>> date+valueX at a specific index.
>>
>> No INSERTs, no DELETEs.
>> This should keep the database compact and of fixed size.
>>
>> Do you still see a fragmentation problem under this conditions?
>>
>> Cheers,
>>   Igor
>>
>>
>>
>