[rrd-users] How to Create the rrd that storeor update million of requests per second

[Kashif Qazi]

Alex van den Bogaerdt-5 wrote:
> 
> ----- Original Message ----- 
> From: "Kashif Qazi" <kaqazi at etilizepak.com>
> To: <rrd-users at lists.oetiker.ch>
> Sent: Friday, February 13, 2009 7:25 AM
> Subject: [rrd-users] How to Create the rrd that storeor update million of 
> requests per second
> 
> 
>>
>> Hi All,
>>
>> I have to store each request data to rrd database. the request
>> granularity
>> can be more or less than one million hits per second.
>> can rrd support granularity in milli/micro seconds.
> 
> It will do sub-second precision when updating, but not in DSes and RRAs.
> In 
> other words: it will take the average (or max, min) of these updates.
> 
>> 1) How much time the request takes on average for each: minute, hour,
>> work
>> day (6am-8pm), day (24 hour), week, month
> 
> That's not a million per second, is it?
> 
> Are you perhaps saying that in a certain second you could get many (that 
> million) requests but you do not really care about which request came
> first, 
> and how many microseconds before the next?
> 
>> 2) Scatter graph of all requests so I can plot the time of every request
>> over a period of time (day, week, month, year) and see where the majority 
>> of
>> requests perform and what the outliers are (really slow, really fast).
> 
> I don't think rrdtool can do this, or it has to be a new feature which I
> did 
> not yet play with.
> 
>> 3) How many requests per second, minute, hour, word day, day, week, month
>>
>> 4) Number of requests in each “performance group”. For example, how many
>> requests are 0-100ms, 101-200ms, 201-300ms, 301-400ms, 401-500ms, etc ...
>>
>> 5) When am I having requests > 1000ms? < 5000ms?  What is associated with
>> these? Certain times of day? Certain traffic volumes?
> 
> Easy, easy, easy.
> 
> I don't think rrdtool is the right tool for you, even if you could work 
> around certain issues.  It comes close though, so if you have not given a 
> correct problem description, rrdtool may prove to be useful after all.
> Only 
> that scatter graph will be a problem (unless, again, there is a new
> feature 
> I have not yet explored).
> 
> HTH
> Alex 
> 
> _______________________________________________
> rrd-users mailing list
> rrd-users at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
> 
> 

Thanks Alex,

For your quick response. I highly appreciate it.

1) How much time the request takes on average for each: minute, hour, work
 day (6am-8pm), day (24 hour), week, month

 That's not a million per second, is it?

yes it is not million request for the above required report. but i have
records of requests that hit my site that may be million.


2) Are you perhaps saying that in a certain second you could get many (that 
million) requests but you do not really care about which request came first, 
and how many microseconds before the next?


If RRDtool is not right to fulfill my requirements. Can you suggest any tool
that can help me to fulfill the requirements.

My requirements are given below. 

I need to be able to create reports and graphs of our usage statistics, but
they are huge files with lots of data, much more than something like Excel
can handle.

Data Sample  data sample in csv file and it can be dumped to Db or any thing
else required by the tool.

date_received,time_received,catalog_key,spexlive_duration,inquire_duration,custom_application_id,
ip_address

2008-07-23    00:00:02    na    48    14    657    208.74.72.79
2008-07-23    00:00:03    spr    466    440    748    65.38.170.141

Request Performance

1) How much time the request takes on average for each: minute, hour, work
day (6am-8pm), day (24 hour), week, month

2) Scatter graph of all requests so I can plot the time of every request
over a period of time (day, week, month, year) and see where the majority of
requests perform and what the outliers are (really slow, really fast).

3) How many requests per second, minute, hour, word day, day, week, month

4) Number of requests in each “performance group”. For example, how many
requests are 0-100ms, 101-200ms, 201-300ms, 301-400ms, 401-500ms, etc ...

5) When am I having requests > 1000ms? < 5000ms?  What is associated with
these? Certain times of day? Certain traffic volumes?


Combination of Graphs

I want to be able to combine graphs and overlay them so I can compare the
impact of different data elements.

For example, how does throughput affect performance?

I would want to overlay the “requests per second” with “request performance”
and see what impact the number of requests per second has on the performance
of those requests.


Distribution Graph

One graph that combines all of the above that I really want is a
distribution graph that correlates performance with time and/or request
throughput.

I want to see:


distribution of all requests according to response time with a list graph
overlaying this with requests/second for throughput so I can how throughput
affects performance 
distribution of all requests according to time showing performance so I can
see  how the time of day affects performance 
combination of all types of requests (search, getproduct, etc) so I can see
how the combination of request types at a specific point in time affects
performance.

Thanks
Kashif Ahmed Qazi


 






-- 
View this message in context: http://n2.nabble.com/How-to-Create-the-rrd-that-storeor-update--million-of-requests-per-second-tp2319825p2323808.html
Sent from the RRDtool Users Mailinglist mailing list archive at Nabble.com.