[rrd-developers] [Ganglia-developers] Integrating rrdcachedwith Ganglia

[Daniel.Pocock at barclayscapital.com]

> -----Original Message-----
> From: rrd-developers-bounces at lists.oetiker.ch 
> [mailto:rrd-developers-bounces at lists.oetiker.ch] On Behalf Of 
> Pocock, Daniel: IT (LDN)
> Sent: 30 September 2008 11:19
> To: bernard at vanhpc.org
> Cc: rrd-developers at lists.oetiker.ch; 
> ganglia-developers at lists.sourceforge.net
> Subject: Re: [rrd-developers] [Ganglia-developers] 
> Integrating rrdcachedwith Ganglia
> 
> 
>  
> > >>
> > >> - RHEL5
> > >> - 64 bit
> > >> - APR 1.2.7
> 
> now using the trunk snapshot for 30/09/2008
> 
> > >> - Ganglia 3.1.0
> > >>
> > >> gmetad stores the data and IO levels are definitely better.
> > >>
> 
> 
> I deleted the rrd files and started again, just monitoring a 
> single server (gmond on the same server as gmetad)
> 
> I get lots of errors like this, and gaps in my graphs:
> 
> Sep 30 11:05:42 servername rrdcached[18002]: queue_thread_main:
> rrd_update_r 
> (/.../rrds/unspecified/servername/disk_total.rrd) failed with 
> status -1. (/.../rrds/unspecified/servername/disk_total.rrd:
> illegal attempt to update using time 1222768885 when last 
> update time is
> 1222768885 (minimum one second step))
> 


I decided to have a closer look at what is happening:

- the errors are only logged when someone retrieves a graph (in other
words, when rrdcached is told to flush everything)

- I increased the polling interval (now 3 seconds)

- compiling with debug symbols and setting a breakpoint on the error, I
discover the following:

Breakpoint 1, queue_thread_main (args=0x0) at rrd_daemon.c:703
703           RRDD_LOG (LOG_NOTICE, "queue_thread_main: "
(gdb) info locals
ci = (cache_item_t *) 0x1658ded0
file = 0x2aaab85e9b80 "/.../unspecified/__SummaryInfo__/mem_buffers.rrd"
values = (char **) 0x2aaaab33b010
status = -1
i = 2
values_num = 59739
now = {tv_sec = 1222775067, tv_usec = 193988}
next_flush = {tv_sec = 1222778538, tv_nsec = 636569000}
final_flush = 0
__PRETTY_FUNCTION__ = "queue_thread_main"

That was just 5 minutes after the process started.

Inspecting the values:

(gdb) x/s *(0x2aaaab33b010)
0x1658dfb0:      "1222774961:314936.000:1"
(gdb) x/s *(0x2aaaab33b010+8)
0x1658f460:      "1222774961:314936.000:1"
(gdb) x/s *(0x2aaaab33b010+16)
0x1658fd70:      "1222774961:314936.000:1"
(gdb) x/s *(0x2aaaab33b010+24)
0x1658df90:      "1222774961:314936.000:1"
(gdb) x/s *(0x2aaaab33b010+32)
0x165913f0:      "1222774961:314936.000:1"
(gdb) x/s *(0x2aaaab33b010+40)
0x16592cc0:      "1222774961:314936.000:1"



_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________