[rrd-users] A plague of NaNs

[Jeff Gilmore]

Hi all,
	I have an application where I read data from electric meters, each into its own RRD file storing only watts into a GUAGE.  The RRD files are all of identical format, with a 60 second heartbeat.

Here is the PHP code that creates them:

if (!file_exists($rrd_file)) {
            $creator = new RRDCreator($rrd_file, $reading_values_to_store[0]->date - 60, 60);
            $creator->addDataSource("watts:GAUGE:60:0:96000");
            $creator->addArchive("AVERAGE:0.5:1:1440");
            $creator->addArchive("AVERAGE:0.5:10:2016");
            $creator->save();

The source data for these RRDs comes from a web app that returns data in XML format after it was uploaded from my meters to that website.   When I update them, I pass in an epoc time value based on the timestamp in the XML.

Most of the meters are read approximately once a minute, but that may vary by a few seconds (generally slightly longer than 60 seconds).  These meters produce RRD files where 90+ percent of the values stored  (as shown in the "database" section of RRDdump) are NaN.  

One of the meters is set to upload data on 30 second intervals (but still feeds an RRD file with a 60 second heartbeat).  The RRD for that meter has almost all of the values stored successfully.

Can you help me understand why this might be the case?  

I assumed that RRDtool's "normalization" process would adjust for the slight variations in timestamp between the actual data and the heartbeat interval of the RRD, fitting the update into the closest timeslot and adjusting the data value accordingly.  It sounds like I don't understand it sufficiently, despite many hours searching the web on this topic.  

In any case, I have a data source that produces data that is not perfectly aligned with the RRD heartbeat.  Can you suggest how my application can deal with this and still take advantage of the power of RRDTool?

I can share chunks of actual data and contents of RRDTool dumps if that is helpful.

Thanks,

Jeff