[rrd-developers] Re: [rrd] rrdtool XDR Format
Tobias Oetiker
oetiker at ee.ethz.ch
Fri Apr 1 07:27:25 MEST 2005
Hi Matt,
this sound very cool ... xdr has been on my feature wishlist for a
long time ... same for the meta-data, many people have expressed a
wish to get this ... so how to go about this ...
I am currently putting the finishing touches on rrdtool 1.2 it will
be released by the end of april ...
Since this is a major new feature, I would like to see this
included once 1.2 is out of the door. Trunk will be open for new
features ...
1.2 will include memory mapped io, so this will help with
performance quite a bit ...
Questions:
* Is there xdr support for win32 ?
* In the past we have had some space issues in the rrdtool format
due to its rigid structure. So I have been thinking to make it
more flexible in the sense that some additional features could be
added to rrdtool at a later stage without requiering a format
update as such ... as long as one does not use the new feature
the old rrdtool version should continue to understand the format
and if the new feature is used, the old rrdtool version would
say: "Unknown Consolidation Function 'xyz' in RRA 3" but it could
still dump the file for example ... On the other hand it should
also be possible to allocate more space for a storing intermediate
data in connection with a new feature without breaking the format
... any thoughts on this ?
* The new XDR version should be able to read the old rrd files from
version 1.0 and 1.1
* I will be checking in the mmap support tonight after some further
testing ...
cheers
tobi
> Tobi et al-
>
> first let me start by giving tobi and the rrdtool developers kudoes for
> building such a great piece of software. i challenge anyone to find a better
> tool for storing and visualizing time series data.
>
> i have a simple proposal to make rrdtool even better with what i think will be
> very little retooling of the rrdtool code.
>
> to give some brief background, i manager the ganglia (http://ganglia.info/)
> distributed system monitoring program that currently runs on dozen of
> platforms.
>
> there are two big limitations to rrdtool that really hurt: (a) rrds are
> platform-specific (non-portable) and (b) every rrd update requires an
> immediate sync to disk.
>
> (a) it would be wonderful if i could move rrds around the network to different
> platforms and not deal with endianness problems. i know that rrdtool allows
> you to "dump" and contents from one platform and "restore" it however xml is
> such a bloated communication format. even more importantly, in the time is
> takes to export the data, move the data over the network, restore the data...
> the original data may have changed
> (and i've burned unecessary CPU in the process parsing XML).
>
> (b) it would also be wonderful if rrdtool supported in-memory databases. the
> disk io cost of writing to a rrd is pretty significant. it is not uncommon
> for ganglia to be monitoring thousands of hosts that update their data every
> 15 seconds or so. that is a lot of disk io!
>
> one workaround to this is to save multiple data points to memory before
> calling rrdtool "update" (although that reason to use rrdtool in the first
> place is to not have to implement storage of time series data yourself).
>
> one solution that solves both of these problems is for me to use jrobin
> (http://jrobin.org/) which is an implementation of rrdtool in java. the
> format of the jrobin rrds is portable and it supports in-memory rrds. however,
> i want the speed i get with C/C++ and don't want to implement portion of
> ganglia in C and others in Java.
>
> here is my solution to problem (a) and (b) above.
>
> leverage the power of the XDR: External Data Representation Standard
> (http://www.faqs.org/rfcs/rfc1832.html). this standard has been around since
> the 1980s and serves as the communication format for RPC (Remote Procedure
> Calls).
>
> this solution doesn't just solve these two problems, but it also opens up a
> number of new possibilities for rrdtool (not the least of which is rrdtool rpc
> services).
>
> here are the steps to make this dream a reality.
>
> 1. copy rrd_format.h to rrd_format.x (.x is the standard extension for
> rpc/xdr protocol description files).
>
> 2. make some slight modifications to rrd_format.x (i attached my final
> version of rrd_format.x to this email). the only real changes that i needed
> to make are as follows...
>
> 2a. change structure definitions from
>
> typedef struct foo_t {
> ...
> } foo_t;
>
> to
>
> struct foo_t {
> ...
> };
>
> 2b. changed unival union definition from
>
> typedef union unival {
> unsigned long u_cnt;
> rrd_value_t u_val;
> } unival;
>
> to
>
> enum unival_type {
> UNIVAL_COUNT,
> UNIVAL_VALUE
> };
>
> union unival switch (enum unival_type type)
> {
> case UNIVAL_COUNT:
> unsigned long u_cnt;
> case UNIVAL_VALUE:
> rrd_value_t u_val;
> };
>
> the 2b change is necessary because xdr protocol descriptions require that
> unions have a defined discriminant. when you think about it, this makes
> sense... if i'm sending a union over the network or to a file or whereever xdr
> needs to know what the format of the data in union is (given that it has
> multiple choices).
>
> 2c. i added the line
>
> typedef double rrd_value_t;
>
> and removed the include for "rrd.h" which had the definition for a
> rrd_value_t.
>
> 2d. i needed to change the contents of stat_head_t from
>
> struct stat_head_t {
> /* Data Base Identification Section ***/
> char cookie[4]; /* RRD */
> char version[5]; /* version of the format */
> ...
> };
>
> to
>
> struct stat_head_t {
> string rrd_cookie<4>; /* RRD */
> string rrd_version<5>; /* version of the format */
> ...
> };
>
>
> 3. run a protocol compiler (rpcgen) on this new protocol description file
> (rrd_format.x).
>
> e.g.
>
> % rpcgen rrd_format.x
>
> rpcgen creates two files from rrd_format.x: rrd_format.h and rrd_format_xdr.c
> for you.
>
> the generated rrd_format.h file is not much different than the original but it
> also has a number of XDR stub functions which are implemented in
> rrd_format_xdr.c.
>
> if you take a look at the man page for xdr (man xdr) or visit
> http://linux.ctyme.com/man/man3885.htm you will see the basic functions for
> XDR. these functions are implemented on every operating system I know of
> (including Cygwin on Windows and 32bit/64bit platforms) and require only that
> link to a single library (-lxdr).
>
> 4. modify the rrdtool code to use these XDR stub functions
>
> the rest of the code in rrdtool would need to call these stub functions
> instead of operating on rrd_t structures directly. XDR will deal with
> serializing the structures you define in order to send them over the wire or
> to a file or to memory.
>
> for example,
>
> rrd_create() as it is written now builds a rrd_t structure (mallocing
> substructures) and then calls
>
> int rrd_create_fn(char *file_name, rrd_t *rrd);
>
> the rrd_create_fn walks the rrd structure and writes the data to the specified
> file using calls to fwrite() which saves the data in a non-portable format.
>
> here is the new implementation of rrd_create_fn() using XDR...
>
> int rrd_create_fn(char *filename, rrd_t *rrd)
> {
> int rval,
> XDR xdrs;
> FILE *rrd_file = fopen(file_name,"wb"));
>
> if(!rrd_file)
> {
> rrd_set_error("creating '%s': %s",file_name,strerror(errno));
> free(rrd->stat_head);
> free(rrd->ds_def);
> free(rrd->rra_def);
> return(-1);
> }
>
> /* Create XDR stream */
> xdrstdio_create(&xdrs, rrd_file, XDR_ENCODE);
>
> /* Returns TRUE on SUCCESS, FALSE on FAILURE */
> rval = xdr_rrd_t(xdrs, rrd);
> if(rval == FALSE)
> {
> rrd_set_error("a file error occurred while creating '%s'",file_name);
> xdr_destroy(xdrs);
> return(-1);
> }
>
> /* We're finished with the stream */
> xdr_destroy(xdrs);
> rrd_free(rrd);
> return(0);
> }
>
> the next function to change would be rrd_open() since it is used to open
> databases for update, graph, etc.
>
> here is the new rrd_open() in using XDR..
>
> int
> rrd_open(char *file_name, FILE **in_file, rrd_t *rrd, int rdwr)
> {
> int rval;
> char *mode = NULL;
> XDR xdrs;
>
> rrd_init(rrd);
> if (rdwr == RRD_READONLY) {
> #ifndef WIN32
> mode = "r";
> #else
> mode = "rb";
> #endif
> } else {
> #ifndef WIN32
> mode = "r+";
> #else
> mode = "rb+";
> #endif
> }
>
> if (((*in_file) = fopen(file_name,mode)) == NULL ){
> rrd_set_error("opening '%s': %s",file_name, strerror(errno));
> return (-1);
> }
>
> /* Create the XDR stream */
> xdrstdio_create(&xdrs, in_file, XDR_DECODE);
>
> /* De-serialize the stream into a rrd_t structure */
> rval = xdr_rrd_t(xdrs, rrd);
> if(rval == FALSE)
> {
> rrd_set_error("'%s' is not an RRD file",file_name);
> free(rrd->stat_head);
> fclose(*in_file);
> return(-1);
> }
>
> return(0);
> }
>
> i kid you not... it's that easy.
>
> if you are running the XDR stream in the XDR_ENCODE direction (as in
> rrd_create_fn() above) then you must have malloc'd all the deep structures of
> rrd_t.
>
> *however*, if you are running the XDR stream in the XDR_DECODE direction (as
> in the rrd_open() above), you don't have to worry about mallocing memory at
> all since the XDR library handles it for you.
>
> if you want to free the deep structures malloc'd in a rrd_t structure by
> XDR... all you need to do is run.
>
> xdr_free( (xdr_proc_t)xdr_rrd_t, rrdptr );
>
> the xdr_free function takes two arguments. the function used to decode the
> data (xdr_rrd_t in this case) and a pointer to the data you want to free.
>
> here is the man page entry for xdr_free()...
>
> void
> xdr_free(proc, objp)
> xdrproc_t proc;
> char *objp;
>
> Generic freeing routine. The first argument is the XDR routine for the
> object being freed. The second argument is a pointer to the object
> itself. Note: the pointer passed to this routine is not freed, but what it
> points to is freed (recursively).
>
> so here is the new rrd_free() using XDR
>
> void rrd_free(rrd_t *rrd)
> {
> xdr_free(xdr_rrd_t, rrd);
> }
>
> easy.
>
> i think you can see how this approach would really make the rrdtool code a lot
> cleaner and easier to understand. it would also make databases completely
> portable... you can just copy them around in the raw to any platform that you
> want and they will just work.
>
> 100% backward compatibility
>
> using this approach will make updating the rrdtool database format much easier
> in the future AND rrdtool will always be 100% backward compatible from this
> point forward. to achieve this you simply need to alter the rrd_format.x that
> i attached to this email a bit more (i didn't want to alter it much from
> original in order to make it simpler to understand).
>
> all we need to do is specify a rrd_versions enum
>
> enum rrd_versions {
> RRD_VERSION_1,
> RRD_VERSION_2
> };
>
> and then define a rrd_t not as a structure but as a discriminant union as such
>
> struct rrd_version_1_t {
> ...
> };
>
> struct rrd_version_2_t {
> ...
> };
>
> union rrd_t switch ( enum rrd_versions )
> {
> case RRD_VERSION_1:
> rrd_version_1 version_1;
> case RRD_VERSION_2:
> rrd_version_2 version_2;
> }
>
>
> this means that all rrdtool data formats from here on out would be part of a
> single union. that would give you 100% backward compatibility.
>
> if a new rrdtool received data from an older rrdtool, it would know because
> the rrd_versions != RRD_VERSION_2. each revision of the format would require
> that you create a conversion function that converts RRD_VERSION_1 data to
> RRD_VERSION_2, and RRD_VERSION_2 to RRD_VERSION_3, and so on.
>
> this might sound like a lot of work but because all the stub functions are
> created by rpcgen you don't need to think about it. you just define a new and
> better format and plop it into the protocol description file and let rpcgen
> worry about it.
>
> by the way, talking about formats... a nice feature addition to rrdtool would
> be to allow user-defined meta-data in rrdtool databases in order to give some
> context to the data (which host is came from, application-specific data, etc).
> it would be easy to add that to the rrdtool format without increasing the size
> of the databases (unless you had user-defined info to store).
>
> this would allow me to collect data from distributed hosts into a central
> repository and have the database define itself (in the user-defined variable)
> as it moves through the system.
>
> i do not know the development contribution process for rrdtool. i'm willing
> to contribute the time to make this proposal a reality.
>
> -matt
>
>
--
______ __ _
/_ __/_ / / (_) Oetiker @ ISG.EE, ETL F24.2, ETH, CH-8092 Zurich
/ // _ \/ _ \/ / System Manager, Time Lord, Coder, Designer, Coach
/_/ \.__/_.__/_/ http://people.ee.ethz.ch/oetiker +41(0)44-632-5286
--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive http://www.ee.ethz.ch/~slist/rrd-developers
WebAdmin http://www.ee.ethz.ch/~slist/lsg2.cgi
More information about the rrd-developers
mailing list