[rrd-developers] [PATCH] rrdcached "SLURP" for extreme update rates

[kevin brintnall]

On Thu, Oct 02, 2008 at 09:16:30PM +0200, Tobias Oetiker wrote:
> > A client on a high-privilge control socket may issue the following
> > command:
> >
> >   ->  SLURP /somewhere/filename
> >   <-  0 Processed 6840502 lines, 6840502 entries, 0 failures
> 
> iiks ... being able to instruct 'network services' to read in data
> from a local file system always sends shivers down my spine.

Note that it respects the base directory restrictions introduced with -B;
under those conditions it's possible to limit reads to "trusted"
directories.

The SLURP files can be protected with the same permissions as the UNIX
socket over which commands could otherwise be introduced, so I don't see
any privilege escalation here.

I think these extra restrictions are a good idea:

 * only read from regular files (i.e. no /dev/zero)
 * only read from files owned by daemon user
 * never read from files that are u+w or g+w

> I would much rather see an extension to the UPDATE command where I could
> tell it how may lines I am going to submit. And then proceede to send
> the data. This should be similarly fast I guess but the accessing of the
> filesystem would be a client responsability.

There are a couple difficulties there..  You may not know how many lines
you'll be submitting until you're done (unless you buffer them all in
memory first to count them).  To get around that, we would need some sort
of special command to terminate output, and a finite state machine to
track which state we're in.

	MULTIUPDATE
	x.rrd 1:2
	y.rrd 2:3
	.

> otoh, what would happen if the normal protocol was enhanced, so
> that the client could submit an id with its update command. The
> server would then tag its responses with these ids. The client
> would then not have to wait for each response, but could match them
> 'later'.

That dramatically complicates both the client and server.  We'd have to
have one of the following:

(1) Both would have to be converted to non-blocking I/O.
    * we have deadlock conditions with buffered I/O (each side is output
      buffering the "\n" that the other is waiting on)

    * to get around previous note, you're back to flush() which creates the
      small read()/write() that we're trying to get away from.

(2) Daemon does blocking read until terminator, all the while buffering
    results in memory.  Client then does blocking read to get all the
    results.

Either way, there are going to be a LOT of context switches between
processes, filling and draining of sockets, etc.  I don't think the
performance will come close to SLURP on a tempfs.

Would you be OK with the extra checks I mentioned above?  Should we
disable SLURP by default and add an option (-S) for users that want to
enable it?

-- 
 kevin brintnall =~ /kbrint at rufus.net/