[rrd-developers] Re: status of threaded RRDTool and Perl

[Peter Stamfest]

Hello,

On Sat, 26 Nov 2005, Ole Bjørn Hessen wrote:

> Eric ZYLBERSTEJN <ezylby at yahoo.fr> writes:
>> Implementing the thread safe API is of course possible
>> in XS. It requires a little bit more coding because
>> the API is not as generic as before.
>> Since some parameters are now required and in a
>> specific order, it could break a lot of the present
>> Perl code. One way to circumvent that could be to keep
>> the low level MT implementation in XS and offer a
>> compatible higher layer in Perl which decodes the
>> variable arguments list and calls the lower layer.
>
> Yes, I came to the same conclusion. But there's a but here. Only a few
> of these functions are thread-safe yet.
>
> Perhaps the best way is to start on a new XS perl-binding exposing the
> low level function in a new module with different names so it can be
> extended one by one function until it is finished.
>
> I startet on a RRDts.pm and perl-shared-thread/RRDts.xs binding below
> defining rrd_error_r, rrd_update_r and rrd_create_r, but I'm not sure
> if I want to do more than this.
>

I just want to shed some light on the initial issues with RRD 
thread-safety, maybe this points anybody going down that road in the 
right direction:

When I did the initial thread safety support for RRD, I did look at many 
other RRD functions to check if I could make them thread safe as well, but 
for the following reasons I couldn't and didn't:

* Most operations other than rrd_update, rrd_error (and rrd_create, to
   support the software I needed the thread support for initially) are
   operations that _normally_ do not get used that heavily/often, so I
   envisioned that most people could live with the restriction to use
   those functions in a non-threaded environment only. One very notable
   exception to this is rrd_graph, which could not be made thread-safe due
   to the next reason.

* Most of the functions in their initial state are/were inherently
   non-thread safe, due to the use of third party libraries with unknown
   thread-safety status and the use of known non thread-safe library
   functions. For those cases where both apply, I did not try hard to take
   care of solvable issues.

* I did think of RRD language bindings (I am using RRD with perl in one
   project), but at that time perl wasn't multithreaded by itself, so there
   was no immediate need for this.

One (seemingly) possible solution for perl would be to introduce a kind of 
"Big RRD lock" (like ye olde Big-Kernel-Lock in linux) and protect all 
currently non-thread safe operations with it. However, this does not solve 
the problem when this code gets embedded in a multi-threaded environment, 
because RRD code might then get in the way of other code using non-thread 
safe library calls. True, there shouldn't be any such code in a 
multi-threaded environment anyway, but think of using a second perl module 
that does the same dirty trick....

The only real solution would be to re-evaluate all the RRD functions to 
check for their current thread-safety status. Maybe the used 3rd party 
libraries have become thread-safe in the meantime and thus some more RRD 
functions can be _r'ed without much trouble.

Another possible solution that comes to mind would be to use a helper 
process (not thread) that does all the dirty RRD work. It would be 
single-threaded and non-thread-safe operations would simply be queued for 
the helper, but this scenario seems rather complex (but doable). If 
anybody wants to try this, it might make sense to make that helper generic 
enough so that it can be used for multiple language bindings. This helper 
would over time be uses less and less, until everything RRD has been made 
thread safe.


peter

--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-developers
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi