[smokeping-users] Smokeping FastCGI Woes

[Simon Liang]

Hi there,

I've been running an early 2.6 version of Smokeping and recently
decided to upgrade to the latest 2.6.9. The polling side of Smokeping
is rock solid, however it's quite the opposite with the change to
FastCGI. I thought this would be a good place to start and hope
someone can help me.

Just some background of my current setup. I have separated the Polling
(forked with FPing) and web service side of Smokeping into two VMs (to
reduce the load the server). The config files are rsynced from the
poller to the web and the RRD files are shared via NFS.

Poller VM - 8 vCPU, 16GB RAM
Web VM - 4 vCPU, 16GB RAM

I understand when the web side of Smokeping starts (after a config
change), it does a lot of things under the hood. To address this
issue, on restarts I create an IP table rule on the web server so only
the poller can access it via port 80, I then use curl to trigger the
first (and only) session. Once it's completed I remove the IP table
rule so it becomes accessible. This works quite work and everything
works fine.

But after a random period of time, the load on the web server spikes
up quite high, memory starts swapping and the server becomes almost
unresponsive.

This is a screenshot of the web server under load (before I upgraded
to 16GB RAM)
https://dl.dropboxusercontent.com/u/11792766/Work/smokeping_load.JPG

The spikes in the graph are when the load on the server just randomly
spikes up and I'm forced to restart Smokeping manually.
https://dl.dropboxusercontent.com/u/11792766/Work/smokeping_stats.JPG

I can assure you there are no cron jobs running which may be loading
up the server. On peak hour traffic I have maybe 100 requests per
minute.

Here is my Apache (fcgid) config: http://pastebin.com/QU6XRcFg

If you need any more information please let me know.

Cheers,
Simon