[smokeping-users] Large deployment problems

[Tom Throckmorton]

On 8/2/11 8:15 PM, Josh Wisman wrote:
> I have a need to do a 5600+ node deployment of Smokeping. Meaning that I
> have 5600 devices to ping. Some 10% -20% have high latency 1000ms+.
> Currently I have about 1200 nodes in smokeping. This is a new build. I am
> building this on a OpenSuse 11.4 x86_64 platform. I am running into
> problems.
> 
> 1. SpeedyCGI does not compile on this version of Suse even after my best
> effort. I have attempted to make use of fastCGI, but I dont know that I am
> doing it right. The menu(smokeping.cgi) is large. Its by State then City in
> my config. It maxes out my cpu (1 of 4)  when you open smokeping, because of
> this, page load times are pretty slow and it will only get worse as I add
> more nodes.
> 
> <deletia>

Aside from tackling this with SpeedyCGI/fastCGI, you could also consider
pre-generating the index pages using the --static option; note that you
trade some of the dynamic graph interaction for the speed of static
content by doing this.  If you make your config modular using @includes,
you should be able to do this using mostly the same underlying configs
pretty easily.

> 2. Because of the number of nodes, fping poller does not finish in 300
> seconds.  I have blazemode enabled.

Blazemode only adds one extra 'sacrificial' ping for every probed
target, so that is probably not having a significant negative impact on
the polling time (it's not making it faster, if that's what you were
thinking).

> Is there a way to run multiple fping
> probes or increase parallelization? Any help would be greatly appreciated.

Setting up multiple probe groups with offsets would probably do most of
what you want to increase concurrency.  You could also break the work up
among several slave machines, with a non-polling master catching all of
the results; added bonus of this approach is that you get to break the
CGI frontend out from the probe executors.

> 3. If smokeping cant practically scale to this size, Does anyone have
> an open sourced suggestion. I am also running cacti and it has scaled rather
> well.

I can't answer to whether it will scale to that many nodes, however, I
believe it's certainly possible.  Cacti's threaded poller does scale
pretty well (I've run it with several thousand data sources), but
finding the collected data in that interface might be more challenging,
especially for end-users.


-tt