[rrd-users] F5 BigIP CPU Usage - Math problem driving me crazy

[nate]

This is driving me crazy.

I've been collecting CPU usage in cacti from BigIP for a long
time now and it matches up with what the BigIP's own CPU
graph shows:
http://portal.aphroland.org/~aphro/bigip-cpu1.png
http://portal.aphroland.org/~aphro/bigip-cpu2.png

I setup a monitor in nagios today to alert on CPU usage and
it doesn't get the same results. In doing the math by hand
I get the same results as the nagios monitor, I can't figure
out why Cacti and BigIP are showing different information. I'm
starting to think it's a bug in RRDTool (both cacti
and BigIP use to draw their graphs).

The strange thing is that on another load balancer the same
check matches up with what cacti reports!

I am doing the EXACT same math in my new monitor as I am in
cacti.

I am polling these OIDs:

sysStatTmTotalCycles .1.3.6.1.4.1.3375.2.1.1.2.1.41.0
Traffic management CPU usage. The total cycles spent in
traffic management.

sysStatTmIdleCycles .1.3.6.1.4.1.3375.2.1.1.2.1.42.0
Traffic management CPU usage. The cycles spent polling with no
traffic.

Results are as follows:
Total Cycles: 2436469200949678
Idle Cycles:  1525494863604033

The command 'b global stats show all' confirms this:
TMM cycles (total, idle, sleep) = (2.437P, 1.525P, 0)

My math is as follows:

Determine Idle CPU usage(%):
((Idle Cycles)/(Total Cycles)) * 100 = 62.61088229678549703000 %

Determine CPU Usage(not Idle, %):
100 - (Idle CPU Usage) = 37.38911770321450297000 %

But Both the BigIP and cacti show ~55% CPU usage!! I wrote the
monitor for cacti and it's the EXACT SAME MATH, I've checked it
a dozen times! I've checked the raw data that I'm feeding
cacti and it matches up with the above calculations! So why is
cacti AND bigip showing this data? It really seems like it is
not accurate, what am I missing?

And as I mentioned above, the standby BigIPs that do not really
have any load show matching data with my hand math, my nagios
monitor, my cacti monitor and the bigip RRDTool-based graph.
(Standby BigIPs reporting ~1% CPU)

For reference this is what cacti uses to generate the rrd graph:
(I removed the sleep cpu from above calculations since it is
always 0 on the SMP boxes, so does not affect the results, and
yes I've confirmed it always returns 0, the below command was
built around a Uniprocessor BigIP which does sometimes return
sleep values, but it's really rare)

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-14400 \
--end=-60 \
--title="pd1-lb01-1 - F5 BigIP (CPU Usage)" \
--rigid \
--base=1024 \
--height=120 \
--width=600 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="CPU (%)" \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/home/cacti/public_html/cacti-0.8.7b/rra/pd1-lb01-1/bigip-stats.rrd":total_cpu:AVERAGE
\
DEF:b="/home/cacti/public_html/cacti-0.8.7b/rra/pd1-lb01-1/bigip-stats.rrd":sleep_cpu:AVERAGE
\
DEF:c="/home/cacti/public_html/cacti-0.8.7b/rra/pd1-lb01-1/bigip-stats.rrd":idle_cpu:AVERAGE
\
CDEF:cdefa=100,c,a,/,100,*,-,b,a,/,100,*,- \
CDEF:cdeff=b,a,/,100,* \
CDEF:cdefba=c,a,/,100,* \
CDEF:cdefbf=a,a,- \
CDEF:cdefbg=b,b,- \
CDEF:cdefbh=c,c,- \
AREA:cdefa#FF0000FF:"CPU Used\:"  \
GPRINT:cdefa:MIN:" min\:%0.0lf"  \
GPRINT:cdefa:AVERAGE:"avg\:%0.0lf"  \
GPRINT:cdefa:MAX:"max\:%0.0lf"  \
GPRINT:cdefa:LAST:"last\:%8.0lf\n"  \
AREA:cdeff#00FF00FF:"CPU Sleep\:":STACK \
GPRINT:cdeff:MIN:"min\:%0.0lf"  \
GPRINT:cdeff:AVERAGE:"avg\:%0.0lf"  \
GPRINT:cdeff:MAX:"max\:%0.0lf"  \
GPRINT:cdeff:LAST:"last\:%0.0lf\n"  \
AREA:cdefba#0000FFFF:"CPU Idle\:":STACK \
GPRINT:cdefba:MIN:" min\:%0.0lf"  \
GPRINT:cdefba:AVERAGE:"avg\:%0.0lf"  \
GPRINT:cdefba:MAX:"max\:%0.0lf"  \
GPRINT:cdefba:LAST:"last\:%8.0lf\n"  \
LINE1:cdefbf#FFFFFFFF:""  \
LINE1:cdefbg#FFFFFFFF:""  \
LINE1:cdefbh#FFFFFFFF:""

The cacti system is using RRDTool 1.2.23, the BigIP seems
to be using RRDTool 1.0.48.

I sent a note to F5 support as well, but not sure what kind of
help they might be.

thanks!

nate