[mrtg] snmpget failing after switching from polling 32 bit to 64 bit interface counters

[Darren Murphy]

Hi,

I have an instance of MRTG that is polling around 1000 hosts (and has
been running for around 15 years).
Recently, I found that I was losing data for several hosts because we
have always polled the 32 bit interface counters, and they'd started
wrapping in between polling cycles.

So we made a change to the configs to force SNMP v2 so that the 64 bit
counters would be polled.

All good, except that several hosts have stopped working as soon as
the change was applied.

Here is what a typical host config looks like (hostname & IP address
anonymized):

Target[my_failing_host_wan]: /a.b.c.d:my_community_string at a.b.c.d:161:::::2
SetEnv[my_failing_host_wan]: MRTG_INT_IP="/a.b.c.d" MRTG_INT_DESCR="wan"
Options[my_failing_host_wan]: growright, bits, unknaszero, pngdate
Timezone[my_failing_host_wan]: Asia/Hong_Kong
MaxBytes1[my_failing_host_wan]: 125000000
MaxBytes2[my_failing_host_wan]: 125000000
AbsMax[my_failing_host_wan]: 137500137
Title[my_failing_host_wan]: my_failing_host - NSE: wan
RRDRowCount[my_failing_host_wan]: 8640
RRDRowCount30m[my_failing_host_wan]: 17520
RRDRowCount2h[my_failing_host_wan]: 13140
RRDRowCount1d[my_failing_host_wan]: 3650

And after enabling some debug options, here is what I see in the log file:

2016-03-03 13:45:02 -- Started mrtg with config '/etc/mrtg/nomadix-mrtg.cfg'
2016-03-03 13:45:02 -- --base: Creating Lockfiles
/var/lock/mrtg/nomadix-mrtg_l,/var/lock/mrtg/nomadix-mrtg_l_29380
2016-03-03 13:45:02 -- --base: Reading Config File: /etc/mrtg/nomadix-mrtg.cfg
2016-03-03 13:45:02 -- --eval: snmpotions timeout => 10, retries => 1
2016-03-03 13:45:02 -- --base: Reading Interface Config cache
2016-03-03 13:45:02 -- --base: Checking Config File
2016-03-03 13:45:02 -- --eval: redef snmpotions timeout => 10, retries => 1
2016-03-03 13:45:02 -- --tarp: Starting my_failing_host_wan ->
/a.b.c.d:my_community_string at a.b.c.d:161:::::2
2016-03-03 13:45:02 -- --tarp: &targparser external done:
'/a.b.c.d:my_community_string at a.b.c.d:161:::::2'
2016-03-03 13:45:02 -- --tarp: &targparser complex done:
'/a.b.c.d:my_community_string at a.b.c.d:161:::::2'
2016-03-03 13:45:02 -- --tarp: New simple target [0]
'/a.b.c.d:my_community_string at a.b.c.d:161:::::2':
  Comu:  my_community_string, Host: a.b.c.d
  Opt:   :161:::::2, IPv4: 1
  Conv:
  OID:   ifHCInOctets, ifHCOutOctets
  IfSel: Ip, Ip
  Key:   a.b.c.d, a.b.c.d
2016-03-03 13:45:02 -- --tarp: &targparser simple done: ' $target->[0]{$mode} '
2016-03-03 13:45:02 -- --tarp: &targparser returning: unique = 0
2016-03-03 13:45:02 -- --base: Loading RRD support
2016-03-03 13:45:02 -- --base: Loading default Locale
2016-03-03 13:45:02 -- --base: Starting main Loop
2016-03-03 13:45:02 -- --base: Populate Target object by polling SNMP
and external Datasources
2016-03-03 13:45:02 -- --snpo: run snmpget from
ifHCInOctets&ifHCOutOctets:my_community_string at a.b.c.d
2016-03-03 13:45:02 -- --snpo: (0) Populate ConfCache for a.b.c.d:161:::::2
2016-03-03 13:45:13 -- --snpo: (0) Confcache Match a.b.c.d -> .2
2016-03-03 13:45:13 -- --snpo: (1) Confcache Match a.b.c.d -> .2
2016-03-03 13:45:13 -- --snpo: SNMPGet from
my_community_string at a.b.c.d:161:::::2 --
ipAdEntIfIndex.a.b.c.d,ifHCInOctets.2,ipAdEntIfIndex.a.b.c.d,ifHCOutOctets.2
2016-03-03 13:45:13 -- SNMP Error:
Received SNMP response with error code
  error status: noSuchName
  index 2 (OID: 1.3.6.1.2.1.31.1.1.1.6.2)
SNMPv1_Session (remote host: "a.b.c.d" [a.b.c.d].161)
                  community: "my_community_string"
                 request ID: 1687324163
                PDU bufsize: 8000 bytes
                    timeout: 10s
                    retries: 1
                    backoff: 1)
 at /usr/local/mrtg-2/bin/../lib/mrtg2/SNMP_util.pm line 492.
2016-03-03 13:45:13 -- SNMPGET Problem for ipAdEntIfIndex.a.b.c.d
ifHCInOctets.2 ipAdEntIfIndex.a.b.c.d ifHCOutOctets.2 on
my_community_string at a.b.c.d:161:::::2:v4only
 at /usr/local/mrtg-2/bin/mrtg line 2242.
2016-03-03 13:45:13 -- --snpo: SNMPfound -- 'undef', 'undef', 'undef', 'undef'
2016-03-03 13:45:13 -- 2016-03-03 13:45:02: WARNING: Could not match
host:'my_community_string at a.b.c.d:161:::::2' ref:'Ip' key:'a.b.c.d'
2016-03-03 13:45:13 -- --base: Act on Router/Target my_failing_host_wan
2016-03-03 13:45:13 -- 2016-03-03 13:45:13: ERROR:
Target[my_failing_host_wan][_IN_] ' $target->[0]{$mode} ' did not eval
into defined data
2016-03-03 13:45:13 -- 2016-03-03 13:45:13: ERROR:
Target[my_failing_host_wan][_OUT_] ' $target->[0]{$mode} ' did not
eval into defined data
2016-03-03 13:45:13 -- --base: Get Current values: in:undef,
out:undef, up:undef, name:undef, time:1456983913
2016-03-03 13:45:13 -- --base: Create Graphics
2016-03-03 13:45:13 -- --base: start RRDtool section
2016-03-03 13:45:13 -- --base: maxi:137500137, maxo:137500137
2016-03-03 13:45:13 -- --base: Check for Thresholds
2016-03-03 13:45:13 -- --base: Exit main Loop
2016-03-03 13:45:13 -- --base: Remove Lock Files
2016-03-03 13:45:13 -- --base: Store Interface Config Cache
2016-03-03 13:45:13 -- Use of uninitialized value in concatenation (.)
or string at /usr/local/mrtg-2/bin/../lib/mrtg2/MRTG_lib.pm line 1603.


It appears that MRTG is getting an invalid response when it attempts
to poll the interface counters, however if I poll them manually I do
get a response, i.e.

# snmpget -v2c -c my_community_string a.b.c.d ipAdEntIfIndex.a.b.c.d
IP-MIB::ipAdEntIfIndex.a.b.c.d = INTEGER: 2
# snmpget -v2c -c my_community_string a.b.c.d ifHCInOctets.2
IF-MIB::ifHCInOctets.2 = Counter64: 1488678187624
# snmpget -v2c -c my_community_string a.b.c.d ipAdEntIfIndex.a.b.c.d
IP-MIB::ipAdEntIfIndex.a.b.c.d = INTEGER: 2
# snmpget -v2c -c my_community_string a.b.c.d ifHCOutOctets.2
IF-MIB::ifHCOutOctets.2 = Counter64: 350584652859

So right now I'm stumped. Is anyone on the list able to offer any
insight or advice?

thanks in advance,
Darren