Custom snmpd extension for port checking

As weird as it sounds, recently I had a task to accomplish port checks without access to the LAN on which daemons listen for connections. Speaking of a monitoring solution, the obvious choice was SNMP, which is the most widespread means of getting health information from network-attached devices, anyway. We perform an “indirect” port check, meaning that it’s sufficient for us to know that a process is listening on a given port without trying to communicate with it.

I found three alternative methods to achieve such a port check via SNMP. They have one thing in common: they utilize Net-SNMP server and/or client binaries.

  • The snmpnetsat utility. A straightforward answer to our question. Provides a netstat-like listing of active connections on the remote host. But it’s got its flaw. It queries the older MIB-II (RFC1213) objects tcpConnectionTable and udpTable, which don’t support IPv6. As a result, server processes listening on all IPs on an IPv6-enabled system doesn’t show up in the list.
  • tcpListenerTable and udpEndpointTable objects. These two were defined as part of TCP-MIB (RFC4022) and UDP-MIB (RFC4113) respectively. As their names suggest, these structures hold all listener processes on the system. Apparently indexes cannot be queried directly, so one needs to walk through the whole table to find a specific port. At least this is the case in Net-SNMP v5.5, but I didn’t investigate any further.
  • Custom snmpd extension. Of course we don’t want to develop a new MIB module for the sake of this single task. Several other ways exist to extend snmpd functionality – they are all described in snmpd.conf man page. The basic idea is to collect the listener data with a script and let snmpd transfer it to the client in a simple form.

So we have two viable options here. One is to let the client (monitoring system) extract the information it needs from tcpListenerTable and udpEndpointTable. The other is to leave decision making to the server (monitored host) and let it serve the other party with a simple OK/NOT-OK value. Needless to say, former was the winner in this competition because of the smaller impact on host configurations.

But, surprise-surprise, I created a working POC solution for the other one, too. In memoriam…

Extending snmpd

As I already mentioned, my idea of an extension was to run an external script from the daemon and provide the client with a single value it needed. An SNMP-aware solution (denoted as a MIB-Specific Extension Command by the documentation) is what fits. In this scenario the daemon acts lazy and delegates all work to the external command: it transfers object ID and request type (GET, GETNEXT, SET) and expects an object value or a report on the outcome.

All we have to do is assign an unused portion of the MIB-tree to the script with a pass directive and grant view rights to the client (“systemview” is “public” by default).

 pass .1.3.6.1.4.1.2021.51 /path/to/script.sh
 view systemview included .1.3.6.1.4.1.2021.51

So it’s now fully up to us, what happens when an object in this subtree is requested. Our script serves objects which have numeric OIDs of the form “ROOT.PORT”, where ROOT is the subtree’s relative root and PORT can be any valid port number. The object value returned corresponds to the number of processes bound to that port. E.g. suppose ROOT is .1.3.6.1.4.1.2021.51, if a GET request is received on .1.3.6.1.4.1.2021.51.22, then the response will hold the number of processes listening on port 22. Obviously the value is 0, if there’s no such process. When responding to GETNEXT requests, only those ports are taken into consideration which are bound to a process.

 #!/bin/sh
 
 # Find ports bound to a single IP address. (Empty means ALL.)
 IP=
 
 # OID of the subtree root object assigned to this extension (with leading dot).
 ROOT=.1.3.6.1.4.1.2021.51
 
 # Paths to binaries
 LSOF=/usr/sbin/lsof
 GREP=/bin/grep
 SORT=/bin/sort
 SED=/bin/sed
 HEAD=/usr/bin/head
 WC=/usr/bin/wc
 
 OID=$  2
 
 # Check wether OID is valid (it's a direct descendant of ROOT or ROOT itself)
 echo $  OID | $  GREP -E ^$  ROOT.?[0-9]*$   > /dev/null || exit
 
 # Extracting the last portion of the OID
 [ "$  OID" != "$  ROOT" ] && port=`echo $  OID | $  GREP -o "[0-9]*$  "` || port=0
 
 [ "$  IP" != "" ] && IP="@$  IP"
 
 case $  1 in
 
 	"-g")
 
 		# Checking if port is in valid range
 		[ "$  port" -gt "65535" ] && exit
 
 		[ "$  port" = "0" ] && echo -e "$  OIDnstringn$  0" && exit
 	;;
 
 	"-n")
 
 		(( port = $  port + 1 ))
 
 		# Checking if port is in valid range
 		[ "$  port" -gt "65535" ] && exit
 
 		# Searching next listener port on the system
 		port=`($  LSOF -i$  IP:$  port-65535 -sTCP:LISTEN -Fnp -P; $  LSOF -iUDP$  IP:$  port-65535 -Fnp -P) | $  GREP "^n" | $  GREP -o "[0-9]*$  " | $  SORT -n | $  HEAD -1`
 		[ "$  port" == "" ] && exit
 		OID="$  ROOT.$  port"
 	;;
 
 	"-s")
 
 		# Refusing SET requests
 		echo not-writable; exit
 	;;
 
 	*)
 		exit
 	;;
 esac
 
 # Output for snmpd (number of processes)
 echo -e "$  OIDninteger"
 ($  LSOF -iTCP$  IP:$  port -sTCP:LISTEN -Fp; $  LSOF -iUDP$  IP:$  port -Fp) | $  SORT -u | $  WC -l

Just an example on how this can be used from the client side. Let’s add a command to Nagios configuration.

 define command{
         command_name    check_snmp_port
         command_line    $  USER1$  /check_snmp -H $  HOSTADDRESS$   -C public -c 1: -o .1.3.6.1.4.1.2021.51.$  ARG1$  
 }

Where $ USER1$ is a user macro holding the path to the Nagios plugin directory, where check_snmp resides. Obviously the first argument ($ ARG1$ ) is the port number, so the check_command directive in a service definition should be something like this: check_snmp_port!22.