Saturday, May 30, 2009

Traffic accounting with SNMP and Python on Windows

If you're not on a flat-rate ISP data-plan, and your ISP made you pay extra last month such as myself :) You might be interested in monitoring your home bandwidth. A nifty little features of most routers or ADSL modems, is support for SNMP. Since both me and my old folks use the same ADSL line, any host based monitoring solution wouldn't really work. I had to monitor the usage using SNMP. Since my folks are using Windows, the solution had to work on that. The solution overview is:

1- Enable SNMP on ADSL modem
2- Use a windows scheduled task to run a windows Batch file, that uses SNMP-tools for windows to pull data from the router
3- Use a windows scheduled task to run a python script that processes the batch file output and computes the total traffic

Data from the batch files are written in a file signifying the current month (example 05.txt for May). Here is the code for the batch file

cd e:\traffic



FOR /F "TOKENS=1,2 eol=/ DELIMS=/ " %%A IN ('DATE/T') DO SET mm=%%B

echo %mm%

snmpget.exe -c public -O v -v 1 . >> %mm%.txt

Note that the IP should be replaced with the internal IP of your ADSL router. Also, the SNMP OID . (symbolically: IF-MIB::ifInOctets.12) is the incoming octets on network interface "12". You might need to change that last "12" to represent your network topology. You can use the "snmpwalk" command to list your interface names like:

snmpwalk -c public -v1
this yields
IF-MIB::ifDescr.12 = STRING: ppp0

I know that my ppp0 is the link to my ISP, so that's the interface I used

And here is the code for the python script

# -*- coding: utf-8 -*-

import glob

import locale

import os

locale.setlocale(locale.LC_ALL, '')

reports = sorted(glob.glob('??.txt'))

totalsReport = open('totals.txt','w')

totalsReport.write(' Month      Traffic \n')


for report in reports:

        datastring = open(report,'r').readlines()

        data = [ int(datastring[i].strip().split()[1]) for i in range(len(datastring)) ]

        diff = [ data[i+1] - data[i] for i in range(len(data) - 1) ]

        trafficDeltas = [ (data[i+1],diff[i])[diff[i] > 0] for i in range(len(diff)) ]

        totalTraffic = sum(trafficDeltas)

        totalsReport.write(' %s => %s \n'% (report.replace('.txt','') , locale.format('%d', totalTraffic, True)) )


Glancing over the Python code, you can already see I'm a list comprehension addict :) The code reads the snmp bytes values from ??.txt (example: 05.txt), computes bandwidth deltas, handles counter resets (32bit overflow, or router loosing power) and computes totals. Then writes the total to a file named "totals.txt"

This solution is resistant to the router loosing power and resetting its internal traffic counter back to zero. This is because the data is continuously being saved to non-volatile medium (PC's hard-disk). Also, the Windows PC running those two scripts, does NOT need to be running 24x7 .. Just most of the time. If however the PC is powered off, AND the modem power cycles or hangs, you might loose some accuracy, the results will still be fairly valid though. Hope that's helpful to anyone out there


arthurH said...

Thanks for posting this. Unfortunately the screen shots have truncated lines making it impossible to see all your code.

Ahmed Kamal said...

Hi arthurH
I've added scroll bars .. You seem to be using a smallish screen browser. Anyway, it should work fine now

arthurH said...

Even when I maximised my browser window the python code was truncated. My desktop screen is 1152 x 864 which is larger than most netbooks.

The scroll bars work fine. Thanks for the addition.

Ahmed El Gamil said...

Extremely Geeky !
Nice piece of art really ..

Ahmed Kamal said...

Thanks Ahmed for the compliment :D

ppetersen said...

I am puzzled:

By using IF-MIB::ifInOctets.26, which corresponds to my ppp interface of the router (called: COM1) I always get much higher numbers than when looking at the bytes of a downloaded file.

For instance, I download a 20 MB file (and take care that no other programs load something, like Instant Messengers etc.)... I would expect that snmp tells me that about 20 MB have been downloaded.

snmpget or snmpnetstatus tells me that more than 40 MB have been downloaded.

Moreover, when doing nothing at all (just being online), the bytes counter grows slowly but remarkably.

This is different behaviour as compared to using a "direct" ppp connection and using, for instance:

xnetload -if ppp0

When in this case I download a 20 MB file, xnetload tells me that it was 20 MB that I downloaded, no more, no less.

Why is the number of the ppp interface when using snmp on the router so much higher?

Would be glad to hear more about this,


thanks ;-)

dissertation said...

it's good to see this information in your post, i was looking the same but there was not any proper resource, thanx now i have the link which i was looking for my research.

Accounting Dissertation Proposal

Term Papers said...

Thanks for posting this. Unfortunately the screen shots have truncated lines making it impossible to see all your code.

Ahmed Kamal said...

There is no image .. It's pure text. Just use a different browser or maximize it on a higher resolution PC. You should be able to drag your mouse and select all text as well. Ciao