Cloud Python hackers wanted

A while back I had blogged (yes the one with the big uncle Sam pic) asking for people interested in working on a python tool to help migrate ebs AMIs across ec2 regions. Since then the project has progressed a bit:

- Project page is now: https://launchpad.net/ec2-migrate-ebs-ami
- Project is now able to launch two utility instances in the two specified regions, and mount the source volume in the source instance
- What remains is attaching a destination volume, and sync'ing over contents. More details can be found on the blueprint: https://blueprints.launchpad.net/ubuntu/+spec/cloud-server-n-ec2-migrate-region

Development pace has now slowed down significantly, as such I am renewing the call. This is a great way to start getting involved with cloud development, and to join the Ubuntu Cloud and server communities! joining in is "easy", you don't even have to know lots of python to start hacking! The steps are very well defined, and I am happy to help with all steps along the way. If you are thinking about it, go ahead and get in touch with me right now (kim0 on irc, kim0 AT ubuntu.com) or join today's irc meeting (6pm-UTC #ubuntu-cloud https://wiki.ubuntu.com/UbuntuCloudMeeting) and start discussing how you can get involved

Hack on Ubuntu Cloud Utils

Want a chance to hack on the cloud? Yes Ubuntu cloud code? What could possibly be cooler! For us cloud geeks, almost nothing! So, here's the deal. Ubuntu aims to be the best operating system for the cloud, period. Part of that is delivering code and utilities to ease common tasks for people working with Ubuntu as a guest cloud OS. For that I've suggested a utility to migrate ec2 ebs based images across different Amazon regions. Conceptually, what's needed is simple, a tool needs to be written to spin up two virtual servers in the two regions and sync the disks across them then terminate. The Ubuntu server team agrees it's a needed utility, and if done right, it should end up being shipped with Ubuntu (coolness!)

I have written step by step, what needs to be done to accomplish this task in a BluePrint. For those new to that term, BluePrints are Ubuntu's way of writing software specs in the open, where everyone can read and comment on them (don't you just love open!). So, if you're ready to start contributing, click that blueprint link, decide whether you want to hack on this tool, ping me and let's start working on it! For any questions or discussions, please drop by on Freenode IRC #ubuntu-cloud (if you're new to IRC, use this web UI instead http://cloud.ubuntu.com/community/irc-chat/). A great time to discuss this, would be today (Wed) 6pm-UTC during the weekly cloud community meeting. If you miss the meeting no problemo, just talk in the channel anytime, most people are there all the time anyway. I'm already hyper excited. Any questions or feedback? let me know in comments below

Taming Xen Cloud Platform Consoles

At $dayjob I have been studying Xen Cloud Platform inside out, and all in all I like it a lot! The API rox, it's very extensive and after the first couple of days makes a lot of sense. We're also see'ing initial very good performance. One part that was particularly not straightforward was the getting a console for a running VM. I'll document here my findings

So, let's jump straight in, let's list the console of our centos VM

[root@xcp03 ~]# xe console-list vm-uuid=90e68b99-0408-3e4c-1dd1-ea28e98fbbad
uuid ( RO)             : e845ef9d-2075-0773-119e-08875fb61a1f
          vm-uuid ( RO): 90e68b99-0408-3e4c-1dd1-ea28e98fbbad
    vm-name-label ( RO): kamal-CentOS-5.3x64
         protocol ( RO): RFB
         location ( RO): https://10.100.170.12/console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9

The "location" you get is the location of the console for the specified VM. However, if you try visiting that with a VM, you'll get a 404 not found error! Time for some magic. In order to do any of the following, we need a "session" on the Xen Cloud. Since I play with the Xen Python API, I'll use that to get a session

In [7]: import XenAPI

In [8]: session = XenAPI.Session('http://10.100.170.12')

In [9]: session.login_with_password('root', 'woohoo')

In [10]: print session._session
-------> print(session._session)
OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb

So we now have a session cookie "OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb". Now let's try to connect to the VNC server. I'll first try to connect by hand (actually telnet) :)

Try 1: Telnet connection

We need to connect using SSL, which telnet does not support. For that we use "socat", a magical little tool

[akamal@matrix tmp]$ socat TCP4-LISTEN:31337,fork OPENSSL:10.100.170.12:443,verify=0

Let's try to telnet to the locally listening port "31337" and socat will take our connection, wrap it in SSL and connect us to the Xen-API server

[akamal@matrix ~]$ telnet localhost 31337
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
CONNECT /console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb HTTP/1.0

HTTP/1.1 200 OK
Connection: keep-alive
Cache-Control: no-cache, no-store

RFB 003.003

Et voila, all the VNC savvy readers will understand that we have been connected to the VNC server. The RFB greeting denotes the begining of the VNC protocol. A few things to note

1- The CONNECT HTTP method is one big line
2- It contains two parts, the console uuid we got first, the second part being "&session_id=" and the session uuid we got from python
3- We connected with HTTP/1.0 and got a response with HTTP/1.1

Try 2: XVP Connection

Create a custom java key store. Download and convert the certificate format and import it. This is important because if you don't the java viewer won't accept the Xen server's self signed certificate (yuck?)

cd /usr/java/jdk1.6.0_17/bin
./keytool -genkey -alias kimo -keyalg RSA -keystore /tmp/kimo.jks
[akamal@matrix bin]$ scp root@10.100.170.12:/etc/xensource/xapi-ssl.pem /var/tmp/
xapi-ssl.pem                                                                                                               100% 1108     1.1KB/s   00:00
[akamal@matrix bin]$ openssl x509 -in /var/tmp/xapi-ssl.pem -inform PEM -out /var/tmp/newxapi.crt -outform DER

[akamal@matrix bin]$ ./keytool -import -trustcacerts -file /var/tmp/newxapi.crt -alias XCP_ALIAS -keystore /tmp/kimo.jks
Enter keystore password:
Owner: CN=10.100.170.13
Issuer: CN=10.100.170.13
Serial number: bb01ad6ac4cdc83e
Valid from: Wed Dec 23 15:27:24 EET 2009 until: Sat Dec 21 15:27:24 EET 2019
Certificate fingerprints:
         MD5:  84:90:D7:CF:7F:76:95:9E:2C:52:5A:66:C7:85:DB:58
         SHA1: BB:7B:C9:B1:47:19:42:F6:75:02:84:55:A1:05:67:7B:A6:B9:4E:82
         Signature algorithm name: SHA1withRSA
         Version: 1
Trust this certificate? [no]:  yes
Certificate was added to keystore

Perfect, now let's get the XVP source code. XVP is basically tightvnc with protocol additions that the Citrix hackers have added to better control VMs

[akamal@matrix tmp]$ wget -q http://www.xvpsource.org/xvp-1.3.2.tar.gz
[akamal@matrix tmp]$ tar xzf xvp-1.3.2.tar.gz
[akamal@matrix tmp]$ cd xvp-1.3.2/
[akamal@matrix xvp-1.3.2]$ cd viewer/

Apply the following patch

diff -ur ./HTTPSConnectSocket.java /tmp/xvp/xvp-1.3.2/viewer/HTTPSConnectSocket.java
--- ./HTTPSConnectSocket.java   2009-11-30 15:37:05.000000000 +0200
+++ /tmp/xvp/xvp-1.3.2/viewer/HTTPSConnectSocket.java   2009-12-29 19:14:56.165479016 +0200
@@ -46,8 +46,10 @@
     ssl = (SSLSocket)ssf.createSocket(proxyHost, proxyPort);
     ssl.startHandshake();

+    System.out.print("CONNECT " + host +
+                             " HTTP/1.0\r\n\r\n");
     // Send the CONNECT request
-    ssl.getOutputStream().write(("CONNECT " + host + ":" + port +
+    ssl.getOutputStream().write(("CONNECT " + host +
                             " HTTP/1.0\r\n\r\n").getBytes());

     // Read the first line of the response
@@ -55,8 +57,8 @@
     String str = is.readLine();

     // Check the HTTP error code -- it should be "200" on success
-    if (!str.startsWith("HTTP/1.0 200 ")) {
-      if (str.startsWith("HTTP/1.0 "))
+    if (!str.startsWith("HTTP/1.1 200 ")) {
+      if (str.startsWith("HTTP/1.1 "))
        str = str.substring(9);
       throw new IOException("Proxy reports \"" + str + "\"");
     }

Build the code

[akamal@matrix viewer]$ make
javac -J-Xmx16m -target 1.1 -source 1.3 -nowarn -O VncViewer.java RfbProto.java AuthPanel.java VncCanvas.java VncCanvas2.java OptionsFrame.java ClipboardFrame.java ButtonPanel.java DesCipher.java CapabilityInfo.java CapsContainer.java RecordingFrame.java SessionRecorder.java SocketFactory.java ReloginPanel.java HTTPConnectSocketFactory.java HTTPConnectSocket.java HTTPSConnectSocketFactory.java HTTPSConnectSocket.java InStream.java MemInStream.java ZlibInStream.java XvpConfirmDialog.java
jar -J-Xmx16m cfm VncViewer.jar MANIFEST.MF VncViewer.class RfbProto.class AuthPanel.class VncCanvas.class VncCanvas2.class OptionsFrame.class ClipboardFrame.class ButtonPanel.class DesCipher.class CapabilityInfo.class CapsContainer.class RecordingFrame.class SessionRecorder.class SocketFactory.class ReloginPanel.class HTTPConnectSocketFactory.class HTTPConnectSocket.class HTTPSConnectSocketFactory.class HTTPSConnectSocket.class InStream.class MemInStream.class ZlibInStream.class XvpConfirmDialog.class

Now let's connect using one more magical and undocumented command line

[akamal@matrix viewer]$ java -Djavax.net.ssl.trustStore=/tmp/kimo.jks -Xmx64m -jar VncViewer.jar HOST "/console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb" PORT 443 PROXYHOST1 10.100.170.12 PROXYPORT1 443 SocketFactory "HTTPSConnectSocketFactory"
Initializing...
Connecting to /console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb, port 443...
HTTPS CONNECT via proxy 10.100.170.12 port 443
CONNECT /console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb HTTP/1.0

Connected to server
RFB server supports protocol version 3.3
Using RFB protocol version 3.3
No authentication needed
Desktop name is XenServer Virtual Terminal
Desktop size is 640 x 384
Using Tight/ZRLE encodings
Closing window
Disconnecting
Updates received: 2 (40 rectangles + 1 pseudo), 0.02 updates/sec
Rectangles: Tight=0(JPEG=0) ZRLE=0 Hextile=40 Raw=0 CopyRect=0 other=0
Pixel data: 983040 bytes, 4295 compressed, ratio 228.88
RFB socket closed

et voila, here's how it looks when it works




Yoohooo, hurray for undocumented ^#^&@%^#%

This post would not have been possible without great help from a friend and guru developer Ahmed Soliman. Also Abdelrahman Hussein helped with some final touches

Hoping that post is of help to some poor souls. I still like XCP though :)

Traffic accounting with SNMP and Python on Windows

If you're not on a flat-rate ISP data-plan, and your ISP made you pay extra last month such as myself :) You might be interested in monitoring your home bandwidth. A nifty little features of most routers or ADSL modems, is support for SNMP. Since both me and my old folks use the same ADSL line, any host based monitoring solution wouldn't really work. I had to monitor the usage using SNMP. Since my folks are using Windows, the solution had to work on that. The solution overview is:

1- Enable SNMP on ADSL modem
2- Use a windows scheduled task to run a windows Batch file, that uses SNMP-tools for windows to pull data from the router
3- Use a windows scheduled task to run a python script that processes the batch file output and computes the total traffic

Data from the batch files are written in a file signifying the current month (example 05.txt for May). Here is the code for the batch file

cd e:\traffic

e:\

FOR /F "TOKENS=1* DELIMS= " %%A IN ('DATE/T') DO SET CDATE=%%B

FOR /F "TOKENS=1,2 eol=/ DELIMS=/ " %%A IN ('DATE/T') DO SET mm=%%B



echo %mm%

snmpget.exe -c public -O v -v 1 192.168.11.1 .1.3.6.1.2.1.2.2.1.10.12 >> %mm%.txt

 

Note that the IP 192.168.11.1 should be replaced with the internal IP of your ADSL router. Also, the SNMP OID .1.3.6.1.2.1.2.2.1.10.12 (symbolically: IF-MIB::ifInOctets.12) is the incoming octets on network interface "12". You might need to change that last "12" to represent your network topology. You can use the "snmpwalk" command to list your interface names like:

snmpwalk -c public -v1 192.168.11.1
this yields
...
IF-MIB::ifDescr.12 = STRING: ppp0
...

I know that my ppp0 is the link to my ISP, so that's the interface I used

And here is the code for the python script

# -*- coding: utf-8 -*-

import glob

import locale

import os

locale.setlocale(locale.LC_ALL, '')

reports = sorted(glob.glob('??.txt'))

totalsReport = open('totals.txt','w')

totalsReport.write(' Month      Traffic \n')

totalsReport.write('====================\n')

for report in reports:

        datastring = open(report,'r').readlines()

        data = [ int(datastring[i].strip().split()[1]) for i in range(len(datastring)) ]

        diff = [ data[i+1] - data[i] for i in range(len(data) - 1) ]

        trafficDeltas = [ (data[i+1],diff[i])[diff[i] > 0] for i in range(len(diff)) ]

        totalTraffic = sum(trafficDeltas)

        totalsReport.write(' %s => %s \n'% (report.replace('.txt','') , locale.format('%d', totalTraffic, True)) )



totalsReport.close()

Glancing over the Python code, you can already see I'm a list comprehension addict :) The code reads the snmp bytes values from ??.txt (example: 05.txt), computes bandwidth deltas, handles counter resets (32bit overflow, or router loosing power) and computes totals. Then writes the total to a file named "totals.txt"

This solution is resistant to the router loosing power and resetting its internal traffic counter back to zero. This is because the data is continuously being saved to non-volatile medium (PC's hard-disk). Also, the Windows PC running those two scripts, does NOT need to be running 24x7 .. Just most of the time. If however the PC is powered off, AND the modem power cycles or hangs, you might loose some accuracy, the results will still be fairly valid though. Hope that's helpful to anyone out there