Monday, September 20, 2010

what I do

I started with Linux and Open source software around 1999, since then and till today I was amazed about the helpful community that form around different open source projects. In the early days, whenever I'd hit a bug or don't know how to do something, I'd jump over to IRC and someone was always there to help debug the problem! This just felt amazingly empowering, I remember thinking to myself, hell this is way much better than what you get with commercial software! In a few years, I transformed from being a passive user, into someone who tries to help IRC users every time someone has a problem. The culture of open source got me I guess. I started giving presentations at Universities, writing for local IT magazines for free to help spread what FOSS is all about. I always wanted to contribute much more to the free software world, however limited free time available was always an issue. Now that my day job is to work with and help grow the community around Ubuntu, I feel extremely excited and thankful for Canonical for giving me this opportunity.

Being the newest member of the "horsemen" team having joined a little over a month now, I feel like I haven't done enough yet. Nevertheless, I'd like to mention a few of the things that have been keeping me busy the past few weeks

  • The very first thing I had done with Canonical was to write the Ubuntu Server Map application. The idea behind it is to encourage and spur the community behind Ubuntu server. Basically the aim is to make Ubuntu server users feel part of a huge user community all over the world. What the application does is to detect the visitor user's city using his IP address after the user accepts (anonymous process) and then marks that city with a cute little orange Ubuntu logo. So far the Map is full of orange Ubuntu logos. It really feels great to be part of such a world wide community of Ubuntu server and cloud users and contributors

  • As part of helping the Ubuntu cloud community grow around open source cloud technologies, I have focused on consolidating any fragmented communication paths available. Thus far, the Ubuntu cloud community has the following #ubuntu-cloud as the official IRC channel for everything Ubuntu cloud related. And the recently created Ubuntu Cloud Forum as the official Ubuntu cloud forum. The Ubuntu Cloud mailing list is an alternate community communication venue as well

  • Recently I've been putting a lot of focus on creating a Ubuntu cloud portal, which aims to be a central hub for the Ubuntu cloud community. You can read all about the portal specs and give me feedback over IRC (kim0 in #ubuntu-cloud). The portal should provide a list of rolling news that relate to Ubuntu cloud, helping interested community members always be on top of all the new happenings. It also helps community new comers by becoming their one stop shop with links to all documentation and support channels. As well as guide new contributers on how to get involved

  • Something which I am thinking about and which will definitely take a lot of focus soon is studying potential hurdles in the way of new contributers to Ubuntu server and cloud. Basically how to make it easy and more fun for newcomers to join in, find all the information they need in place and start contributing and engaging with the community. Part of that is giving tutorials over IRC or other mediums as well as sponsorship and guidance along the way

It is definitely great being part of such a great community such as the Ubuntu one, and I hope the next period is going to be very exciting for the open source Cloud communities in general

Monday, September 6, 2010

Why our Internet2.0 is broken

The modern Internet which I'll refer to as Internet2.0 is being seen as a new applications platform. It is no longer a series of "pages" that you click through, it is rather a collection of applications. I just needed this intro in case you still thought of the Internet as pages. So the Internet is now the Operating System, and different web sites (Gmail, Facebook, Twitter...etc) are the new applications if you will. I've got news for you, this Internet2.0 thing, is horribly broken! Here is why

To begin understanding the kind of problems we have to go through using the online systems of today, let us apply the same online mechanisms, to standard old and boring desktop apps. Let's imagine the following workflow, You are 3 friends working on an important report, and you all share your progress online, each person via his blog

  • You login to your computer, you start your email application. You find your friend has edited the report, emailed you the new copy and blogged about his progress

  • You check your email, find the attachment, you download that

  • You must copy the attachment over a USB stick in order to be able to open it in any other application. You copy it to the USB stick

  • You open your word processor. You authenticate to it!

  • You plugin your USB stick, open the document, edit it, save it

  • Re-copy it to the USB stick

  • Re-open your email application, upload the attachment, send it

  • You visit your first friend's blog, you add a comment that you updated the report.

  • Your third friend is not notified of your comment


You get the idea. If that user experience sounds horrible, it is pretty similar to what we face today with web applications especially if you want to use different applications from different providers in concert. It is just plain broken. Here is my criticizm spot on

  • Why do I have to login separately to each and every web application (Google, Facebook, Zoho, Twitter, MS...)? On my Linux PC I don't have to go through that

  • Why do I have to teach each web application my social graph. Reconnect to all my friends on facebook, then twitter, then Google Buzz...? The connectivity between me and my friend belongs to us, it does not belong to Facebook. Any other app on the Internet which I allow to access this data, should be able to. It should not be held captive by the likes of Facebook

  • Assuming I love to use a live.com email address (which I don't :) and like to use Google docs to edit my email attachments. Why do I have to download the attachment to my PC first, reupload to Google to edit, save, download, upload, email, yikes! The Unix architecture designed 10s of years ago, was all about sharing data between apps (pipes), why in this modern age are we unable to easily connect different webapps especially from different providers

  • When I write a comment on my friend's blog, the comment is MINE. I created it, I own it. My friend's blog maybe displaying it currently, but it should not own it! If my friends are interested in seeing my comments on every website I visit, and if I allow them to, they should be able to do so very easily. The 20 comments I've written today, should not die if the websites I have written them onto decide to die

  • The same actually goes for using online editors a la Google docs. Google should not "own" the document. The document is mine. It should be stored in a place I control. I may "Allow" Google web based editor to read/write it now, because I "choose" to. Because it maybe the best editor around. Not because I have to, and not because I need to "migrate" my data off of Google should I want to use something else. If I decide to use Zoho editor tomorrow, I should be able to allow it to access my data in-place. Just like how on the desktop one can use MS Office, OpenOffice and Mac Pages to open a presentation on one's desktop



As a new application pops up, say like Apple's new Ping service, you need to (again) rebuild your social graph teaching it all your friends. Afterwards any music you purchase will show on Ping, but won't really show up on Facebook because Apple and Facebook couldn't agree on that. Can you say that again, "I" purchased a music track, and I want to tell "MY" friends about it, and I can't because Apple and Facebook couldn't agree! Can you see how horribly broken our Internet2.0 is


How I see things could improve is as follows. Each user needs to own a certain web exposed storage space somewhere. All of my online trails (documents, friendship, Likes, comments, blogs...) should live there. I should "allow" Google docs to read/write to those document if I like it. Tomorrow I could revoke that access and allow Zoho for example or any other online editor. My connections to my friends should be stored in that storage space. Any application that I allow can see who all or some of my friends are. For example I can allow Linkedin to access my work friends, while allowing Ping to access my music friends. If I make a comment somewhere on the Internet, this is content I created, If I Facebook style "Like" something, it is again content that revolves around me. It should be stored in my storage area, and Facebook should be notified of it and allowed to display it should I want to. This architecture makes it quite easy for a new young startup to create a Facebook or Gmail killer tomorrow. Since the application from day one will have access to all my data, emails, friends, comments, blogs...etc i.e. as much content as I want to give it. We become no longer locked into different online service providers. We can switch at will. This is the way how it can and should be done. The diaspora and FreedomBox folks are doing some awesome work and designing their systems along similar lines of thought. They are however more ambitious, and are after user data confidentiality, while all I'm asking for is open-data access and portability. They want to replace all of today's web applications with distributed clones that you can run inside your home. That would be fantastic, however I still it as advantageous and easier if a user can use "closed" cloud apps like Gmail or Facebook should she want to, and switch to a different provider at will, while still owning her online digital trail and every piece of content she ever created

Monday, August 9, 2010

Ubuntu Server 10.04.1 virtual release party

Ubuntu Server LTS 10.04.1 is targeting to be released this Thursday on the 12th of August. This point release is especially important for the "server" variant of Ubuntu. This is because many conservative sys-admins wait for the first point release before deploying a server OS. The idea being, any bugs the fine QA people at Canonical might have missed, would be caught by the community during the period between the GA and point releases. This means that for many, this next 12th of August is the "actual" release date for Lucid server LTS! In short, on the 12th Ubuntu Server 10.04.1 is ready to rock the world!

In order to celebrate this awesomeness, we'll be having our own little server release party in the clouds :) What a mouthful, well basically I've put up a nice little web application that tracks all cities around the world that are running Ubuntu Server 10.04. Of course you will need to hit that web app with your server box first so that your city becomes registered and shows up on the map! So if you're already running 10.04 server, be sure to hit that application. If you're not, then you should be! Go grab yourself the 10.04 server iso, install it, then hit the app with it

But wait a minute, you say you don't have a web browser on your server box. No problemo. I'll list a few cool ways (well in a serverish way at least) to hit that web app without leaving your comfortable bash shell! Hey community, if you can think of cooler CLI methods, be sure to put them in your comments. if something is very cool, I'll add it to this post. Here we go


1- elinks --dump http://maps.ubuntu.com/hit/

2- curl http://maps.ubuntu.com/hit/

3- wget http://maps.ubuntu.com/hit/

4- python -c 'import urllib; urllib.urlopen("http://maps.ubuntu.com/hit/").read()'

5- /bin/echo -e 'GET /hit/ HTTP/1.1\nHost: maps.ubuntu.com\n\n' | socat - tcp:maps.ubuntu.com:80


Well I guess that should be enough to wet your appetite. Of course "technically" you don't have to visit the web app using your server box. Hitting it from your Ubuntu desktop has the same effect, but where's the fun in that! If you're running Ubuntu Servers in multiple cities, you are encouraged to hit that app from all cities in which you operate. If you coordinate the hit with some cool stuff (maybe a puppet recipe or bounce over ec2 or something of that sort) let me know about it :)

Whichever CLI way you choose to hit the app with, you'd be missing out on seeing lots of cute little Ubuntu logos over a world-wide Google map. Now that is something you wouldn't wanna miss, so be sure to visit the app from your graphical browser as well to view the list of marked cities running Ubuntu Server. On release day (12th of Aug) it would be very cool to have every major city in the world marked on that map, so please spread the word as much as you can (yes that means retweet it, digg it, slashdot it, tell you mom about it ...etc)

Tuesday, August 3, 2010

bash oneliner, get GPS location + street address

Hey there folks,

It's been quite some time since I last blogged, just been busy with $reallife. Today I'll show you how to an answer the eternal question of "Where am I" from the comfort of your bash shell

The following code only works if:
- You run it in a root shell (Ubuntu users do: "sudo -i" then paste it)
- You are connected via Wifi to some access point
- Your wireless adapter is called wlan0 (otherwise replace it with the correct name)
- You're using a Linux system or something similar (i.e. Windows won't really work here)

et voila


/bin/echo '{"version": "1.1.0","host": "maps.google.com","request_address": true,"address_language": "en_GB", "wifi_towers": [{"mac_address": "' $( iwlist wlan0 scan | grep Address | head -1 | awk '{print $5}' | sed -e 's/ //g' ) '","signal_strength": 8,"age": 0}]}' | sed -e 's/" /"/' -e 's/ "/"/g' > /tmp/post.$$ && curl -X POST -d @/tmp/post.$$ http://www.google.com/loc/json | sed -e 's/{/\n/g' -e 's/,/\n/g'


I didn't really spend much time cleaning it up, since I'm busy, so that's like the first thing that worked. If any of you guys can skip writing to the file, let me know and I'll update it

PS: If the returned information is wrong, or some kind of "unknown" .. Consider yourself lucky, very lucky! That means Google does not (yet?) know where your wifi AP is. For the rest of us .. tin-foil all the way

Saturday, April 10, 2010

Android map navigation in Cairo

woot! Android map navigation working in Cairo now! See how it works in the following animated sequence. A few cool things to notice:

1- It can start navigation from my location, which it knows by cell tours or GPS
2- I don't write my destination, I speak it out, it travels to google servers, gets recognized and comes back as text
3- It searches local areas, to identify the exact destination
4- then provides turn-by-turn instructions .. woohoo .. the future is now

android-map-magic

Now the negatives:
1- No voice instructions .. you actually have to read ;)
2- The choice of routes, is not always what would make sense to me. I'll try to see how can I make it change its routing decisions

Thanks Google

Friday, January 1, 2010

Taming Xen Cloud Platform Consoles

At $dayjob I have been studying Xen Cloud Platform inside out, and all in all I like it a lot! The API rox, it's very extensive and after the first couple of days makes a lot of sense. We're also see'ing initial very good performance. One part that was particularly not straightforward was the getting a console for a running VM. I'll document here my findings

So, let's jump straight in, let's list the console of our centos VM


[root@xcp03 ~]# xe console-list vm-uuid=90e68b99-0408-3e4c-1dd1-ea28e98fbbad
uuid ( RO) : e845ef9d-2075-0773-119e-08875fb61a1f
vm-uuid ( RO): 90e68b99-0408-3e4c-1dd1-ea28e98fbbad
vm-name-label ( RO): kamal-CentOS-5.3x64
protocol ( RO): RFB
location ( RO): https://10.100.170.12/console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9


The "location" you get is the location of the console for the specified VM. However, if you try visiting that with a VM, you'll get a 404 not found error! Time for some magic. In order to do any of the following, we need a "session" on the Xen Cloud. Since I play with the Xen Python API, I'll use that to get a session


In [7]: import XenAPI

In [8]: session = XenAPI.Session('http://10.100.170.12')

In [9]: session.login_with_password('root', 'woohoo')

In [10]: print session._session
-------> print(session._session)
OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb


So we now have a session cookie "OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb". Now let's try to connect to the VNC server. I'll first try to connect by hand (actually telnet) :)

Try 1: Telnet connection

We need to connect using SSL, which telnet does not support. For that we use "socat", a magical little tool

[akamal@matrix tmp]$ socat TCP4-LISTEN:31337,fork OPENSSL:10.100.170.12:443,verify=0


Let's try to telnet to the locally listening port "31337" and socat will take our connection, wrap it in SSL and connect us to the Xen-API server


[akamal@matrix ~]$ telnet localhost 31337
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
CONNECT /console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb HTTP/1.0

HTTP/1.1 200 OK
Connection: keep-alive
Cache-Control: no-cache, no-store

RFB 003.003


Et voila, all the VNC savvy readers will understand that we have been connected to the VNC server. The RFB greeting denotes the begining of the VNC protocol. A few things to note

1- The CONNECT HTTP method is one big line
2- It contains two parts, the console uuid we got first, the second part being "&session_id=" and the session uuid we got from python
3- We connected with HTTP/1.0 and got a response with HTTP/1.1

Try 2: XVP Connection

Create a custom java key store. Download and convert the certificate format and import it. This is important because if you don't the java viewer won't accept the Xen server's self signed certificate (yuck?)


cd /usr/java/jdk1.6.0_17/bin
./keytool -genkey -alias kimo -keyalg RSA -keystore /tmp/kimo.jks
[akamal@matrix bin]$ scp root@10.100.170.12:/etc/xensource/xapi-ssl.pem /var/tmp/
xapi-ssl.pem 100% 1108 1.1KB/s 00:00
[akamal@matrix bin]$ openssl x509 -in /var/tmp/xapi-ssl.pem -inform PEM -out /var/tmp/newxapi.crt -outform DER

[akamal@matrix bin]$ ./keytool -import -trustcacerts -file /var/tmp/newxapi.crt -alias XCP_ALIAS -keystore /tmp/kimo.jks
Enter keystore password:
Owner: CN=10.100.170.13
Issuer: CN=10.100.170.13
Serial number: bb01ad6ac4cdc83e
Valid from: Wed Dec 23 15:27:24 EET 2009 until: Sat Dec 21 15:27:24 EET 2019
Certificate fingerprints:
MD5: 84:90:D7:CF:7F:76:95:9E:2C:52:5A:66:C7:85:DB:58
SHA1: BB:7B:C9:B1:47:19:42:F6:75:02:84:55:A1:05:67:7B:A6:B9:4E:82
Signature algorithm name: SHA1withRSA
Version: 1
Trust this certificate? [no]: yes
Certificate was added to keystore


Perfect, now let's get the XVP source code. XVP is basically tightvnc with protocol additions that the Citrix hackers have added to better control VMs


[akamal@matrix tmp]$ wget -q http://www.xvpsource.org/xvp-1.3.2.tar.gz
[akamal@matrix tmp]$ tar xzf xvp-1.3.2.tar.gz
[akamal@matrix tmp]$ cd xvp-1.3.2/
[akamal@matrix xvp-1.3.2]$ cd viewer/


Apply the following patch


diff -ur ./HTTPSConnectSocket.java /tmp/xvp/xvp-1.3.2/viewer/HTTPSConnectSocket.java
--- ./HTTPSConnectSocket.java 2009-11-30 15:37:05.000000000 +0200
+++ /tmp/xvp/xvp-1.3.2/viewer/HTTPSConnectSocket.java 2009-12-29 19:14:56.165479016 +0200
@@ -46,8 +46,10 @@
ssl = (SSLSocket)ssf.createSocket(proxyHost, proxyPort);
ssl.startHandshake();

+ System.out.print("CONNECT " + host +
+ " HTTP/1.0\r\n\r\n");
// Send the CONNECT request
- ssl.getOutputStream().write(("CONNECT " + host + ":" + port +
+ ssl.getOutputStream().write(("CONNECT " + host +
" HTTP/1.0\r\n\r\n").getBytes());

// Read the first line of the response
@@ -55,8 +57,8 @@
String str = is.readLine();

// Check the HTTP error code -- it should be "200" on success
- if (!str.startsWith("HTTP/1.0 200 ")) {
- if (str.startsWith("HTTP/1.0 "))
+ if (!str.startsWith("HTTP/1.1 200 ")) {
+ if (str.startsWith("HTTP/1.1 "))
str = str.substring(9);
throw new IOException("Proxy reports \"" + str + "\"");
}



Build the code


[akamal@matrix viewer]$ make
javac -J-Xmx16m -target 1.1 -source 1.3 -nowarn -O VncViewer.java RfbProto.java AuthPanel.java VncCanvas.java VncCanvas2.java OptionsFrame.java ClipboardFrame.java ButtonPanel.java DesCipher.java CapabilityInfo.java CapsContainer.java RecordingFrame.java SessionRecorder.java SocketFactory.java ReloginPanel.java HTTPConnectSocketFactory.java HTTPConnectSocket.java HTTPSConnectSocketFactory.java HTTPSConnectSocket.java InStream.java MemInStream.java ZlibInStream.java XvpConfirmDialog.java
jar -J-Xmx16m cfm VncViewer.jar MANIFEST.MF VncViewer.class RfbProto.class AuthPanel.class VncCanvas.class VncCanvas2.class OptionsFrame.class ClipboardFrame.class ButtonPanel.class DesCipher.class CapabilityInfo.class CapsContainer.class RecordingFrame.class SessionRecorder.class SocketFactory.class ReloginPanel.class HTTPConnectSocketFactory.class HTTPConnectSocket.class HTTPSConnectSocketFactory.class HTTPSConnectSocket.class InStream.class MemInStream.class ZlibInStream.class XvpConfirmDialog.class


Now let's connect using one more magical and undocumented command line


[akamal@matrix viewer]$ java -Djavax.net.ssl.trustStore=/tmp/kimo.jks -Xmx64m -jar VncViewer.jar HOST "/console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb" PORT 443 PROXYHOST1 10.100.170.12 PROXYPORT1 443 SocketFactory "HTTPSConnectSocketFactory"
Initializing...
Connecting to /console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb, port 443...
HTTPS CONNECT via proxy 10.100.170.12 port 443
CONNECT /console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb HTTP/1.0

Connected to server
RFB server supports protocol version 3.3
Using RFB protocol version 3.3
No authentication needed
Desktop name is XenServer Virtual Terminal
Desktop size is 640 x 384
Using Tight/ZRLE encodings
Closing window
Disconnecting
Updates received: 2 (40 rectangles + 1 pseudo), 0.02 updates/sec
Rectangles: Tight=0(JPEG=0) ZRLE=0 Hextile=40 Raw=0 CopyRect=0 other=0
Pixel data: 983040 bytes, 4295 compressed, ratio 228.88
RFB socket closed


et voila, here's how it looks when it works

xcp-console


Yoohooo, hurray for undocumented ^#^&@%^#%

This post would not have been possible without great help from a friend and guru developer Ahmed Soliman. Also Abdelrahman Hussein helped with some final touches

Hoping that post is of help to some poor souls. I still like XCP though :)

Monday, November 30, 2009

Create Mega Android Playlists from PC

Wanting to create a large playlist for my HTC Hero android based phone, I found out this is a fairly painful task since the obvious method involves adding songs one by one!

Fear not fellow androiders, here is how to create mega playlists easily (although it does involve typing a command OMG :) )

1- Plug in the phone over USB. My phone appears as drive G:
2- Open a terminal window, and "cd" into the Folder containing the music you would like to make a "playlist". In my case

g:
cd MP3/80sMusic

3- Here comes the magic

dir /B > 80s-best-Music.m3u
exit

4- Et voila, the terminal window exits, safely remove your USB disk, turn off the USB connection from the phone and that is it!

Addendum for Linux users
1- Mount your phone on some directory, and "cd" to it. In my case

sudo mount /dev/sdb1 /media/androidy
cd /media/androidy

2- Create the playlist file

cd MP3/80sMusic
ls > 80s-best-Music.m3u

3- Eject and Et voila. It's really almost identical to the Windows case

cd / ; sudo umount /media/androidy


I was pleasently surprised to find that in Android ALL music apps share the same playlists. So that newly created playlist is seen by the built-in Music application, as well as with the 3rd party Music player MixZing. It is also pleasantly surprising that Android seems to scan the whole folder structure for the .M3U files wherever they may be and use them as playlists! Sweet