FOSS Boss

Oneiric server, Deploy Server fleets p2

2011-09-23T20:53:00.001+02:00

Welcome to the second installment of this article series. In the first part of this article we installed an Ubuntu server instance, made sure it became an orchestra installation server. If this is new to you, Orchestra is a new Oneiric server feature that enables admins to very easily deploy fleets of Ubuntu servers. Let's pick up where the first article stopped

First, let's check where we are. You see installing the orchestra server, it automatically downloads and imports various Ubuntu server ISOs and creates all the needed structure (distros, profiles ...etc) in the underlying cobbler system. Let's see what have we

$ sudo cobbler list
distros:
   hardy-i386
   hardy-x86_64  
   lucid-i386
   lucid-x86_64  
   maverick-i386 
   maverick-x86_64
   natty-i386
   natty-x86_64  
   oneiric-i386  
   oneiric-x86_64

profiles:
   hardy-i386
   hardy-i386-juju
   hardy-x86_64
   hardy-x86_64-juju
   lucid-i386
   lucid-i386-juju
   lucid-x86_64
   lucid-x86_64-juju
   maverick-i386
   maverick-i386-juju
   maverick-x86_64
   maverick-x86_64-juju
   natty-i386
   natty-i386-juju
   natty-x86_64
   natty-x86_64-juju
   oneiric-i386
   oneiric-i386-juju
   oneiric-x86_64
   oneiric-x86_64-juju

systems:

repos:
   hardy-i386
   hardy-i386-security
   hardy-x86_64
   hardy-x86_64-security
   lucid-i386
   lucid-i386-security
   lucid-x86_64
   lucid-x86_64-security
   maverick-i386
   maverick-i386-security
   maverick-x86_64
   maverick-x86_64-security
   natty-i386
   natty-i386-security
   natty-x86_64
   natty-x86_64-security
   oneiric-i386
   oneiric-i386-security
   oneiric-x86_64
   oneiric-x86_64-security

images:

mgmtclasses:
   orchestra-juju-acquired
   orchestra-juju-available

woah! that sure makes my life easier. If you're interested to see where the isos were downloaded (like I was) here you are

ls /var/lib/cobbler/isos/
hardy-i386-mini.iso    lucid-i386-mini.iso    maverick-i386-mini.iso    natty-i386-mini.iso    oneiric-i386-mini.iso
hardy-x86_64-mini.iso  lucid-x86_64-mini.iso  maverick-x86_64-mini.iso  natty-x86_64-mini.iso  oneiric-x86_64-mini.iso

Let's create a new virtual box VM, to serve as our new "server" that needs to be installed. Here's how it looks for me

One thing is worth noting however, it's that the NIC is placed on the "intnet" network, which has the IP range 192.168.77.0/24 that we configured in the first part of this article

now the only "real" thing you have to do, is to add a profile on the orchestra server for your new bare server. The profile binds its mac address, to a name and an installation profile (think OS to install, kickstart ..etc)

sudo cobbler system add --name="oneiric01.ubuntu.lan" --mac-address="08:00:27:B7:76:2A" --ip-address="192.168.77.33" --dns-name="oneiric01.ubuntu.lan" --hostname="oneiric01.ubuntu.lan" --profile="oneiric-x86_64-juju" --mgmt-classes="orchestra-juju-available" --kopts=" DEBCONF_DEBUG=developer netcfg/dhcp_timeout=120 netcfg/choose_interface=eth0"

Boot the server, choose PXE (For vbox that's F12 then "l" that's an L)

Watch the installer fly by (look ma hands free)

and your box is ready!

That's how easy it is to install a fresh server off your orchestra box! So basically the only thing you need to do per server, is to attach it to a profile and that's it. Boot it and it installs whatever you provisioned for it. Of course any good admin already did that manually before, but it took effort and it wasn't standardized. Now you can count on Ubuntu server covering your back when you're tasked with installing a hundred servers

How cool was that! Got thoughts, comments or rotten tomatoes ? Shoot me a comment

Oneiric server, Deploy Server fleets p1

2011-09-21T01:11:00.000+02:00

I'm gonna be posting a series of articles on new features and cool technology bits that are landing in Ubuntu Oneiric (11.10) server. Why? I like servers, I like cloud, I like Ubuntu, it all mixes well, what's not to like :)

During this first article, I'll be demoing (in a graphically intensive way :) what it takes (hint: not much!) to deploy a server fleet with Oneiric server. Orchestra is the name of a wonderful piece of technology that lands in Oneiric, that's been created on top of the open-source cobbler project. Orchestra is super easy to install and get started with, and enables you to very rapidly deploy tens or hundreds of physical servers. I'll be using virtualbox to build a small test "lab" on my laptop for purposes of this article. I did actually try KVM first, but faced some trouble getting PXE booting reliably, so I opted for virtualbox which worked flawlessly (kudos vbox guys, you rock!)

Let's get started, I created a VM to represent the very first "head node", that will install the rest of all nodes. Here is a summary of its configuration

Pop in the virtual CD, boot it, press F6, add "priority=critical locale=en_US url=http://bit.ly/uquick" (Thanks Dustin!) so it looks like

The uquick profile answers all the installer questions, such that the installation is fully automatic. Since the VM contains two NICs however, we'll need to select a primary one (eth0 in my case)

The installation runs like a champ, fully automated, give it a few minutes till it finishes everything and reboots into the server OS (oh that was easy!)

Now I configure eth1 to have a static IP address of 192.168.77.1/24 (I made any address up), here is a snapshot of /etc/network/interfaces and I started eth1 using ifup

At this stage, I rebooted the server but you definitely don't have to. Let's start actually installing Orchestra

sudo apt-get update
sudo apt-get install ubuntu-orchestra-server -y

Everything proceeds automatically, for any question you get during package installation, I'll provide a picture with the answer :)

That's it! You've just installed and configured your first Ubuntu Orchestra server, and you're now ready to install a fleet of Ubuntu servers the easy way! In part 2 of this article, I'll go through creating a second server, PXE booting and installing it from the orchestra server. (Extra credit: If you can't wait, try PXE booting a fresh server right now. Note that after installation, orchestra actually downloads and auto-imports a few Ubuntu mini ISOs, thus will need a few minutes depending on your internet connection speed)

So, what do you think of this coolness? Is this easier than the last time you tried building yourself an automated network installer? Shoot me a comment, let me know what you think

Torrent download Cloud appliance

2011-09-12T14:19:00.002+02:00

The Why

A friend of mine who's a Linux systems geek as well, was tasked with building a library of Linux distro ISOs, this involved downloading tens of ISOs many of which were only offered in torrent form. Even if that were not the case, it would still be good practice to download such large binaries from torrent to avoid loading a certain mirror too much. Anyway, we were chatting about it, and since where I live bandwidth (especially upload) is a scarce resource, he was considering paying some service to download the torrents he needed and convert them to HTTP!

I mentioned I could build something to do just that in about an hour! It wouldn't even be complex. Armed with Ensemble I can very simply launch an EC2 instance, install rtorrent (my fav cli torrent client) and rtgui (rtorrent Web UI) and have it ready to crunch on any of your torrenting needs. We both became interested in seeing how well that would work and so here we go...

The How

I'll assume you already know how to get started with Ensemble. Let's see what it takes to deploy my torrent appliance

$ bzr branch lp:~kim0/+junk/rtgui
$ ensemble bootstrap
# Wait for ec2 to catch up (2~5 mins)
$ ensemble deploy --repository . rtgui
$ ensemble expose rtgui

That is basically all you need to "use" this appliance! Give another few minutes for the rtgui appliance to boot, install and configure itself. You can check status with

$ ensemble status 
2011-09-12 13:23:53,868 INFO Connecting to environment.
machines:
  0: {dns-name: ec2-50-19-19-234.compute-1.amazonaws.com, instance-id: i-2c37ee4c}
  1: {dns-name: ec2-107-20-96-125.compute-1.amazonaws.com, instance-id: i-1c28f17c}
services:
  rtgui:
    exposed: true
    formula: local:rtgui-9
    relations: {}
    units:
      rtgui/0:
        machine: 1
        open-ports: [80/tcp, 55556/tcp, 55557/tcp, 55558/tcp, 55559/tcp, 55560/tcp,
          6881/udp]
        relations: {}
        state: started
2011-09-12 13:24:01,334 INFO 'status' command finished successfully

The import bit to watch for is "state: started", if it's something else, that means the ec2 instance is still being configured. It's nice to note that the following ports have been opened 55556-55560 since rtorrent is configured to use those ports, port 80 was opened for the Web UI, and port 6881 UDP was opened for the DHT network. I am in no way a torrent expert, so this could be completely unoptimized, but hey it seems to work

Ready to test? Machine 1 runs rtgui, so go ahead and visit it in a browser, for me that's http://ec2-107-20-96-125.compute-1.amazonaws.com/rtgui (replace that DNS name with the right one for your instance, and don't forget the trailing /rtgui like I always do). Click "Add torrent" and pass it the URL to a torrent file, I'm gonna be testing with Ubuntu 11.10 beta1 amd64 torrent file. Once the torrent is added, click the green play button to start it. Since EC2 instances have quite some bandwidth available to them, this Ubuntu torrent downloaded in a just few seconds. I am shipping a default configuration with rtorrent that limits upload speed to 100KB (since you're paying for bandwidth), but you can change that from the web UI. Here's how the whole thing looks

Once a torrent file is downloaded, you can download it through http://ec2-107-20-96-125.compute-1.amazonaws.com/complete (again replace the machine DNS name, with the correct name in your case)

A single torrent appliance is not ofcourse limited to a single torrent! You can keep adding as much as you want, however eventually you're going to hit some limit (disk IO, network IO, disk space ...etc). As such (probably only if you're after downloading really large number of torrents) you may need to "scale up" this torrent download appliance (well it's a cloud for God's sake!). If that's what you wish for, you only need to

$ ensemble add-unit rtgui

Simple as that, as with everything Ensemble! So now you know how you can download your 11.10 copy without loading Ubuntu's servers, actually you'd be helping them and all millions of Ubuntu users if you use this method on release day. Once you're done playing with the appliance, you need to destroy it (to stop paying Amazon for the machines)

$ ensemble destroy-environment
WARNING: this command will destroy the 'sample' environment (type: ec2).
This includes all machines, services, data, and other resources. Continue [y/N]y
2011-09-12 13:53:33,018 INFO Destroying environment 'sample' (type: ec2)...
2011-09-12 13:53:36,641 INFO Waiting on 2 EC2 instances to transition to terminated state, this may take a while
2011-09-12 13:54:18,617 INFO 'destroy_environment' command finished successfully

Want to improve it?

Things I wish I had time to improve:

Once a file is downloaded, upload it to S3. You can then terminate the appliance, and still download the files at your own pace
Parameterize rtorrent rc configuration file, such that you can pass it parameters from Ensemble (such as upload rate...etc)
Integrate notification upon download completion (SMS me, email me, IM me ...etc)
Add an auto-redirect to /rtgui :)
Figure out a way to download completed files from within the rtgui web UI

If you're interested in improving that appliance, drop by in #ubuntu-ensemble on IRC/Freenode and ping me (kim0) or any of the friendly folks around.

What kind of skills do you need to hack on that project? Just bash shell scripting foo! The feature I love most about Ensemble as a cloud orchestration tool, is that it doesn't twist you into using some abstracted syntax. You get to write in whatever language you feel like using, for me that's bash. You can find the script that does all of the above right here.

Interested to learn more about Ensemble and automating Ubuntu server deployments in the cloud or on physical servers ?
Want to hack on this torrent appliance, or do something similar?
Have comments or a better idea?

Let me know about it, just drop me a comment right here! You can also grab me (kim0) over Freenode irc

Battling Hunger in the Horn of Africa

2011-08-21T12:18:00.000+02:00

A humanitarian crisis has slowly unfolded in the Horn of Africa. Drought, conflict, and rising food prices have affected more than 13 million people in the region. On 20 July, famine conditions were declared in several southern regions of Somalia. The Food Security and Nutrition Analysis Unit (FSNAU) forecasts that famine conditions will spread if humanitarian assistance does not increase. In response, WFP is planning to feed over 11.5 million people, including 3.7 million people in Somalia, 3.7 million in Ethiopia, and 2.7 million in Kenya.

Restricted aid access

Access to some vital areas is restricted to humanitarian aid organizations. The hatched area on the map shows areas in which some aid organizations are unable to work— including the places where people are most in need of assistance.

Operational efficiency

The figure of USD $0.50 per person per day is based on the average combined daily costs of World Food Programme's operations within Somalia, Ethiopia, & Kenya, as well as the number of people reached by those efforts.

If you cannot see the embedded map above, click here: http://horn.wfp.org/main.html

Save a child today!

How can Govs help FOSS businesses

2011-08-21T00:19:00.000+02:00

Dear Lazyweb,

A group of FOSS ambassadors (of which I am one) have been invited to visit government officials who are "interested" in open-source. The goal will be to pitch open-source and why adopting it would have various benefits on a national level.

A more specific point that should be discussed is How can the government help local FOSS businesses grow in order to help, support and grow a FOSS-oriented eco-system (Developers, Support professionals, VARs...)

If you were in that meeting, what points would you make ?

Thanks!

Ensemble meets Hadoop on the cloud

2011-08-08T16:39:00.000+02:00

So you wanted to play with hadoop to crunch on some big-data problems, except that, well getting a hadoop cluster up and running in not exactly a one minute thing! Let me show you how to make it "a one minute thing" using Ensemble! Since Ensemble now has formulas for creating hadoop master and slave nodes, spinning up a hadoop cluster could not be easier! Check this video out

If you can't see the embedded video, here's a direct link http://youtu.be/e8IKkWJj7bA

Yep that's how simple it is! If you want to scale-out the cluster, you only need to ask Ensemble to do it for you:
$ ensemble add-unit hadoop-slave

So is this easier than configuring a hadoop cluster manually? Leave me a comment, let me know your thoughts! Also let me know what you'd like to see deployed next with Ensemble

Ubuntu takes UFOs to the cloud

2011-07-29T16:44:00.002+02:00

I've always believed in UFOs as a kid, and while I've never seen one (yet?) I am still more on the believer side! I was interested to stumble upon a database of UFO sightings at http://www.infochimps.com/tags/ufo# A shout-out at infochimps (you guys are great!). Downloading the sightings DB (around 80MB), I found a listing of 60,000 documented sightings, hmm interesting! I started thinking I could crunch on this data in some useful and fun way, what about finding the most commonly spotted UFO shape?! Sounds like I could use hadoop for that, just for the coolness factor really, the data is not that large anyway, but hey why not! I had no-idea how to get started with hadoop though and wasn't really interested in learning up all the gory details!

Well Ensemble to the rescue, hadoop master and slave formulas exist, which means someone else packaged the knowledge needed to setup and run a hadoop cluster for me. All I needed to do was ask Ensemble to deploy me a couple of cloud instances and start playing. Let's see how you can do that for yourself

I won't repeat the instructions to get started with Ensemble, since the documentation is a good place for that (and it's so easy anyway!). If you feel you need more help there, this little video should be helpful. If you're still stuck, you can always drop by on irc/freenode at #ubuntu-ensemble and ask your questions

Hadoop node, with an extra slave please

So, let's start ensembling

bzr branch lp:~negronjl/+junk/hadoop-master
bzr branch lp:~negronjl/+junk/hadoop-slave
ensemble bootstrap

wait a minute or two for EC2 to spin up the instance, then

ensemble status

which’ll give you output like

$ ensemble status
2011-07-12 15:20:54,978 INFO Connecting to environment.
The authenticity of host 'ec2-50-17-28-19.compute-1.amazonaws.com (50.17.28.19)' can't be established.
RSA key fingerprint is c5:21:62:f0:ac:bd:9c:0f:99:59:12:ec:4d:41:48:c8.
Are you sure you want to continue connecting (yes/no)? yes
machines:
  0: {dns-name: ec2-50-17-28-19.compute-1.amazonaws.com, instance-id: i-8bc034ea}
services: {}
2011-07-12 15:21:01,205 INFO 'status' command finished successfully

Now let's deploy a two node hadoop cluster

ensemble deploy --repository . hadoop-master
ensemble deploy --repository . hadoop-slave
ensemble add-relation hadoop-master hadoop-slave

Yeah it's that easy! Ensemble formulas manage all the kung-fu for you. The hadoop cluster is ready, let's ssh into the master node and switch to user hdfs

ensemble ssh hadoop-master/0
sudo -su hdfs

Downloading UFOs

Download the infochimps sightings database here, unzip it and locate the TSV file (tab separated values) file. Note that you can download the file from infochimps without registering on their website (didn't I say these guys were great :)

Upload the TSV DB to hadoop's distributed filesystem

hadoop dfs -copyFromLocal ufo_awesome.tsv ufo_awesome.tsv

Almost ready, the corpus has been uploaded. Now we need to write some map/reduce jobs to do the actual crunching. Not being a pro developer, the thought of writing that in java was like (oh no ew), so python to the rescue! Thanks to the great instructions at Michael Noll's blog, I was able to massage some of that code to get it to do what I wanted. I pushed my code to launchpad, so that you can grab it directly from the hadoop master node

cd /tmp
bzr branch lp:~kim0/+junk/ufo-ensemble-cruncher
cd ufo-ensemble-cruncher

Unleashing the elephant

Now for the big moment, let's launch the elephant

hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-*.jar -file ./mapper.py -mapper mapper.py -file ./reducer.py -reducer reducer.py -input ufo_awesome.tsv -output ufo-output
packageJobJar: [./mapper.py, ./reducer.py, /tmp/hadoop-hdfs/hadoop-unjar1418682529553378062/] [] /tmp/streamjob5701745574334998473.jar tmpDir=null
11/07/29 12:27:52 INFO mapred.FileInputFormat: Total input paths to process : 1
11/07/29 12:27:53 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-hdfs/mapred/local]
11/07/29 12:27:53 INFO streaming.StreamJob: Running job: job_201107290935_0010
11/07/29 12:27:53 INFO streaming.StreamJob: To kill this job, run:
11/07/29 12:27:53 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=domU-12-31-39-10-81-8E.compute-1.internal:8021 -kill job_201107290935_0010
11/07/29 12:27:53 INFO streaming.StreamJob: Tracking URL: http://domU-12-31-39-10-81-8E.compute-1.internal:50030/jobdetails.jsp?jobid=job_201107290935_0010
11/07/29 12:27:54 INFO streaming.StreamJob:  map 0%  reduce 0%
11/07/29 12:28:11 INFO streaming.StreamJob:  map 10%  reduce 0%
11/07/29 12:28:12 INFO streaming.StreamJob:  map 19%  reduce 0%
11/07/29 12:28:14 INFO streaming.StreamJob:  map 72%  reduce 0%
11/07/29 12:28:16 INFO streaming.StreamJob:  map 100%  reduce 0%
11/07/29 12:28:33 INFO streaming.StreamJob:  map 100%  reduce 100%
11/07/29 12:28:37 INFO streaming.StreamJob: Job complete: job_201107290935_0010
11/07/29 12:28:37 INFO streaming.StreamJob: Output: ufo-output

Woohoo success! Now let's grab the results, sorting it to easily see the most popular sighting shape

Is the answer really 42

hadoop dfs -cat ufo-output/part-00000 | sort -k 2,2 -nr
light   12202
triangle        6082
circle  5271
disk    4825
other   4593
unknown 4490
sphere  3637
fireball        3452
oval    2869
formation       1788
cigar   1782
changing        1546
flash   990
cylinder        982
rectangle       966
diamond 915
chevron 760
egg     664
teardrop        595
cone    265
cross   177
delta   8
round   2
crescent        2
pyramid 1
hexagon 1
flare   1
dome    1
changed 1

The answer is "light" then! Wow that was a blast! I had fun doing this exercise. Now I am no hadoop expert in any way (so direct those hadoopy questions to someone who can actually answer them), however I was quite pleased Ensemble could help me get up and running that fast. The Ensemble community is doing a great job wrapping many free software with formulas, such that you can always get up and running with any app you need in seconds rather than days (months?). You too can write Ensemble formulas for your favorite (server?) application. Hop on to #ubuntu-ensemble and grab me (kim0) or any of the dev team and ask any questions on your mind! We're a happy community

So was that fun? Can you think of something cooler you want to see done? Leave me a comment, let me know about it

Ensemble deploy and scale cloud apps

2011-06-28T12:40:00.000+02:00

Think deploying and scaling your cloud application is hard ? Think again!

Check out this video demo, where I deploy a multi-tiered cloud application. I'm deploying

HAproxy load balancer
Memcached caching server
MediaWiki application server
MySQL DB

connecting them together and getting it working. After which, I'm scaling the whole thing from two application servers, to four

How long it takes me ? Well basically 5 minutes for the whole thing!

The secret: Ensemble !

Here's the video

Here is a direct link if you can't see the player

So what do you guys think, like the demo ? Leave me a comment, let me know which demo you'd like to see next. Also if you'd like to see your favorite application deployed with Ensemble, leave me a comment or ping me (kim0) on freenode irc

Ensemble user tutorial p2

2011-06-23T18:58:00.002+02:00

Welcome to the second part of this Ensemble user tutorial. In part 1 we bootstrapped an Ensemble environment, deployed a sample wordpress service and a MySQL service. Related the two services together and got ourselves a working wordpress installation. In this second part, let's check out viewing the debug-log output to understand the asynchronous nature of hook execution. You'll see how easy it is to "scale-up" a service deployment

Tracing hook execution
An Ensemble user should never have to trace the execution order of hooks, however if you are the kind of person who enjoys looking under the hood, this section is for you. Understanding hook order execution, the parallel nature of hook execution across instances, and how relation-set in a hook can trigger the execution of another hook is quite interesting and provides insight into Ensemble internals

Here are a few important messages from the debug-log of this Ensemble run. The date field has been deliberately left in this log, in order to understand the parallel nature of hook execution.

Things to consider while reading the log include:

The time the log message was generated
Which service unit is causing the log message (for example mysql/0)
The message logging level. In this run DEBUG messages are generated by the Ensemble core engine, while WARNING messages are generated by calling ensemble-log from inside formulas (which you can read in the examples folder)

Let’s view select debug-log messages which can help understand the execution order:

14:29:43,625 unit:mysql/0: hook.scheduler DEBUG: executing hook for wordpress/0:joined
14:29:43,626 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
14:29:43,660 unit:wordpress/0: hook.scheduler DEBUG: executing hook for mysql/0:joined
14:29:43,660 unit:wordpress/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
14:29:43,661 unit:wordpress/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:29:43,789 unit:mysql/0: unit.hook.api WARNING: Creating new database and corresponding security settings
14:29:43,813 unit:wordpress/0: unit.hook.api WARNING: Retrieved hostname: ec2-184-72-156-54.compute-1.amazonaws.com
14:29:43,976 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:29:43,997 unit:wordpress/0: hook.scheduler DEBUG: executing hook for mysql/0:modified
14:29:43,997 unit:wordpress/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:29:44,143 unit:wordpress/0: unit.hook.api WARNING: Retrieved hostname: ec2-184-72-156-54.compute-1.amazonaws.com
14:29:44,849 unit:wordpress/0: unit.hook.api WARNING: Creating appropriate upload paths and directories
14:29:44,992 unit:wordpress/0: unit.hook.api WARNING: Writing wordpress config file /etc/wordpress/config-ec2-184-72-156-54.compute-1.amazonaws.com.php
14:29:45,130 unit:wordpress/0: unit.hook.api WARNING: Writing apache config file /etc/apache2/sites-available/ec2-184-72-156-54.compute-1.amazonaws.com
14:29:45,301 unit:wordpress/0: unit.hook.api WARNING: Enabling apache modules: rewrite, vhost_alias
14:29:45,512 unit:wordpress/0: unit.hook.api WARNING: Enabling apache site: ec2-184-72-156-54.compute-1.amazonaws.com
14:29:45,688 unit:wordpress/0: unit.hook.api WARNING: Restarting apache2 service

Scaling the ensemble
Assuming your blog got really popular, is having high load and you decided to scale it up (it’s a cloud deployment after all). Ensemble makes this magically easy. All what is needed is:

$ bin/ensemble add-unit wordpress
INFO Connecting to environment.
INFO Unit 'wordpress/1' added to service 'wordpress'
INFO 'add_unit' command finished successfully
$ bin/ensemble status
machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
  2: {dns-name: ec2-184-72-156-54.compute-1.amazonaws.com, instance-id: i-9c7e29f3}
  3: {dns-name: ec2-50-16-156-106.compute-1.amazonaws.com, instance-id: i-ba6532d5}
services:
  mysql:
    formula: local:mysql-11
    relations: {db: wordpress}
    units:
      mysql/0:
        machine: 1
        relations:
          db: {state: up}
        state: started
  wordpress:
    formula: local:wordpress-29
    relations: {db: mysql}
    units:
      wordpress/0:
        machine: 2
        relations:
          db: {state: up}
        state: started
      wordpress/1:
        machine: 3
        relations:
          db: {state: up}
        state: started

The add-unit command starts a new wordpress instance (wordpress/1), which then joins the relation with the already existing mysql/0 instance. mysql/0 notices the database required has already been created and thus decides all needed configuration has already been done. On the other hand wordpress/1 reads service settings from mysql/0 and starts configuring itself and joining the ensemble. Let’s review a short version of debug-log for adding wordpress/1:

14:36:19,755 unit:mysql/0: hook.scheduler DEBUG: executing hook for wordpress/1:joined
14:36:19,755 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
14:36:19,810 unit:wordpress/1: hook.scheduler DEBUG: executing hook for mysql/0:joined
14:36:19,811 unit:wordpress/1: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
14:36:19,811 unit:wordpress/1: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:36:19,918 unit:mysql/0: unit.hook.api WARNING: Database already exists, exiting
14:36:19,938 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:36:19,990 unit:wordpress/1: unit.hook.api WARNING: Retrieved hostname: ec2-50-16-156-106.compute-1.amazonaws.com
14:36:20,757 unit:wordpress/1: unit.hook.api WARNING: Creating appropriate upload paths and directories
14:36:20,916 unit:wordpress/1: unit.hook.api WARNING: Writing wordpress config file /etc/wordpress/config-ec2-50-16-156-106.compute-1.amazonaws.com.php
14:36:21,088 unit:wordpress/1: unit.hook.api WARNING: Writing apache config file /etc/apache2/sites-available/ec2-50-16-156-106.compute-1.amazonaws.com
14:36:21,236 unit:wordpress/1: unit.hook.api WARNING: Enabling apache modules: rewrite, vhost_alias
14:36:21,476 unit:wordpress/1: unit.hook.api WARNING: Enabling apache site: ec2-50-16-156-106.compute-1.amazonaws.com
14:36:21,682 unit:wordpress/1: unit.hook.api WARNING: Restarting apache2 service

Destroying the environment
Once you are done with an Ensemble deployment, you need to terminate all running instances in order to stop paying for them. The shutdown command helps terminate all running instances:

$ bin/ensemble shutdown

Ensemble will ask for user confirmation of shutdown before proceeding as this will destroy service data in the environment as well.

Hope you enjoyed the article! If you find Ensemble interesting, please do visit us in #ubuntu-ensemble. We're a friendly bunch :) Do you want to write Ensemble formulas? Want to get the satisfaction of Ensemblizing your favorite application, grab me (kim0) on irc, and I will help you do it

Ensemble user tutorial p1

2011-06-16T14:35:00.000+02:00

I've been adding lots of documentation to help a new Ensemble user find her way around Ensemble. I thought it would nice to share this documentation as a series of blog posts, to raise its exposure and help interested users find their way quickly. Here is the first of a series of posts, which I hope you'll enjoy. If you think we can improve the docs in any way, please do let me know

Introduction

This tutorial demonstrates basic features of Ensemble from a user perspective. An Ensemble user would typically be a devops or a sys-admin who is interested in automated deployment and management of servers and services.

Bootstrapping

The first step for deploying an Ensemble system is to perform bootstrapping. Bootstrapping launches a utility instance that is used in all subsequent operations to launch and orchestrate other instances:

$ bin/ensemble bootstrap

Note that while the command should display a message indicating it has finished successfully, that does not mean the bootstrapping instance is immediately ready for usage. Bootstrapping an instance can require a couple of minutes. To check on the status of the Ensemble deployment, we can use the status command:

$ bin/ensemble status

If the bootstrapping node has not yet completed bootstrapping, the status command may either mention the environment is not yet ready, or may display a connection timeout such as:

INFO Connecting to environment.
ERROR Connection refused
ProviderError: Interaction with machine provider failed:
ConnectionTimeoutException('could not connect before timeout after 2
retries',)
ERROR ProviderError: Interaction with machine
provider failed: ConnectionTimeoutException('could not connect before timeout
after 2 retries',)

This is simply an indication the environment needs more time to complete initialization. It is recommended you retry every minute. Once the environment has properly initialized, the status command should display:

machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  services: {}

Note the following, machine “0” has been started. This is the bootstrapping node and the first node to be started. The dns-name for the node is printed. Also the EC2 instance-id is printed. Since no services are yet deployed to the Ensemble system yet, the list of deployed services is empty

Starting debug-log
While not a requirement, it is beneficial for the understanding of Ensemble to start a debug-log session. Ensemble’s debug-log provides great insight into the execution of various hooks as they are triggered by various events. It is important to understand that debug-log shows events from a distributed environment (multiple-instances). This means that log lines will alternate between output from different instances. To start a debug-log session, from a secondary terminal issue:

$ bin/ensemble debug-log
INFO Connecting to environment.
INFO Enabling distributed debug log.
INFO Tailing logs - Ctrl-C to stop.

This will connect to the environment, and start tailing logs.

Deploying service units
Now that we have bootstrapped the Ensemble environment, and started the debug-log viewer, let’s proceed by deploying a mysql service:

$ bin/ensemble deploy --repository=examples mysql
INFO Connecting to environment.
INFO Formula deployed as service: 'mysql'
INFO 'deploy' command finished successfully

Checking the debug-log window, we can see the mysql service unit being downloaded and started:

Machine:1: ensemble.agents.machine DEBUG: Downloading formula
local:mysql-11...
Machine:1: ensemble.agents.machine INFO: Started service unit mysql/0

It is important to note the different debug levels. DEBUG is used for very detailed logging messages, usually you should not care about reading such messages unless you are trying to debug (hence the name) a specific problem. INFO debugging level is used for slightly more important informational messages. In this case, these messages are generated as the mysql formula’s hooks are being executed. Let’s check the current status:

$ bin/ensemble status
machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
services:
  mysql:
    formula: local:mysql-11
    relations: {}
    units:
      mysql/0:
        machine: 1
        relations: {}
        state: null

We can see a new EC2 instance has now been spun up for mysql. Information for this instance is displayed as machine number 1 and mysql is now listed under services. It is apparent the mysql service unit has no relations, since it has not been connected to wordpress yet. Since this is the first mysql service unit, it is being referred to as mysql/0, subsequent service units would be named mysql/1 and so on.

Note

An important distinction to make is the difference between a service and a service unit. A service is a high level concept relating to an end-user visible service such as mysql. The mysql service would be composed of several mysql service units referred to as mysql/0, mysql/1 and so on.

The mysql service state is listed as null since it’s not ready yet. Downloading, installing, configuring and starting mysql can take some time. However we don’t have to wait for it to configure, let’s proceed deploying wordpress:

$ bin/ensemble deploy --repository=examples wordpress

Let’s wait for a minute for all services to complete their configuration cycle and get properly started, then issue a status command:

$ bin/ensemble status
machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
  2: {dns-name: ec2-184-72-156-54.compute-1.amazonaws.com, instance-id: i-9c7e29f3}
services:
  mysql:
    formula: local:mysql-11
    relations: {}
    units:
      mysql/0:
        machine: 1
        relations: {}
        state: started
  wordpress:
    formula: local:wordpress-29
    relations: {}
    units:
      wordpress/0:
        machine: 2
        relations: {}
        state: started

mysql/0 as well as wordpress/0 are both now in the started state. Checking the debug-log would reveal wordpress has been started as well

Adding a relation
While mysql and wordpress service units have been started, they are still isolated from each other. An important concept for Ensemble is connecting various service units together to create a bigger ensemble! Adding a relation between service units causes hooks to trigger, in effect causing all service units to collaborate and work together to reach the desired end state. Adding a relation is extremely simple:

$ bin/ensemble add-relation wordpress mysql
INFO Connecting to environment.
INFO Added mysql relation to all service units.
INFO 'add_relation' command finished successfully

Checking the Ensemble status we see that the db relation now exists with state up:

$ bin/ensemble status
machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
  2: {dns-name: ec2-184-72-156-54.compute-1.amazonaws.com, instance-id: i-9c7e29f3}
services:
  mysql:
    formula: local:mysql-11
    relations: {db: wordpress}
    units:
      mysql/0:
        machine: 1
        relations:
          db: {state: up}
        state: started
  wordpress:
    formula: local:wordpress-29
    relations: {db: mysql}
    units:
      wordpress/0:
        machine: 2
        relations:
          db: {state: up}
        state: started

You can now point your browser at the public dns-name for instance 2 (running wordpress) to view the wordpress blog

Patience is a virtue, you did read the full thing after all! I would really appreciate some feedback, what you liked, or did not like. And how we can improve the docs. Drop me a comment, shoot what's on your mind

Ensemble IRC meeting today

2011-06-08T13:30:00.000+02:00

When? Today 6pm-UTC
What? Ensemble cloud community meeting
For? Ensemble simplifies the deployment, management, and scaling of services in the cloud. We've answered the what is Ensemble question previously, but there's nothing like one to one discussions! Interested to learn more? ask your questions ? connect to the development team ? Start writing formulas to deploy your software or even start hacking on Ensemble core? Then this meeting is perfect for you. Join us on IRC in #ubuntu-cloud today 6pm-UTC

Ensemble at UDS-O

2011-05-20T18:16:00.001+02:00

UDS-O is now over, I had a chance to meet with the Ensemble team (a bunch of awesome engineers), also had a chance to attend or lead a few sessions concerning future directions of Ensemble. I'll try to summarize UDS outcome wrt Ensemble from a project newcomer perspective

Ensemble is now able to do a multi-machine deployment and orchestration
It can do Dynamic reconfiguration which means passing parameters to running formulas to adjust behavior. More work needs to land here, but the foundations are there
Does formula upgrades, more bits landing soon
Firewall auto-configuration (expose, unexpose services). Again, still more bits to land soonish
It has a ppa and docs from trunk live at https://ensemble.ubuntu.com/docs/

The focus for the 11.10 Oneiric cycle is going to be stability and polish. While currently Amazon EC2 is the only supported deployment target, the 11.10 cycle should hopefully see more targets added such as Linux Containers (LXC) for local development and testing of Ensemble formulas. Having LXC support in 11.10 is a bit optimistic, so if you can lend a helping hand, please do! Also Eucalyptus cloud support is the works (isn't this just great!)

During 11.10 as well, the infrastructure for Ensemble as a project should be improved. This involves adding better and more structured content to the Ensemble website. Adding more documentation, guides and screencasts. While you can start writing and contributing Ensemble formulas today! (ping me if you're interested) More work will go into refining the process and integrating it into the Ensemble command line tools.

That's mostly it. It is an incredibly interesting time for cloud technologies! Diving into discussions with the Ensemble team about the vision and decisions they have taken, I was blown away. I firmly believe Ensemble is a game changer with respect to rapid provisioning and orchestration. Right now, is such a great time to get involved with Ensemble. If you're interested in joining the Ensemble project, or in learning more about it in any way, please do leave me a comment here, or ping me on irc (kim0)

Cloud Portal one-click launch

2011-05-11T11:33:00.000+02:00

Announcing two nice little additions to the cloud portal AMI tool that should make everyone's cloud life a little easier

Amazon now allows passing parameters to the AWS console to basically be able to choose the region and AMI-ID to launch. This is now integrated in the cloud portal so that you search for the ami, click the link and it's pre-selected, ready to be launched in AWS console. Here's a screenshot

As that image shows as well, you can search now for "cluster" to get Ubuntu cloud cluster-ready images!

Interested to add any features? Have any thoughts or comments, let me know, leave me a word

Help Squash Natty Server Bugs

2011-04-18T14:14:00.000+02:00

Mirroring a post from Ubuntu Server developer Dave Walker, If you're capable and willing to help squash some server bugs, your effort and contribution is really needed now! Join us on irc #ubuntu-server, ask around if you need help. Thanks in advance

Here is a list of current bugs that should hopefully be fixed for Natty release. The bugs with "Not assigned to anyone :(", could really do with someone volunteering to take. Please feel free to assign yourself, if you are willing to take on working towards a fix.

If you are an assignee, please make sure that the current status (and comments) accurately represent the status before tomorrows server team meeting.

== Bugs for the server team ==

[655533] [likewise-open] [master] package likewise-open 5.4.0.42111-2ubuntu2 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1 (http://pad.lv/655533)

* Bug Status: Confirmed

* Not assigned to anyone :(

* Last updated: 2011-04-18

[726769] [eucalyptus] package eucalyptus-common 2.0.1 bzr1255-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1 (http://pad.lv/726769)

* Bug Status: Confirmed

* Assigned to: Dave Walker

* Last updated: 2011-04-18

[661453] [dovecot] dovecot.conf always shows as having been locally modified on update (http://pad.lv/661453)

* Bug Status: Confirmed

* Assigned to: Clint Byrum

* Last updated: 2011-04-18

[745946] [cloud-init] cloud-final job did not run in ec2-automated-tests (http://pad.lv/745946)

* Bug Status: Confirmed

* Not assigned to anyone :(

* Last updated: 2011-04-18

[759943] [mod-wsgi] mod_wsgi.so-3.2 gives errors (http://pad.lv/759943)

* Bug Status: Confirmed

* Not assigned to anyone :(

* Last updated: 2011-04-15

[764391] [cobbler] cobbler fails to manage bind9 (http://pad.lv/764391)

* Bug Status: Confirmed

* Assigned to: James Page

* Last updated: 2011-04-18

[745930] [cloud-init] cloud-init timeout waiting for metadata service on EC2 (http://pad.lv/745930)

* Bug Status: New

* Not assigned to anyone :(

* Last updated: 2011-04-15

[760288] [Ubuntu] JeOS is oversized (http://pad.lv/760288)

* Bug Status: Confirmed

* Not assigned to anyone :(

* Last updated: 2011-04-15

== Bugs being worked on in other areas, more for information ==

[728088] [debian-installer] iscsi root with or without auth fails to boot (http://pad.lv/728088)

* Bug Status: Confirmed

* Assigned to: Colin Watson

* Last updated: 2011-04-14

[747090] [linux] wrong return address sometimes pushed for INT in kvm (not qemu) (http://pad.lv/747090)

* Bug Status: Fix Committed

* Assigned to: Andy Whitcroft

* Last updated: 2011-04-18

[566818] [plymouth] Cryptsetup passphrase prompt during boot: every character typed repeats the prompt (http://pad.lv/566818)

* Bug Status: Confirmed

* Assigned to: Surbhi Palande

* Last updated: 2011-04-16

[580319] [upstart] dhcp3-server launches before upstart brings all interface, thus failing to start (http://pad.lv/580319)

* Bug Status: Triaged

* Assigned to: James Hunt

* Last updated: 2011-04-18

[759545] [grub2] user prompted to update unmodified grub configuration during Ubuntu server upgrade (http://pad.lv/759545)

* Bug Status: New

* Last updated: 2011-04-18

Cloud Portal Intro

2011-04-15T19:25:00.000+02:00

Just written a nice introduction to Ubuntu cloud portal on Ubuntu News. If you don't know about that, or are new to the Ubuntu cloud scene, check it out and ping me back for any questions or comments

Ensemble Cloud Community Meeting Summary

2011-04-14T19:12:00.000+02:00

Yesterday was the first Ubuntu Cloud community meeting to include Ensemble. It was such a blast. Personally I find the ensemble project to be incredibly cool, it's such a paradigm shift into how services (rather than servers) should be managed especially in this cloud age. Here's a quick summary of the main points mentioned during yesterday's meeting

The ensemble-team just landed some changes to formula authorship, specifically the relation-changed-hook was broken out into three separate hooks the relation-joined, relation-changed, relation-departed

joined denotes that a unit of the related service came online
changed denotes the unit of a related service changed its settings
departed denotes the unit of a related service went offline

This hook split is helpful to avoid subtle race conditions, and make formula authorship easier
Ensemble team is now working on formula upgrades

For now the team is only tackling an initial step of upgrading just the formula itself and only if it's already running
Example command would be: ensemble upgrade-formula --repository=examples mydbservice
That will look into the repo directory for a formula matching the formula of myblog, verify its a newer revision, upload and it mark the units for upgrade
On the unit side, they'll detect they need upgrades, download the formula, and most importantly, execute the upgrade-hook from the new formula
A suggestion came up, that the upgrade hook would be informed which version the upgrade process is starting from

A Question comes up, whether hooks should be implemented in a specific language. The answer is that all hooks are effectively just executables, those executables can be in any language and use any tools they'd like
Ensemble team is also working on orchestrating around firewall addressing such that the firewall is configurable by the ensemble user (ensemble expose, ensemble unexpose), with corresponding hooks (exposed, unexposed), and hook commands (open-port, close-port)

The draft for that is on http://people.canonical.com/~niemeyer/ensemble/drafts/expose-services.html
expose and unexpose are for now operating on top of ec2 security-groups, in the future however multiple backends can be implemented (cisco firewalls..etc)! Without even needing to change the formula, quite impressive!

A question was asked whether ensemble is ready for public collaboration, the answer to which is a definite yes!

Grab the code today: bzr branch lp:ensemble
Join the mailing list: https://lists.ubuntu.com/mailman/listinfo/Ensemble
Join IRC channels #ubuntu-cloud, #ubuntu-ensemble

If you liked this, be sure to join the next meeting (Every Wed, 18:00-UTC in #ubuntu-cloud). Have questions or comment? Leave me a comment right here, or ping me (kim0) on IRC

Ensemble Cloud Community Meeting

2011-04-12T18:38:00.000+02:00

Ensemble is an exciting new cloud technology innovated by Ubuntu. If you’re curious what it does (I bet you are) it is a service orchestration framework with focus on rapid cloud deployment. That is to say, it allows one to rapidly deploy cloud instances, bootstrap them, install software on them, configure them as well as link them all together to provide you (as a sys-admin or DevOp) with an end-user facing high-level “service” (Load Balanced CMS with sharded Database, Multinode email service installation…etc). Ensemble has been compared to being APT for the cloud! Now does that sound interesting, it sure is if you ask me! Welcome to next generation Linux infrastructure automation.

If that got you interested, be sure to attend the weekly IRC meeting on Wed 18:00-UTC on #ubuntu-cloud.

Feel free to tweet/dent and spread this around the globe. Got feedback? Leave me comments on cloud.ubuntu.com or on my blog

Help Shape Maps.Ubuntu.com

2011-04-11T14:31:00.000+02:00

About either months ago, I had started on a fun little project to raise awareness of how widely used Ubuntu Server is around the world. The result was http://maps.ubuntu.com/ a web tool allowing anyone using Ubuntu Server to "mark" his location on a global map. This was launched with celebrations for 10.04.1, and thousands of cities were marked (woohoo). The project code was open-sourced and lives at https://launchpad.net/mapuntu/trunk

It's been a while since the project was last updated, a few days ago, an Ubuntu community member ( hey Ronnie ) stepped up to re-shape the mapuntu project and relaunch it into an inspiring map that helps anyone record and locate Ubuntu events around the globe! The first part of that effort is creating a proper back-end API, everyone is encouraged to click that link and leave comments. If you are inspired by that project, and would like to help, please get involved! Things you can do to help include

Review and enhance the proposed API
Help write python code to implement the backend api
Write html/css/js for the front-end
Play with the code, test it, report bugs, adapt to your own website!

If you want to talk more about this, Please join #ubuntu-locoteams on IRC and ping Ronnie or myself (kim0, also on #ubuntu-cloud).

Creating Customized UEC and EC2 Images

2011-04-01T18:35:00.000+02:00

Want to create a customized Ubuntu cloud image that you can deploy to EC2 or UEC ? No problem! Check out this 5 min video where I demo everything needed to download, customize, rebundle and publish the new image to EC2! Isn't this just great :)

If you can't see the embedded player, here's a direct link: http://www.youtube.com/watch?v=K1IwTGLWbu0

Ubuntu Cloud Days, Day-2 Summary

2011-03-25T15:03:00.000+02:00

Yesterday was the second and final day of cloud days. It was such a blast! Very interesting stuff, and very cool community to be part of! In case you missed it, you can still read up the logs, and you can reach me and others for questions and comments on #ubuntu-cloud .. Let's quickly zip through yesterday's sessions

16.00 UTC smoser gave a very good introduction to the different ways with which you can rebundle UEC or EC2 images for Ubuntu, how you can customize them and the different pros and cons. Scott also answered lots of questions
17.00 UTC tetet started by presenting some interesting new features and tricks that can be used with UEC namely persistency, which means kexec'ing the kernel to load from an EBS volume resulting in persistent UEC instances. Thorsten zipped through the session quickly, but as a pro instructor, he was very well prepared with links to graphics, links and documentation
18.00 UTC Daviey wowed the crowd with a puppet introductory session, he showed how to configure apache with puppet and what it takes and deploy a basic system to get running fast. It was a great session
19.00 UTC edulix took over to present an awesome session about using hadoop and its map/reduce programming model to crunch on large (we're talking terabytes) of data on lots of machines. Very interesting and useful session
20.00 UTC obino had the final say on cloud days. The session was about UEC/Eucalyptus. He presented a detailed architecture of the system, what each node does, how things work. He also explained how to get started contributing to the eucalyptus community.

It was a great event. Big Thank-Yous to everyone who presented in UCD. Till next time o/

Day-2 Ubuntu Cloud Days

2011-03-24T15:23:00.000+02:00

Yesterday was great! It was Ubuntu Cloud Days, the event was packed with very interesting sessions, hundreds of attendees, lots of questions got answered, tons of ideas and comments were made. I personally had lots of fun, and was literally stuck to my keyboard the full time of the event

Let's zip through what happened yesterday (If you missed it, Click the time to read the logs!)
16.00 UTC kim0 (me) took-over, gave a quick introduction about cloud computing, differences between public and private clouds and what the cloud fuss is all about
17.00 UTC semiosis started a great session about building scalable distributed storage on the EC2 Cloud using Glusterfs for improving redundancy, and performance
18.00 UTC SpamapS gave a cool presentation and hands-on demo of a new Ubuntu ground breaking "service" management framework called Ensemble. While in early development phase, it's already working well and is changing how we look at server management
19.00 UTC hallyn took over presenting a technically deep discussion and hands-on demo about LXC Linux containers technology. This is a light weight Linux virtualization (or container) technology
20.00 UTC soren took over to talk about OpenStack, one of the cloud stacks that Ubuntu ships with 11.04 that's been getting lots of cloud people excited lately. Soren explained how the whole thing got started, explained the architecture, answered lots of questions and discussed what we should be seeing in the future

Here's a sneak peak of today's sessions

Thu 24th Mar
16.00 UTC rebundling/re-using Ubuntu’s UEC images — smoser
17.00 UTC UEC persistency — tetet
18.00 UTC Puppet Configuration Management — Daviey
19.00 UTC Using hadoop, divide and conquer — edulix
20.00 UTC UEC/Eucalyptus Private Cloud — obino

Let's get crunching on even more cool sessions today! If you’re new to IRC, you can simply use this web page to join. For more information, check out the wiki page, also feel free to ping “kim0″ on IRC!

Today is Ubuntu Cloud Days

2011-03-23T13:33:00.000+02:00

Today is .. dum roll ... Ubuntu Cloud Days! The event starts at 4pm UTC on irc #ubuntu-classroom. Updates to the schedule include an OpenStack session, a Eucalyptus session and a puppet session! Those attending the Ensemble session are in for a treat :) During Cloud Days, you’ll learn lots of exciting stuff, you’ll interact with tons of smart people, and you’ll just have fun! Spread the word, join in and bring your friends!

Here’s the sessions schedule

Wed 23rd Mar
16.00 UTC Cloud Computing 101, Ask your questions — kim0
17.00 UTC Scaling shared-storage web apps in the cloud with Ubuntu & GlusterFS — semiosis
18.00 UTC What is Ensemble? – Presentation and Demo — SpamapS
19.00 UTC Using Linux Containers in Natty — hallyn
20.00 UTC Open-Stack Introduction — soren

All session happen in IRC (Freenode) in #ubuntu-classroom. If you’re new to IRC, you can simply use this web page to join. For more information, check out the wiki page, also feel free to ping “kim0″ on IRC

Additional links:

Ubuntu Cloud Days approaching

2011-03-21T14:10:00.000+02:00

Just a reminder, Ubuntu Cloud Days is approaching, it's only two days away! Yes it's this Wed/Thu. I just posted the full details on the Ubuntu Cloud Portal. Check it out, and leave me a comment if you have any questions

Cloud Python hackers wanted

2011-03-09T14:57:00.000+02:00

A while back I had blogged (yes the one with the big uncle Sam pic) asking for people interested in working on a python tool to help migrate ebs AMIs across ec2 regions. Since then the project has progressed a bit:

- Project page is now: https://launchpad.net/ec2-migrate-ebs-ami
- Project is now able to launch two utility instances in the two specified regions, and mount the source volume in the source instance
- What remains is attaching a destination volume, and sync'ing over contents. More details can be found on the blueprint: https://blueprints.launchpad.net/ubuntu/+spec/cloud-server-n-ec2-migrate-region

Development pace has now slowed down significantly, as such I am renewing the call. This is a great way to start getting involved with cloud development, and to join the Ubuntu Cloud and server communities! joining in is "easy", you don't even have to know lots of python to start hacking! The steps are very well defined, and I am happy to help with all steps along the way. If you are thinking about it, go ahead and get in touch with me right now (kim0 on irc, kim0 AT ubuntu.com) or join today's irc meeting (6pm-UTC #ubuntu-cloud https://wiki.ubuntu.com/UbuntuCloudMeeting) and start discussing how you can get involved

Two weeks to Ubuntu Cloud Days

2011-03-09T12:36:00.000+02:00

Two weeks from now, Ubuntu Cloud Days begin! The very first dedicated Ubuntu Cloud Days event from March 23^rd to March 24^th 2011!

Cloud computing is changing the face of IT and Ubuntu Server is arguably the most popular Cloud based Operating System with millions of instances launched a year! Are you interested to know more about using Ubuntu Server on the Cloud ? Find out from March 23^rd to March 24^th 2011

Ubuntu Cloud Days is a series of online sessions where you can learn more about:

Answering your questions about Ubuntu and the Cloud
Ubuntu Enterprise Cloud Images, using Ubuntu with Amazon EC2 cloud
Building your private cloud over Ubuntu Server platform
New virtualization and container technologies in Ubuntu Server
Crunching Big-Data on the cloud with hadoop
Scaling your web-apps on the cloud with Ubuntu
much more...

Read more about UCD at https://wiki.ubuntu.com/UbuntuCloudDays

If you are interested to deliver a session, please get in contact with me! While the timetable is now "full", there's no reason why we can't extend an extra session (or day!). Ping me (kim0 AT ubuntu.com) or (kim0 on IRC)

Mark your calendar, Join in and Bring your friends ;)

Additional links:

FAST Win7 KVM VirtIO{Disk-Net} Install

2011-03-02T18:44:00.000+02:00

Everyone who has to use Windows, please let's at least keep it contained inside a virtual machine! In this article I'll demo how to "Install Windows7 over Ubuntu 11.04 Natty, using KVM with System Disk over VirtIO". Quoting the libvirt wiki "Virtio is a Linux standard for network and disk device drivers where just the guest's device driver "knows" it is running in a virtual environment, and cooperates with the hypervisor. This enables guests to get high performance network and disk operations, and gives most of the performance benefits of paravirtualization"

Here's a little convenience script to download virtio Windows drivers and launch virt-install to Install Windows. Copy/Paste this script into an empty folder, chmod +x it, and run it.

#!/bin/sh
WINISO=/path/to/win7.iso    #Windows ISO
INSTALLDISK=win7virtio.img  #Disk location. Can be LVM LV
VFD=http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-1.1.16.vfd
DRVRISO=http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-1.1.16.iso

[ -e $(basename $VFD) ]     || wget $VFD
[ -e $(basename $DRVRISO) ] || wget $DRVRISO
[ -e $INSTALLDISK ]         || qemu-img create $INSTALLDISK 30G

sudo virt-install -c qemu:///system --virt-type kvm --name win7virtio --ram 1024 --disk path="$INSTALLDISK",bus=virtio \
--disk $(basename $VFD),device=floppy --os-variant win7 --cdrom $(basename $DRVRISO) --cdrom "$WINISO" --vcpus 2

et voila Windows installer kicks in, we're greeted with the usual

Next the welcome screen, and agree on the license agreement (if you want!)

In this next screenshot, we see how Windows Installer does not see any "disks". This is because Windows now does not have any drivers for the virtio disk that we attached!

Let's fix that, clicking "Load Driver" and clicking OK, Windows starts reading the VFD virtual floppy image we attached, and detects the driver

Clicking Next, Windows can now see the virtio disk

Then it's business like usual

After installation completes, be sure to check the device manager and confirm the disk (and network) are both using optimized virtio drivers

et voila, Windows in its full glory running over optimized virtio drivers. The only part that sucks is desktop VNC performance, something that I'll probably write about very soon :)

First Ever Ubuntu Cloud Days

2011-03-01T19:31:00.001+02:00

I am very excited to announce the very first Ubuntu Cloud Days! UCD is an online event that is designed for everyone interested in cloud computing, as well as someone who is already using Ubuntu server and who's thinking about "that cloud thing" everyone is talking about. If the words "virtualization", "cloud", "server", "scalability" or "automation" intrigue and amuse you, then you should definitely attend! It's a chance to learn a lot, get your questions answered as well as share your experiences and have fun with the rest of the cloud community!

In fact I'll go one bit further, not only should you attend, you should even deliver a session! Ubuntu as usual is all about community, learning from others, and sharing your knowledge to others. It's quite easy to deliver a session! My first session, I was quite surprised how I didn't have to prepare huge content to "fill" an hour, just answering questions and discussing with people, easily gets you by and you get a warm fuzzy feeling that you just helped lots of people and shared your knowledge.

So, Ladies and Gentlemen, mark your calendars! March 23rd and March 24th

Fore more information: https://wiki.ubuntu.com/UbuntuCloudDays/

To contribute a session, (or if you're thinking about it) talk to me (kim0) on irc #ubuntu-cloud or email [ kim0 =AT= ubuntu.com ]

Volunteer QA to Ubuntu Server

2011-02-18T14:23:00.000+02:00

A great way to help contribute to Ubuntu and ensure Ubuntu Server remains the rock solid platform that it is, is by committing to helping perform specific QA tests in predefined scenarios. If you're interested, the following test cases specifically need someone to step up and commit to them

Install (JeOS on ESX) QA Testcase
Install (JeOS on KVM) QA Testcase
iSCSI Authenticated Root Installation QA Testcase
iSCSI Unauthenticated Root Installation QA Testcase

These tests are being marked as NOT required for ISO release exit criteria for Ubuntu Server. Quoting Robbie Williamson "Our decision was based on the lack of consistently available resources to verify these configurations. With that said, we are keeping them optional"

Robbie adds "Anyone wishing to reverse this decision, please contact me directly. I have no problem adding them back along with your agreement to run them for each ISO release"

Again that's a great way to contribute to Ubuntu. If interested please contact Robbie directly, or contact me (leave a comment right here) and I will help get you on track

Cloud Backup LAMP Stack like a pro

2011-02-09T21:38:00.000+02:00

Running Ubuntu server in the cloud to run your own LAMP environment is extremely popular. Backing up that environment is definitely something that almost everyone should think about, unless it's a test env. In this screencast I'll demo using the ec2-consistent-snapshot tool to perform consistent database backups over a XFS formatted ebs volume

If you can't see the embedded player, here's a direct link
http://www.youtube.com/watch?v=SPVqJWWiLVI

Shoot me a comment if there's something specific you'd like to see, or if you'd like to contribute one such screencast! Till next time

Hack on Ubuntu Cloud Utils

2011-01-19T15:31:00.000+02:00

Want a chance to hack on the cloud? Yes Ubuntu cloud code? What could possibly be cooler! For us cloud geeks, almost nothing! So, here's the deal. Ubuntu aims to be the best operating system for the cloud, period. Part of that is delivering code and utilities to ease common tasks for people working with Ubuntu as a guest cloud OS. For that I've suggested a utility to migrate ec2 ebs based images across different Amazon regions. Conceptually, what's needed is simple, a tool needs to be written to spin up two virtual servers in the two regions and sync the disks across them then terminate. The Ubuntu server team agrees it's a needed utility, and if done right, it should end up being shipped with Ubuntu (coolness!)

I have written step by step, what needs to be done to accomplish this task in a BluePrint. For those new to that term, BluePrints are Ubuntu's way of writing software specs in the open, where everyone can read and comment on them (don't you just love open!). So, if you're ready to start contributing, click that blueprint link, decide whether you want to hack on this tool, ping me and let's start working on it! For any questions or discussions, please drop by on Freenode IRC #ubuntu-cloud (if you're new to IRC, use this web UI instead http://cloud.ubuntu.com/community/irc-chat/). A great time to discuss this, would be today (Wed) 6pm-UTC during the weekly cloud community meeting. If you miss the meeting no problemo, just talk in the channel anytime, most people are there all the time anyway. I'm already hyper excited. Any questions or feedback? let me know in comments below

Help Ubuntu Server, Voice your Opinion

2011-01-17T17:36:00.000+02:00

Hello Ubuntu Cloud and Server communities. Now is your chance to voice your opinion about Ubuntu server, as well as help make it better! The Ubuntu marketing team would like more insight into why your organization has chosen to deploy Ubuntu on the server/cloud sides (or why they did not) as well as your opinion as to how Ubuntu server can improve. If you're a pro-IT person deploying or supporting Ubuntu server or cloud in any capacity, please help make Ubuntu better by helping us better understand your needs. Here's a survey designed for just that

http://survey.ubuntu.com/

Thanks!

Ubuntu Server news storm

2011-01-13T20:23:00.000+02:00

Check out my interview with Dustin Kirkland. Take a sneak peak into Ubuntu Server's future directions. The server team is working on some pretty exciting technologies for the natty release and later. Things touched on

Virtualization and LXC improvements
Higher quality Java packaging
Automated Server ISO testing, ensuring rock solid releases
Even more Upstart polish
Improving Ubuntu server's Cluster stack
Datacenter power efficiency with powernap
UEC and EC2 image improvements, pv-grub support
Cloud "desktop" images! wow!
HUGE project for cloud/data-center automatic deployment, management, monitoring, debugging. IMO this takes Ubuntu server way above anything out there. The technology is all about picking and integrating the best of breed open-source server technologies. While you could always do that work yourself, having a tightly integrated stack that is known to work, is what makes Ubuntu server an easy and fun platform to build upon

Exciting times indeed, check out the full interview at

Video link: http://www.youtube.com/watch?v=CWNjzwci5e8

Upgrade Lucid to Maverick on EC2

2011-01-13T00:52:00.001+02:00

So you're running Ubuntu server Lucid on EC2 cloud EBS root, and decide you don't really want to wait till the next LTS release (12.04)! You want to upgrade your server to Maverick, well, wait no further. Here's how to do it

Let's download the list of ec2 images, get a list lucid ebs i386 amis

ec2-describe-images --all > /tmp/ec2-images
$ grep '099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386' /tmp/ec2-images

IMAGE ami-714ba518 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20100427.1 099720109477 available public  i386machine aki-754aa41c   ebs paravirtual
IMAGE ami-1234de7b 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20100827 099720109477 available public  i386 machine aki-5037dd39   ebs paravirtual
IMAGE ami-6c06f305 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20100923 099720109477 available public  i386 machine aki-3204f15b   ebs paravirtual
IMAGE ami-480df921 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20101020 099720109477 available public  i386 machine aki-6603f70f   ebs paravirtual
IMAGE ami-a2f405cb 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20101228 099720109477 available public  i386 machine aki-3af50453   ebs paravirtual

Start our instance, if you were already running a lucid server, of course you wouldn't have to do that, but for our demo, we'll create a lucid instance to upgrade it to maverick

$ ec2-run-instances ami-a2f405cb --instance-type t1.micro -k default
RESERVATION r-33271c59 553172479171 default
INSTANCE i-05cd0169 ami-a2f405cb   pending default 0  t1.micro 2011-01-12T20:42:12+0000 us-east-1d aki-3af50453   monitoring-disabled     ebs     paravirtual 
$ ec2-describe-instances | awk '-F\t' '$1 == "INSTANCE" { print $4 }'
ec2-67-202-26-253.compute-1.amazonaws.com

Let's ssh into our to-be-upgraded server, install latest lucid upgrades

ssh ubuntu@ec2-67-202-26-253.compute-1.amazonaws.com
screen -S upgrade

sudo apt-get update
sudo apt-get dist-upgrade

Normally, you wouldn't be able to update a LTS release, except to another LTS release. We'll change that to upgrade to maverick, and start upgrading

$ sudo sed -i.bak -e 's@lts$@normal@' /etc/update-manager/release-upgrades

$ do-release-upgrade
Checking for a new ubuntu release
Done Upgrade tool signature
Done Upgrade tool
Done downloading
extracting 'maverick.tar.gz'
authenticate 'maverick.tar.gz' against 'maverick.tar.gz.gpg'
tar: Removing leading `/' from member names

Reading cache

Checking package manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Building data structures... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
Building data structures... Done

Updating repository information
WARNING: Failed to read mirror file
100% [Working]
Checking package manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Building data structures... Done

Calculating the changes

Calculating the changes

Do you want to start the upgrade?

2 installed packages are no longer supported by Canonical. You can
still get support from the community.

3 packages are going to be removed. 29 new packages are going to be
installed. 274 packages are going to be upgraded.

You have to download a total of 124M. This download will take about
15 minutes with a 1Mbit DSL connection and about 4 hours with a 56k
modem.

Fetching and installing the upgrade can take several hours. Once the
download has finished, the process cannot be cancelled.

Continue [yN]  Details [d]

Agree to any prompts while upgrading, till it actually finishes. Let's not reboot the server, instead we power it off

System upgrade is complete.
Restart required 

To finish the upgrade, a restart is required. 
If you select 'y' the system will be restarted. 

Continue [yN]n
$ sudo poweroff

Now what happened is the server has actually been updated from lucid to maverick. However, Amazon would still boot the server using a lucid kernel. That's because for ec2 the kernel is stored outside of the ebs image itself. However, recently amazon added the ability to use pv-grub for booting, that's basically a way to chainload the kernel from inside the ebs image and boot that. That actually makes the cloud server behave just like a bare metal server, which is always a good thing :) To get the aki ID for the pv-grub kernel, we simply grep for the latest maverick images, since those already have pv-grub attached to them

$ grep '099720109477/ebs/ubuntu-images/ubuntu-maverick-10.10-i386' /tmp/ec2-images
IMAGE ami-508c7839 099720109477/ebs/ubuntu-images/ubuntu-maverick-10.10-i386-server-20101007.1 099720109477 available public  i386machine aki-407d9529   ebs paravirtual
IMAGE ami-ccf405a5 099720109477/ebs/ubuntu-images/ubuntu-maverick-10.10-i386-server-20101225 099720109477 available public  i386machine aki-407d9529   ebs paravirtual

Now let's switch the upgraded image to use the pv-grub kernel

ec2-modify-instance-attribute i-05cd0169 --kernel aki-407d9529
kernel i-05cd0169 aki-407d9529
$ ec2-start-instances i-05cd0169
INSTANCE i-05cd0169 stopped pending

$ ec2-describe-instances
RESERVATION r-33271c59 553172479171 default
INSTANCE i-05cd0169 ami-a2f405cb ec2-50-16-133-253.compute-1.amazonaws.com ip-10-112-53-225.ec2.internal running default 0  t1.micro 2011-01-12T21:27:51+0000 us-east-1d aki-407d9529   monitoring-disabled 50.16.133.253 10.112.53.225   ebs     paravirtual 
BLOCKDEVICE /dev/sda1 vol-dc7bc9b4 2011-01-12T20:42:22.000Z

We're done, let's ssh into the new instance, and make sure it's running maverick kernel

ssh ubuntu@ec2-50-16-133-253.compute-1.amazonaws.com
$ uname -r
2.6.35-24-virtual

voila, mission accomplished. Don't forget to terminate your instance

$ ec2-terminate-instances i-05cd0169
INSTANCE i-05cd0169 running shutting-down

Jono'ed at Dallas sprint

2011-01-11T20:51:00.000+02:00

I'm attending Ubuntu's sprint at Dallas, and I've just been Jono'ed. Check out this video where Jono asks me about everything cloud communitish. If you wanna know how to get involved with the ubuntu cloud community, there are some good pointers in there

Cloud LAMP Stack in 60 seconds

2011-01-06T18:25:00.003+02:00

Need to setup a LAMP development or test environment ? on the cloud ? Ubuntu server ? Think it's hard ? Think again! See me setup a LAMP stack, get wordpress up and running in less seconds than most people can eat a sandwich! Check out this video

If you'd like to download the cloud-init file used, as well as other similar files, visit the awstrial project

Advanced cloud-init custom handlers

2011-01-03T19:09:00.000+02:00

I love cloud-init, the Ubuntu cloud technology that enables a cloud instance to bootstrap itself and customize itself into whatever you want it to be (coming from a generic image). I had previously created a screencast introducing cloud-init, and written an article on running cloud-init locally over KVM. This time however, I demonstrate some advanced cloud-init foo, namely how to write a custom content handler in python. Pass the code and data over the cloud's user-data, and watch your code crunch on the data. The possibilities are endless, using this technique you are basically writing plugins for cloud-init allowing you to do almost anything. Ok, enough babbling, let's do some cool stuff

We need the "write-mime-multipart" script from cloud-init. I would not recommend installing cloud-init on your own machine, since cloud-init is designed to be run on cloud instances (not physical nodes). If you do install it on your own machine, it blocks the boot process waiting for the cloud userdata service to appear (which it never does), so you end up waiting a lot! To get the script, we just check out the code directly

bzr branch lp:cloud-init

You'll find that script in the tools/ directory. Assuming you could run cloud-init on local KVM, we now need to replace the user-data file, which in the previous article was written using cloud-config syntax, with a new file. The new user-data file is a multipart file composed of custom python code and data you want your python code to chew on! Here is how you create the file

./write-mime-multipart --output user-data part-handler.txt one:text/plain two:text/plain

Let's take a look at the contents of those files. Files "one" and "two" are the data, while "part-handler.txt" is the python code adapted from cloud-init. In our case, I chose to let our part handler be a "user-creation" provider, i.e. you supply a list of user names in the data file, the code loops over them creating them. Simple enough for an example I hope. Let's check out the data files

$ cat one
jhonny
cash
$ cat two 
agent
smith

These two files hold the 4 users to be created! I split them into 2 files, just to demo you could have multiple input files. Now let's check out the code living in part-handler.txt

#part-handler
# vi: syntax=python ts=4

def list_types():
    # return a list of mime-types that are handled by this module
    return(["text/plain", "text/go-cubs-go"])

def handle_part(data,ctype,filename,payload):
    # data: the cloudinit object
    # ctype: '__begin__', '__end__', or the specific mime-type of the part
    # filename: the filename for the part, or dynamically generated part if
    #           no filename is given attribute is present
    # payload: the content of the part (empty for begin or end)
    if ctype == "__begin__":
       print "my handler is beginning"
       return
    if ctype == "__end__":
       print "my handler is ending"
       return

    print "==== received ctype=%s filename=%s ====" % (ctype,filename)
    import os
    for user in payload.splitlines():
        print " == Creating user %s" % (user)
        os.system('sudo useradd -p ubuntu -m %s' % (user) )
    print "==== end ctype=%s filename=%s" % (ctype, filename)

Most of the code is just boiler plate. The code needs to implement two functions "list_types()" that returns a list of content types that this code can handle. In our case, we return "text/plain" and "text/go-cubs-go". Note that when we ran "write-mime-multipart" we used the mime type text/plain, and that is the reason cloud-init would invoke this code to handle it. Because the code advertises it can handle text/plain types. The second function the code needs to implement is handle_part. This is called at the very beginning and very end for initialization and tear-down. It is also called on each input file (2 times in our case). A sample run looks like

and sure enough, the 4 users were created

That should be everything you need to know about writing custom mime type handlers as extensions to cloud-init. Indeed that is some pretty amazing stuff! Any questions or comments, leave a comment

Cloud Community Flash, Adnane

2010-12-31T16:29:00.000+02:00

Being my last post of 2010 (Hello 2011!) I wanted it to be special, thus I would like to thank every member and contributor to the Ubuntu cloud community, the whole Ubuntu community, and more generally Linux and the rest of the Open-Source world!

Let's meet Adnane Belmadiaf. Adnane is a committed Ubuntu community member, who has been rocking for the past few months. He's done some great work on the Ubuntu cloud portal. I asked him for a few words about himself, and here's what he has to say

AB: Hi, I'm Adnane, a 22 years old Ubuntu user based in Morocco. I was born to be a Web developer (Yes i am a CSS ninja XD) and I am a happy Linux user since 2008. I started actively contributing in 2009. I participated in many projects for Ubuntu such as (The Ubuntu Manual Project, LoCo Directory, and the Ubuntu Cloud Portal). It is a very good experience for me because I learnt and still learn a lot of things from others! The Ubuntu community is something that I’m very passionate about. It’s just awesome to see how it grows and evolves.it’s an inspiring environment to be part of!

AK: What keeps you motivated contributing to Ubuntu?

There are a lot of things that help me to stay motivated, but first and foremost - it's the great people i am working with, the atmosphere is always fun and everybody around you is there for support. You know that you're not alone. The appreciation for the work i have done - When a project/work is done there is that quite nice "Thank You" that keeps me motivated all the time. I'm constantly challenged to learn new things and I simply enjoy coding and solving problems. I would tell to everyone who want to be involved on the community, that patience and good work are the keys to success. it’s almost one year since i joined the community and the results are quite surprising. I now think I am ready to apply for an Ubuntu membership, which I am currently pursuing. Wish me luck :)

Cloud Instance with Cloud-Init on KVM

2010-12-30T15:02:00.000+02:00

I wanted to run an Ubuntu server cloud instance locally on KVM hypervisor. I also wanted to run cloud-init on the local setup in order to experiment a bit with it. So that means, downloading a UEC image, booting it under KVM, and passing cloud-init parameters to it as it boots. Much to my surprise things were far easier than I expected, all thanks to our rocking Ubuntu cloud team. Here's the script I cobbled together, most of it is basically a rip off from this UEC Images wiki page.

So what this does is, it:

Downloads a UEC image of natty server daily i386 if it doesn't exist
Sets a few variables
Creates a qcow disk image, shadowing/cowing/differencing the downloaded image. This is to keep the originally downloaded image pristine
Downloads some sample user-data and meta-data (warning, this runs arbitrary commands and injects keys inside your VM, only use for testing!). To experiment with cloud-init you'd have to modify the user-data to your liking
Runs a local simple webserver (port 8000)
Boots the cow image in a local kvm. As it boots, you'll notice on your terminal the following two requests being made "GET /meta-data" and "GET /user-data". Cloud-init uses this data to customize the image as it boots
Once you close kvm, kills the web-server

So all you need to do, is create an empty directory, put this script in it and run it

uecnattyservercurrent=http://uec-images.ubuntu.com/server/natty/current/natty-server-uec-i386.tar.gz
tarball=$(basename $uecnattyservercurrent)
[ -f ${tarball} ] || wget ${uecnattyservercurrent}
contents=${tarball}.contents
tar -Sxvzf ${tarball} | tee "${contents}"
cat natty-server-uec-i386.tar.gz.contents
base=$(sed -n 's/.img$//p' "${contents}")
kernel=$(echo ${base}-vmlinuz-*)
floppy=${base}-floppy
img=${base}.img
qemu-img create -f qcow2 -b ${img} disk.img
wget http://smoser.brickies.net/ubuntu/uec-seed/meta-data
wget http://smoser.brickies.net/ubuntu/uec-seed/user-data
python -m SimpleHTTPServer &
websrvpid=$!
kvm -drive file=disk.img,if=virtio,boot=on -kernel "${kernel}" -append "ro init=/usr/lib/cloud-init/uncloud-init root=/dev/vda ds=nocloud-net;s=http://192.168.122.1:8000/ ubuntu-pass=ubuntu"
kill $websrvpid

Ubuntu Cloud Tech Content

2010-12-27T18:52:00.000+02:00

I had sent out calls to the Ubuntu cloud community asking what kind of technical content or training materials (tutorials, screencasts ..etc) they were interested in seeing. I got quite some feedback that boils down to the following being the top 10 topics

Modifying Ubuntu images and rebundling to EC2
Creating images from scratch with vmbuilder
P2V and V2V conversions (Physical, VirtualBox, VMware...)
Advanced cloud-init (custom handlers, multi-server, includes)
Provisioning and deploying Web applications (e.g. rails, django) to the cloud
Best practices for upgrading a server install cross-release
Load balanced LAMP multi-tier installation
Best practices around creating cloud instance snapshots
Backing up live cloud instances
Restoring cloud servers from backup/snapshots

Now, what's a better way to spend your holidays than hacking together content addressing those needs! Yup, nothing beats helping your fellow Ubuntuians :) So if you're feeling like contributing content to any of the above, drop me a comment or shoot me an email (kim0 AT ubuntu.com). If you'd like to contribute to some other content, do grab me as well! If you're unsure, or want to talk about how you can get involved (there's always a way) tune in to the weekly cloud community hour

Ubuntu Cloud Community Meeting

2010-12-15T15:17:00.000+02:00

Just a quick update, the weekly cloud community meeting time has been changed to 6pm-UTC, so that the US is awake. Everyone is invited, it's a free party! Once in the meeting, we'll all be helping each other out, trying to answer questions, having discussions, agreeing, disagreeing but most certainly having fun :) If you're not sure whether or not you should attend, trust me you should. It's a free form meeting where everything goes! So come along and bring your friends.

More details on how to join are at: https://wiki.ubuntu.com/UbuntuCloudMeeting

Fix Eucalyptus for Natty yourself

2010-12-13T14:18:00.001+02:00

In a previous post I mentioned how the Ubuntu server community needs your help fixing some broken packages on the next shiny Ubuntu release "Natty". I wanted to see just how difficult can it be to actually fix those bugs, so I decided to go crunching on a couple. Much to my surprise, fixing those bugs turned out to be way much easier than I thought. I created a couple of patches, mostly just adding one parameter or moving things around to make the build process happy, submitted the branches for review, et voila it gets reviewed, merged, and the bug which is attached to your branch gets automatically closed. Quite an easy process indeed! Once the bug closes, you get an exceptionally warm fuzzy feeling that you just helped the world, contributed to making millions of Ubuntu users happy!

Which brings us to this post, I see the Eucalyptus package is broken on natty as well. The list of remaining packages is at: http://tinyurl.com/server-ftbfs As can be seen, the Eucalyptus packages are "High Importance", so this is quite a significant contribution from anyone who cares about the helping the cloud community (or the server community in general). I would like to encourage you to start fixing those bugs, if you'd like help the #ubuntu-motu channel should be very helpful

Help Fix Ubuntu Server Packages FTBFS

2010-12-10T13:33:00.000+02:00

With every transition to a new Ubuntu release, certain packages break due to different reasons such as newer versions of components, an updated tool-chain and so on. Since we all love and care about Ubuntu Server, here is a list of Ubuntu server packages that currently FTBFS (Fails to Build from Source)
http://tinyurl.com/server-ftbfs

This means that those packages are not compiling and building the intended binary packages on Natty. This is a great way for you to get involved! If you know how to build packages from source code (the trio ./configure ; make ; make install) and can debug things like missing development packages, missing headers ...etc, then this will be a walk in the park for you :) Everyone's favorite Daniel Holbach is working on a proper guide for fixing these things, however for now you can find lots of useful information at
https://wiki.ubuntu.com/Bugs/HowToFix

In case you need help, the channel #ubuntu-motu on freenode IRC is a good place to ask

Introducing Ubuntu Cloud-Init Technology

2010-12-08T16:16:00.000+02:00

Ubuntu Cloud-init is an awesome piece of software that helps Ubuntu run as great as it does on the cloud. Cloud-init kicks in as the server boots, and starts converting your server from the generic template it has been started from, into the server image you need. Coupled with the easy to use cloud-config syntax, it's just so easy and quick getting this rolling. Check out this screencast, where I introduce cloud-init and demo what it can be used for

Questions ? Comments ? I'm all ears, leave me a note in the comments section and I'll reply back
Want to create your own screencasts (it's real easy), ping me (kim0 on irc) and I'll be sure to help you publish them

Announcing Ubuntu Cloud Portal

2010-12-03T18:19:00.000+02:00

Another rocking day for the Ubuntu Cloud community! The Ubuntu Cloud Portal has just been launched hurray. The portal helps new-comers to the Ubuntu cloud community quickly find interesting information they may care about such as documentation to read/edit, projects that may interest them and so on. In this first release the following is available

Front page: Lists important news to the Ubuntu Cloud community, latest tweets and happenings (RSS). The front page also features the widely anticipated (drum roll please ... ) AMI Locator application which helps anyone using Ubuntu on the EC2 cloud quickly locate the AMI needed. Go ahead and bookmark this right now
Documentation: Quick links to useful documentation pages on the wiki. If you'd like to start contributing, let me know
Community: Pointers to where you should be, mailing lists you should subscribe to, forums and IRC rooms that should interest you
Developer: Once you're ready to start contributing, this page should list all open-source projects that relate to Ubuntu and the cloud. I have added quick links to locate code/bugs/community/features for every project
Planet: Collecting Every word written about Ubuntu Cloud on the world wide web (RSS). If you think your blog should be aggregated here, let me know!

This is only the first release, so I'm sure lots could be added. Please do ping me if you would like to help improve the portal in anyway, or generally if you'd like to contribute to Ubuntu Cloud efforts. I've put together a quick video demo'ing the portal

I would like to thank the Ubuntu server and cloud teams for their help and guidance. I would also like to thank Adnan Belmadiaf (daker on IRC) for his continued contributions to this portal. Adnan is a 22 years old Ubuntu user living in Salé, Morocco, he works as a Web Developer. He's a member of the Ubuntu Morrocan Team, and involved in different projects such as (The Ubuntu Manual Project, The Loco Directory, The elementary project). It's that fuzzy warm feeling of contributing to Ubuntu that keeps its community rocking. Let me know what you guys think in comments.

Ubuntu Cloud Forums a great place

2010-12-03T11:46:00.000+02:00

The Ubuntu Cloud Forum is a great place to be in. We launched the forum about 3 months ago, thinking it may not see much activity. However surprisingly (in a positive sense) we're getting tens of active threads per month, each with hundreds if not thousands of views! Thanks to everyone in the Ubuntu forums community for making this such a great place to be in!

In case you're wondering, you can easily access the Ubuntu Cloud Forum at:
http://ubuntuforums.org/forumdisplay.php?f=392&order=desc

And if you're an efficiency superhero who wants the information to come right at your fingertips, you can follow the forums RSS feed at:
http://ubuntuforums.org/external.php?type=RSS2&forumids=392

The Cloud Forum is not only for people wanting to ask questions (although it's great for that!), it is also a great place for people to share experience and knowledge. If you have just setup UEC on a brand new server farm in some fancy way, or if you've built your infrastructure on top of Ubuntu on EC2 and you're proud of what you've built, chances are others, like yourself, are going to want to learn about how to do the same. So join in and share the love

What Cloud sessions interest you

2010-12-01T16:32:00.000+02:00

In case you didn't know yet, the Ubuntu cloud community is invited for a weekly gathering on IRC in #ubuntu-cloud. You can find more details here. Once inside feel free to shoot any question you may have about Ubuntu Enterprise Cloud, or running Ubuntu as a guest OS on top of a commercial cloud such as EC2 or otherwise. Also I think it'd be a good idea to start hosting some technical sessions through those meetings. I am volunteering to start hosting, but please grab me if you'd like to host some sessions yourself. I'm reaching out for the community to identify topics you would be interested in discussing through those sessions

Leave me a comment mentioning what sort of topics you'd be interested in seeing discussed
Leave me a comment if you're willing to host one of those sessions
Leave me a comment if you're running an Ubuntu cloud setup and would like to share your experiences

You get the idea, leave me a comment because every piece of feedback is valuable. Grab kim0 on irc in #ubuntu-cloud, or shoot me an email to kim0 _AT_ ubuntu.com

Ubuntu Server in EC2 Cloud, Easy!

2010-11-30T14:02:00.000+02:00

I am starting to create a series of screencasts to demonstrate various topics relating to running Ubuntu on the cloud or as the cloud. The first video demos how easy it is to start your very first Ubuntu server in the Amazon EC2 cloud. If you ever wanted to play with Ubuntu server in the cloud and had any doubts, this video should put them to rest :)

If you think that is cool, and if you want to contribute your own, please grab me.It's real easy to create such screencasts, I can help get you kick-started. And hey you'd be helping the Ubuntu community, and on your way to becoming an online celebrity, what's not to like! If you'd like to follow similar screencasts, subscribe to this youtube channel

Let me know in comments what you would like to see in future screencasts, or whether certain topics interest you. If you are using Ubuntu server in the cloud professionally, I'm very interested to hear back from you. Grab me (kim0) on #ubuntu-cloud on IRC (freenode) for a chat. Awaiting the flood of screencasts :)

Ubuntu Cloud Q+A weekly meeting

2010-11-24T12:23:00.000+02:00

The Ubuntu cloud community is coming together today [ 3pm UTC/GMT / 7am Pacific / 10am Eastern ] for its first weekly Q+A meeting. If you use Ubuntu as a guest OS on a public cloud, or if you have built your own private cloud infrastructure on top of Ubuntu, or even if you're just interested in any of that cloud babel please do join us in this first meeting for a great chance to connect to other users and developers of Ubuntu cloud technology. Through those online meetings you will get a chance to connect to the rest of the Ubuntu cloud community, share experiences, ask questions, find areas that interest you and perhaps start contributing to them

Information on how to connect and details can be found here

Ubuntu Cloud Screencasts Volunteers

2010-11-22T13:29:00.000+02:00

Interested in Ubuntu cloud community ? Want to help ? Awesome! here is your chance

Screencasts are a great way to introduce new-comers to something new. I always find it helpful to view a couple of short videos to "get-a-feel" of thingX before I actually start reading and working on it. That's why I'd like to start a screencast series introducing running Ubuntu in the cloud. The target is to start by simple stuff (no voodoo here sorry) in order to demo how simple running Ubuntu in the cloud really is. Of course this can grow into a gigantic series, however for starters I'd like to focus on basic and very common use cases. Here are a few casts I would like to begin with:

Creating your first Ubuntu server in the cloud (GUI, CLI or both)
Introducing Ubuntu Cloud-Init technology
Customizing (Re-bundling) available Ubuntu images (AMIs)
Launching a LAMP app on the cloud
Backing up your Ubuntu LAMP cloud instance
Creating and Load Balancing a multi-tier LAMP app

This list is by no means set in stone :) This is a dynamic list that will change according to feedback. Feel free to join the ubuntu-cloud mailing list at: https://lists.ubuntu.com/mailman/listinfo/Ubuntu-cloud to discuss and change those topics.

If you're interested to record any of those casts, please do shout at me! You can email me kim0 [AT] ubuntu.com or grab me for a chat in #ubuntu-cloud IRC channel on Freenode

If you're new to all this cloud stuff, and would like to see a screencast covering a certain topic, please let me know in the comments (or email, mailing list, irc ...). If there's some demand on a specific topic, I'll try to cover it. Of course, if you can contribute and cover it yourself, that would be awesome indeed. After all, it's all about the community. Those wishing more information about recording screencasts, can read more information here

Awaiting the flood of excited contributors :)

Cloud Computing 101, p2

2010-11-18T19:17:00.001+02:00

Continuing my part-1 post about cloud computing basics, this second post should continue to define different types of "clouds" as well as what you gain and loose by using them

If you look at a cloud solution, it's really a bunch of software layers stacked on top of each other. You have the hardware (servers, disks, switches, routers), you have bare metal operating systems, hypervisors, virtual servers and inside those you have programming languages (python, java, php), development frameworks, database servers, and your own business logic code living on top! Clouds are categorized as either IaaS, PaaS or SaaS. The type of cloud is basically defined by which layers of the stack the cloud abstracts away from you, and which layers you "own" and control. Another categorization scheme is private, public and hybrid clouds. Let's take a quick tour on what each of those cloud types mean

IaaS is Infrastructure as a Service. The cloud abstracts away as little as possible from you. Basically the cloud provides you with virtual servers, networking and storage and that's it. You use those building blocks, just as you would use them in any physical datacenter to build your own compute infrastructure. The only difference is that you don't worry about how the servers are powered, cooled, what brand of disk or SAN is used ..etc. All you care about is your provider's SLA as mentioned in part-1. Other than that, it's business as usual

PaaS is Platform as a Service. The cloud is abstracting away the infrastructure and some more. The cloud in this case is no longer composed of virtual servers and disks, it is however a "development framework". When you write code, you are coding against the platform, against the cloud itself. A PaaS cloud, assuming you're creating a web application, would tell you how to route requests to your handlers, how to write code to handle specific requests, would provide an API for storage, would perhaps provide an API for database (SQL, or noSQL doesn't matter here). Your application code is written against the API of the cloud. As such, you have no idea about "low level" details such as networking, IP addresses, failed servers or even the number of virtual servers running your code! So essentially you upload your code archive and it just runs on the cloud, no questions asked

SaaS is Software as a Service. In this case, you're only using software running on the cloud that someone else had written. If you've used facebook, Gmail, linkedin, Google docs, SalesForce ...etc that's it. In essence the service you're getting is the actual final "application" you need. This is the highest level of abstraction. You do not concern yourself with infrastructure, nor with code to build an application with. You pay to use the application itself and the SLAs you get are for the application availability and your data availability

What type of cloud suits you best, is a question that needs some thought and that depends on one's set of requirements. IaaS clouds provide the least abstraction and the most control! They are a good first step to migrating off-the-shelf software to the cloud and benefiting from cheap, on-demand, elastic infrastructure. Since they provide the least abstraction, if you'd like a scalable infrastructure, you would have to do all the work yourself. It is generally not so painful to migrate from an IaaS cloud to another. PaaS clouds however, since they provide higher levels of abstraction, are much easier to manage and scale. It essentially auto-scales delivering cloud computing holy grail. However the big price is that you generally have to rewrite your application to the particular cloud platform. Not only is that painful, but it also may lock you in to the cloud vendor making it extremely hard to change vendors afterwards. Which is why I think the open-source world needs great open-source PaaS cloud frameworks (Have a favorite? drop me a line in the comments section). If a SaaS application meets your needs at a good price point, then the only potential disadvantage would be data lock-in, as well as the (in)ability to mashup the SaaS application with other tools. A good piece of advice here is to choose SaaS applications that provide full API access to your data such that you can easily pull off all data and meta-data should you need to.

A different categorization of clouds is private vs public. Private simply means that the cloud infrastructure is built in-house behind the firewall. For example you could turn your corporate datacenter into a private cloud. The benefits being, you gain better efficiency and datacenter utilization across different departments as well as being able to provide an elastic and fast response to your enterprise's departmental IT needs. Should you want to start playing with a private cloud solution, Ubuntu Enterprise Cloud is a good start. Public clouds pertain to a cloud run by a third party service provider. A public cloud is either Iaas, PaaS or SaaS or even a mix of some. Why you would want to migrate some workloads to a public cloud, is simply because public cloud vendors due to their economies of scale are able to provide equivalent if not better service, at a significantly lower price point coupled by the ability to instantly grow. A hybrid cloud on the other hand is a private cloud that can "burst" to a public cloud when its resources are exhausted. The goal is to bring the best of both worlds, the control, data-security of private clouds with the elasticity and economies of large scale public clouds. More and more work-loads are being migrated to the cloud and it's all just starting.

Has your organization migrated some workloads to the cloud already, are you planning on that? Are you planning on building your own private cloud? Please let me know in the comments, let me know the motivations and the challenges you faced. If you have any questions in general, let me know as well

Show Off Ubuntu Desktop on Cloud

2010-11-12T17:07:00.000+02:00

Want to show off your Ubuntu desktop in the cloud ? Perhaps you want to demo it to some Windows or OSX friends. Perhaps new users at your loco event want to play with Ubuntu for a bit. Well, look no further. In this article I will create an Ubuntu maverick 10.10 desktop in the Amazon ec2 cloud, connect to it using the x2go terminal server, which leverages the excellent NX remote display libraries

Start by launching the following AMI (ami-1a837773). I chose the official Ubuntu 32bit ami, so that we can run it on a m1.small instance. If you're not sure how to launch this instance, you might want to review my point-n-click guide. After launching the instance and logging in, I do my customary

ssh ubuntu@xxxxx   #replace with your instance's public dns name
sudo -i
screen
apt-get update && apt-get dist-upgrade -y

Let's install x2go terminal server

# gpg --keyserver wwwkeys.eu.pgp.net --recv-keys C509840B96F89133
# gpg -a --export C509840B96F89133 | apt-key add -
# echo "deb http://x2go.obviously-nice.de/deb/ lenny main" >> /etc/apt/sources.list
# apt-get update
# apt-get install x2goserver-home

Optional step: Switch system to libjpeg-turbo

I like to break my Ubuntu system by installing unsupported software, so I will be switching the system's default libjpeg into a newer variant that utilizes your CPU's SIMD instruction set to provide better performance. Since connecting to a desktop remotely, heavily utilizes jpeg compression I suspected this step would provide me a performance boost. It is however not recommended, especially to someone who wouldn't be comfortable fixing his system using console only. You need to do the following on the ec2 server and on your own system. I am assuming 32bit systems, you can 32/64 bit versions here

# wget 'http://sourceforge.net/projects/libjpeg-turbo/files/1.0.1/libjpeg-turbo_1.0.1_i386.deb/download' -O libjpeg-turbo_1.0.1_i386.deb
# dpkg -i libjpeg-turbo_1.0.1_i386.deb
Selecting previously deselected package libjpeg-turbo.
(Reading database ... 25967 files and directories currently installed.)
Unpacking libjpeg-turbo (from libjpeg-turbo_1.0.1_i386.deb) ...
Setting up libjpeg-turbo (1.0.1-20100909) ...

# ls -l /usr/lib/libjpeg.so.62
lrwxrwxrwx 1 root root 17 2010-11-12 12:35 /usr/lib/libjpeg.so.62 -> libjpeg.so.62.0.0
# rm -rf /usr/lib/libjpeg.so.62
# ln -s /opt/libjpeg-turbo/lib/libjpeg.so.62.0.0 /usr/lib/libjpeg.so.62

End-Of-Optional-Step

Install the Ubuntu desktop itself (The GUI)

apt-get install ubuntu-desktop

This takes a good 10-15 minutes. After which your system is ready. Grab yourself a favourite x2go client here. Send your friends links to the Windows and OSX clients and let them see the light :) In my case I just used my Ubuntu system to connect remotely. So I added the same repo we added before and installed "x2goclient" which is a qt4 client. Here are the settings I used

I am using my ssh key to login to the Ubuntu virtual desktop. If you're using Win/OSX and perhaps wouldn't want to use the ssh key, reset the Ubuntu user password and connect using the password. Once connected we see our familiar and beautiful Ubuntu desktop

I was pleasantly surprised to hear the drums-beating sound of Ubuntu booting! wow! That was just awesome. x2go uses pulseaudio to remotely connect and bring audio right to your desktop. I also could easily forward my local files to the instance in the cloud. Anyone already using Ubuntu desktop in a cloud ? let me know about it ? What kind of use cases you'd use such a setup for ? If you have some fancy setup, let me know about it as well

OpenStack dev env on EC2

2010-11-10T13:52:00.000+02:00

Just as I previously blogged about running your own UEC on top of EC2 (cloud on cloud), here is another cloud on cloud post showing you how to run an OpenStack compute development environment on top of EC2. All of the heavy lifting is really done by the awesome novascript! I started by launching Ubuntu server 10.10 64bit (ami-688c7801) on a m1.large instance. If you're not sure how to get this done, please check my visual pointnclick guide to launching Ubuntu VMs on EC2

Once ssh'ed into my Ubuntu server instance I fire an update

sudo -i
apt-get update && apt-get dist-upgrade

Let's see the available ephemeral storage

root@ip-10-212-187-80:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  579M  8.8G   7% /
none                  3.7G  120K  3.7G   1% /dev
none                  3.7G     0  3.7G   0% /dev/shm
none                  3.7G   48K  3.7G   1% /var/run
none                  3.7G     0  3.7G   0% /var/lock
/dev/sdb              414G  199M  393G   1% /mnt

As you can see, /mnt is auto-mounted for us. We don't really need this. For nova (OpenStack compute component) to start it needs an LVM setup with a LVM volume group called "nova-volumes", so we unmount that /mnt and use sdb for our LVM purposes


# umount /dev/sdb
# apt-get install lvm2

root@ip-10-212-187-80:~# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created
root@ip-10-212-187-80:~# vgcreate nova-volumes /dev/sdb
  Volume group "nova-volumes" successfully created
root@ip-10-212-187-80:~# ls -ld /dev/nova*
ls: cannot access /dev/nova*: No such file or directory
root@ip-10-212-187-80:~# lvcreate -n foo -L1M nova-volumes
  Rounding up size to full physical extent 4.00 MiB
  Logical volume "foo" created
root@ip-10-212-187-80:~# ls -ld /dev/nova*
drwxr-xr-x 2 root root 60 2010-11-10 10:27 /dev/nova-volumes

I had to create an arbitrary volume named "foo" just to get /dev/nova-volumes to be created. If there's some other better way, let me know folks. Let's go checkout the novascript. You need to do that somewhere that has more open permissions than /root :) so /opt is perhaps a good choice


# cd /opt
# apt-get install git -y
# git clone https://github.com/vishvananda/novascript.git
Initialized empty Git repository in /opt/novascript/.git/
remote: Counting objects: 121, done.
remote: Compressing objects: 100% (114/114), done.
remote: Total 121 (delta 42), reused 0 (delta 0)
Receiving objects: 100% (121/121), 16.62 KiB, done.
Resolving deltas: 100% (42/42), done.

From here, we simply follow the novascript instructions to download and install all components


# cd novascript/
# ./nova.sh branch
# ./nova.sh install
# ./nova.sh run

Watch huge amounts of text scroll by as all components are installed. The final "run" line starts a "GNU/screen" session with all nova components running in screen windows. That is just awesome! For some reason though, my first run was unsuccessful. I had to detach from screen, ctrl-c kill it. I then tried starting the nova-api component manually, which did work fine! I then tried to run the script again, and strangely enough this time it worked flawlessly. Probably an initialization thing only. Thought I'd mention this in case any of you guys face this issue. Here's what I did, which you may or may not have to do

./nova/bin/nova-api --flagfile=/etc/nova/nova-manage.conf
# ./nova.sh run   # works this time .. duh

Almost there! Nova's components are now running inside screen. You're dropped into screen window number 7. From there we proceed to create some keys, launch a first instance and watch it spring to life


# cd /tmp/
# euca-add-keypair test > test.pem
# euca-run-instances -k test -t m1.tiny ami-tiny
RESERVATION     r-yehvnkwa      admin
INSTANCE        i-3fxfo2        ami-tiny        10.0.0.3        10.0.0.3        scheduling      test (admin, None)      0               m1.tiny 2010-11-10 10:50:27.337898                      
# euca-describe-instances
RESERVATION     r-yehvnkwa      admin
INSTANCE        i-3fxfo2        ami-tiny        10.0.0.3        10.0.0.3        launching       test (admin, ip-10-212-187-80)  0               m1.tiny 2010-11-10 10:50:27.337898                      
# euca-describe-instances
RESERVATION     r-yehvnkwa      admin
INSTANCE        i-3fxfo2        ami-tiny        10.0.0.3        10.0.0.3        running test (admin, ip-10-212-187-80)  0               m1.tiny 2010-11-10 10:50:27.337898

Let's ssh right in


# chmod 600 test.pem
# ssh -i test.pem root@10.0.0.3
The authenticity of host '10.0.0.3 (10.0.0.3)' can't be established.
RSA key fingerprint is ab:96:c3:ee:22:84:28:2f:77:ad:d9:a9:52:63:7c:f9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.3' (RSA) to the list of known hosts.
--
-- This lightweight software stack was created with FastScale Stack Manager
-- For information on the FastScale Stack Manager product,
-- please visit www.fastscale.com
--
-bash-3.2# #Yoohoo nova on ec2

Once you detach from screen, all nova services are killed one by one to clean things up. On that setup, you can immediately hack on the code, then re-launch nova components to see the effect. You can use bzr to update the codebase and so on. In case you're wondering if this works on KVM on your local machine, it does work beautifully! Of course instead of the LVM setup on the ephemeral storage step, you'd have to pass a second KVM disk to the VM. Other than that, it's about the same. How awesome is that. Let me know guys if you have any questions or comments, also feel free to jump on IRC on #ubuntu-cloud and grab me (kim0). Have fun

Cloud Computing 101

2010-11-05T17:11:00.001+02:00

I get asked every now and then, what is this cloud thing, why is it cool, why is everyone talking about it, why should I care! As you see, that's a lot of Whys! In this post I attempt to put an end to those Whys with some Answers. Most of the blogosphere around cloud computing, gets caught in fine details, and the latest bits and pieces of technology, while ignoring newcomers who are not quite sure why is everyone so hyped about cloud to begin with. I hope to help newcomers gain a better view of what the fuss is all about

Let's assume for a moment, you're called Jim and you're the IT manager at a fictional organization. Your boss walks in, tells you the development team is ready to deploy their ultra scalable video sharing web application. It's very hard to determine how well the market accepts the new web app, we could be the next Youtube, or we could have a much harder start. We estimate to need anywhere between 10 and 100 servers the first couple of weeks, and anywhere between 1 and 50 terabytes of storage depending on market demand. The boss opens the door ready to leave, then he turns around and tells you, can you please have that ready by the end of this week! Talk about poor management in this hypothetical company, in reality things are not that bad, well and in many cases not much better either. So, if you were Jim, you would now probably be thinking of ways to end your life, or at least you'd be writing a farewell email! With the advent of cloud computing however, you have other options. You can snap your fingers, and have a hundred servers created, snap them again and have 50TB of storage appear right next to them, ready to serve you. If you think that's more "magical" than the iPad launch, you would be right although I'm sure Steve Jobs would disagree. That magic is what hypes many IT people about clouds! Well technically, instead of snapping your fingers, you'd perform an API call to a cloud provider. That means you either run a command or click a button in some management tool and those resources spring to life! Can you already feel how enabling this cloud thing is!

Cloud is called "cloud" because you don't really know what's inside it or how it is built. A cloud icon was and is the standard representation of the "Internet" or a remote network that you don't really care about, or don't control. It is in essence a black box to you, with traffic going in and coming out the other end. You should not know or care how it is built, you're not involved in its daily operation. It provides you "services" that you use. In Jim's case, those services were large numbers of servers, storage and of course networking (you still need a way to access those remote resources anyway!). Jim requested his 50TB of storage and got them, he does not really know what is the physical backing store to this storage service. Are those terabytes of storage (which are holding his company's most precious data) living on a fiber connected high-end SAN storage, low-end SAN or is it a NAS filer. How are the servers accessing the storage network, is it high performance infiniband? fiber connections ? iscsi ? AoE? Lots of options, but Jim doesn't really know. Whether or not he cares is a different story. I would say, he should not care about the technology used to build the solution, however he should care about the SLA his money is buying him. i.e. When you buy storage you are not only buying capacity, you're also buying redundancy and performance. Which is probably why many IT people care about the brand of the SAN storage, and the server to storage connectivity to begin with. What they really care about is "Is my data safe on that storage", "Will that storage deliver the performance I need". They really care about the SLA the cloud services provider is able to achieve

A common misconception about "cloud" is that cloud equals virtualization. This is not really true. You could very well build a cloud solution that does not use virtualization, instead a physical server would be powered on, PXE booted, deployed and be ready to service you! It would probably end up being too expensive, inflexible, and with limited billing options but it would not be impossible to build. That's why most commercial cloud vendors end up using some kind of virtualization technology as a building block in their "cloud compute service" i.e. the CPU and memory cloud layer. Virtualization is a neat trick to split up a physical server into multiple virtual machines, each running its own operating system and each having its own completely separate software stack. It enables the cloud service provider to carve up different sizes of virtual servers from the underlying physical servers. As a cloud consumer, you end up paying for only the size your workload needs.

The reason why cloud computing usually ends up being compared to the electricity grid is because both provide you with on-demand services meeting a certain SLA, and that you end up paying for only the amount you used. In case of electricity, you don't care what equipment the electricity company is using to generate your current, you only care that it meets a certain SLA (say 220V able to pump current of upto 100A, and being online for 99.999% of the time). You could run your own generators, but it would be inefficient (expensive) to do so, it would be a hassle to keep everything running, it would require skilled workers keeping everything online, it would not scale if you suddenly need more current! That is why most people do not run their own electricity generators and instead depend on the grid. However, with all the disadvantages mentioned, some businesses still choose to own and operate their own diesel engines for generating electricity at least as a backup solution. Why that is the case, is because those businesses are seeking more "security". They want to be in control, they don't want the electricity company to control such an important resource to their business. Everything mentioned so far about the electricity company applies to cloud vendors as well. Cloud vendors are the IT equivalent of the electricity grid. Running your own datacenter is the analogue of owning a diesel engine. Of course almost every business nowadays owns, builds and operates its own datacenter (diesel engine). However that might be changing rapidly, we're already seeing signs of workloads shifting into the cloud, which is what all the fuss is about. Cloud is the electricity grid of the IT world, and perhaps in the no too distant future it would be powering the vast majority of our personal and professional IT needs. Cloud is all about the commoditization of IT resources and services, coupled with a new business model for consumption lowering the entry barrier for smaller businesses and helping them focus on their core competency instead of focusing on IT

I hope to have helped shed some light on the topic, I'll probably be writing a part-2 soon touching on types and key properties of a cloud as well as adoption barriers and compromises. I understand many "cloud people" disagree about what qualifies as a cloud, it's definition and well basically everything about cloud is debatable, do let me know (politely :) if you disagree with any of the points mentioned. Let me know in the comments what you think are the key properties of a cloud

Update: Continue reading part 2

Egypt LoCo Maverick release party

2010-11-02T16:14:00.000+02:00

The fun is everywhere :)

A link to the whole set
http://www.flickr.com/photos/maggieosama/sets/72157625277893568/show/

Ubuntu is free, fun and global!

Ubuntu Cloud Community needs You

2010-10-29T18:04:00.000+02:00

"I'm interested in Ubuntu and the cloud, how do I get involved" is a question I got a few times already. I thought it would be a good idea to answer this as a blog post. I believe one of the very first things you'd want to do, is to make sure you're on the main communication channels, talking to the community, asking questions, seeing other questions being answered, trying to answer some yourself, sharing opinions and generally "connecting" with the rest of the community. That is a great first step. So I'll highlight the main communication venues for the Ubuntu cloud community, as well as way to get kick-started.

Places to be

Ubuntu Cloud Forums, while pretty young, there has been some pretty good stir in the forums. While IRC and mailing lists may be more focused on "asking questions", the Forums are a great way to get in touch with other community members. To share your experience building your private clouds, the hardware used, software configuration, tuning and optimization, challenges faced ...etc. Come join in, if you would like to ask questions, or if you would like to share opinions, tips or tricks, get on the forums and make some splash :)
The Ubuntu-Cloud mailing list is a great technical resource where most of the experts and developers are subscribed. For very technical discussions, questions, feature suggestions, RFEs, development discussions the mailing list is a great resource.
The EC2Ubuntu mailing list is a great resource that focuses on running Ubuntu in the Amazon EC2 public cloud. This list is active with a wealth of info on the topic
IRC chat has long always been a primary real-time communication tool used by free software enthusiasts. The Ubuntu cloud IRC room is (surprise, surprise) #ubuntu-cloud on Freenode. Jump in, and engage

Once connected, things you can do include playing with the latest technology such as creating yourself a private UEC cloud, verifying latest features work as advertised, report and fix bugs, suggest features, design and implement new projects to advance the state of Ubuntu on the cloud. While the community is very welcoming, I definitely understand we need to create better new-comer friendly engagement paths, more hand-holding if you will. A better mentoring program from senior members as well as low hanging fruit are things the Ubuntu cloud and server communities need to identify and improve to make it easier to attract and engage fresh talent

PointnClick guide to running Ubuntu in the cloud

2010-10-27T14:59:00.001+02:00

My previous post about running uec on ec2, drew some comments as being a fairly complex process. That may be true because essentially you're attempting to hack together a configuration which is not entirely supported. Anyway, that led me to want to demo a "point-n-clink" guide to running your first Ubuntu server in the EC2 cloud. i.e. no command line allowed, just point and click :) It doesn't get any easier than this, so let's hit it

Assuming you have setup an account with Amazon and could login to the Amazon console, you should see

Before we begin, let's set ourselves a "key-pair". Which is basically a ssh public/private key-pair that enables you to login to your instance once launched. Click on "Key Pairs" and click "Create key pair", I'm gonna name my key "ubuntu" and click create

The key is promptly created and pushed for download through your browser. Let's proceed with another "preparatory" setup. By default, EC2's firewall denies all inbound traffic to the instance, which means you cannot even ssh into your instance. Let's open ports 22 for ssh, and 80 just in case we wanna test by running an apache2 server. So, add your from-to ports and source IPs as per the next screenshot and "Save" it

Great, now we're ready to actually launch an instance. Click "EC2 Dashboard" link, click the button "Launch Instance", a wizard starts asking for which AMI we would like to use. An AMI is a template that will be copied to your instance and used to start it. Click "Community AMIs", it may take a moment for the AMIs to load. Now here's a trick, to quickly "zoom-in" on the official ubuntu AMIs, use the following search string "ubuntu-images/" in order to locate EBS based images, and "ubuntu-images-us/" in order to locate instance store based images. This is in no way a "supported" feature of neither Amazon nor Ubuntu, it's just a convenient hack that works today and may not work tomorrow. We choose an EBS based AMI because we plan on launching a micro instance and those require EBS AMIs

click the "Select" button beside it. Click continue and choose a t1.micro instance and click continue a couple of times

You're now on the key-pair page, we simply choose our previously generated "Ubuntu" key pair and click continue

On the firewall configuration page, we choose the "default" security group, since this is what we had configured to open ports 22 and 80 previously

Voila, we're ready. Review and confirm the settings on the page and click "Launch"

The cloud starts deploying your virtual server and you get the following message

In a minute your instance should be up and running. Let's locate it and login to it. Clicking the "Instances" link, then clicking our only instance so far and locating its "Public DNS" entry setting allows us to ssh to it

Now we're all set, let's jump to our terminal to ssh into the instance. It seems the "ubuntu.pem" key-pair the browser downloaded gets by default permissions that are too open for ssh's taste. Thus we need to "chmod" it to 700, this is the part where I lied about not using any CLI, but hey chmod doesn't really count ;) let's then ssh straight into our instance using the ubuntu.pem key-pair

Awesome! We're in! let's do what the greeting message tells us to "sudo tasksel --section server" and choose to install a LAMP Server. We get asked for a MySQL password twice, the system installs and configures a full LAMP stack for us. Let's visit the apache web server

That concludes this graphical guide. As you can see launching your first Ubuntu server instance in the cloud can't really get much easier than this :) Before you go for a cup of coffee, do not forget to "Terminate" your instance. If you don't, you keep on getting billed by the hour for it.

Let me know in the comments if a step was unclear. Also, let me know if there are other topics you'd want me to demo. If you're interested in running Ubuntu in a cloud context and have any doubts, drop by on IRC channel #ubuntu-cloud on freenode, and grab me "kim0"

Cloud on Cloud, UEC on EC2

2010-10-22T19:14:00.003+02:00

So you wanted to play with Ubuntu Enterprise Cloud (UEC), but didn't have a couple of machines to play with ? Want to start a UEC instance right now, no problem. You can use an Amazon EC2 server instance as your base server to install and run UEC on! Of course the EC2 instance is itself a virtual machine, thus running a VM inside that would require nested virtualization which AFAIK wouldn't work over EC2. The trick here is that we switch UEC's hypervisor temporarily to be qemu. Of course this won't win any performance competitions, in fact it'd be quite slow in production, but for playing, it fits the bill just fine.

If you're thinking doing all that is gonna be complex, you have a point, however it won't! In fact it'll be very easy due to efforts of Ubuntu's always awesome Scott Moser. Scott has written a script that automates all needed steps. But wait, it gets better, we're not even going to run this script ourselves, we're passing it as a parameter to the EC2 instance invocation and due to Ubuntu's cloud-init technology, that script is going to be run upon instance boot-up, doing its work automagically. Now let's get started

On your local machine, let's install bzr and get the needed script

cd /tmp
sudo apt-get install bzr -y
bzr branch lp:~smoser/+junk/uec-on-ec2
cd uec-on-ec2/

The file "maverick-commands.txt" contains the script needed to turn the generic Ubuntu image on ec2 into a fully operational single-node UEC install. If you don't have "ssh keys" (seriously?) generate some

ssh-keygen -t rsa

Now let's do a neat trick to import the keys into EC2, marking them with the name "default"

for r in us-east-1 us-west-1 ap-southeast-1 eu-west-1; do ec2-import-keypair --region $r default --public-key-file ~/.ssh/id_rsa.pub ; done

Let's open a few needed ports in EC2's default security group

for port in 22 80 8443 8773; do ec2-authorize default -p $port ; done

Very well .. We now launch our EC2 instance, passing in the "maverick commands" file. What happens is, the server instance is created, booted, Ubuntu's cloud-init reads up the maverick commands script we passed to it, and executes it, it starts downloading, installing and configuring UEC in the background while you ssh into your new EC2 instance

ec2-run-instances ami-688c7801 --instance-type m1.large -k default --user-data-file=maverick-commands.txt

Give it a minute to boot, then try to ssh in

ec2-describe-instances
ssh ubuntu@ec2-a-b-c-d.compute-1.amazonaws.com

Replace the DNS name for the EC2 instance, with the proper one you get from ec2-describe-instances. Once logged into the EC2 instance, I would start byobu and tail the log file to monitor progress. The whole thing takes less than 5 minutes

byobu
tailf uec-setup.log

The script once finished configuring UEC, actually downloads a tiny linux distro and registers its image in UEC, so that you can start your own instances! You know the script has finished when you see a line that looks like

emi="emi-FDC21818"; eri="eri-53721963"; eki="eki-740D19EC";

UEC is now up and running, let's check the web interface! You login with the default credentials admin/admin

Let's navigate to the Images tab, to get the EMI ID (Equivalent of an AMI ID)

Is that cool or what, Hell Yes! Now let's start our own VM inside UEC that's inside EC2. Remeber to replace emi-FE03181A with the EMI ID you got from the web interface

euca-run-instances --key mykey --addressing private emi-FE03181A

You can use "euca-describe-instances" to get the new instance internal IP address and ssh to that

ubuntu@domU-12-31-38-01-85-91:~$ ssh -i euca/mykey.pem ubuntu@172.19.1.2
The authenticity of host '172.19.1.2 (172.19.1.2)' can't be established.
RSA key fingerprint is db:9b:47:a4:06:81:26:d7:cf:38:a4:0e:6c:05:54:0d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.19.1.2' (RSA) to the list of known hosts.

Chop wood, carry water.

$ uname -r
2.6.35-16-virtual
$ df -h                                                                                                                                                                                                                                     
Filesystem                Size      Used Available Use% Mounted on
/dev/sda1                23.2M     14.1M      7.9M  64% /
tmpfs                    24.0K         0     24.0K   0% /dev/shm

et voila, you're ssh'ed into a ttylinux instance running inside qemu managed by UEC running over EC2 :) If you do find that cool, what about contributing back. How about you start hacking on that script to make it even more awesome, such as maybe installing over multiple nodes or whatever crazy idea you can think of! If you're interested to start hacking, drop me a hi in #ubuntu-cloud at Freenode

Free Ubuntu Server for a year at Amazon

2010-10-22T13:40:00.002+02:00

Yes, you can get your very own Free Ubuntu server in the clouds for one full year! The folks at Amazon have just announced

"Beginning November 1, new AWS customers will be able to run a free Amazon EC2 Micro Instance for a year, while also leveraging a new free usage tier for Amazon S3, Amazon Elastic Block Store, Amazon Elastic Load Balancing, and AWS data transfer. AWS’s free usage tier can be used for anything you want to run in the cloud: launch new applications, test existing applications in the cloud, or simply gain hands-on experience with AWS"

You can find the details of the offer at: http://aws.amazon.com/free/

Ubuntu being arguably the most popular Amazon guest image is also available to you for free today! Get the list of Official Ubuntu images created by the Ubuntu's very own server team at

Maverick http://uec-images.ubuntu.com/server/maverick/current/
Lucid http://uec-images.ubuntu.com/server/lucid/current/

For the free offer, you will want to launch a t1.micro instance, and it seems you will want to wait till Nov 1st for registering your account (credit card needed). This is great news! If you ever wanted to play with Ubuntu server on the cloud, now is the best time to get started!

Update: The terms page mentions "Only accounts created after October 20, 2010 are eligible for the Offer"

Update2: Please read Scott's comment below, on why you will be charged 0.5$/mo if you run the standard Ubuntu image (or Amazon's own AMIs)

Hello Planet-Ubuntu

2010-10-22T09:43:00.004+02:00

Hello World! I just became an Ubuntu member yesterday, which means I will no longer be on mute ;) I'm working as the Ubuntu Cloud community liaison, which means I finally bring some cloud and server content to this space instead of all being around fluffy desktops :) If you're interested in using or contributing to Ubuntu in a "cloud" context, feel free to grab me (kim0 on Freenode #ubuntu-cloud). Feel free to drop in, say hi, ask questions or whatever you need. Besides this blog, you may also be interested in my tweets. I'm very proud being part of this great community. Rock on!

OpenWeek Introduction to Cloud IRC session logs

2010-10-15T13:20:00.004+02:00

If you couldn't attend the Ubuntu Open Week session live on IRC, don't despair! You can read the logs at
http://irclogs.ubuntu.com/2010/10/13/%23ubuntu-classroom.html#t17:01

what I do

2010-09-20T11:07:00.003+03:00

I started with Linux and Open source software around 1999, since then and till today I was amazed about the helpful community that form around different open source projects. In the early days, whenever I'd hit a bug or don't know how to do something, I'd jump over to IRC and someone was always there to help debug the problem! This just felt amazingly empowering, I remember thinking to myself, hell this is way much better than what you get with commercial software! In a few years, I transformed from being a passive user, into someone who tries to help IRC users every time someone has a problem. The culture of open source got me I guess. I started giving presentations at Universities, writing for local IT magazines for free to help spread what FOSS is all about. I always wanted to contribute much more to the free software world, however limited free time available was always an issue. Now that my day job is to work with and help grow the community around Ubuntu, I feel extremely excited and thankful for Canonical for giving me this opportunity.

Being the newest member of the "horsemen" team having joined a little over a month now, I feel like I haven't done enough yet. Nevertheless, I'd like to mention a few of the things that have been keeping me busy the past few weeks

The very first thing I had done with Canonical was to write the Ubuntu Server Map application. The idea behind it is to encourage and spur the community behind Ubuntu server. Basically the aim is to make Ubuntu server users feel part of a huge user community all over the world. What the application does is to detect the visitor user's city using his IP address after the user accepts (anonymous process) and then marks that city with a cute little orange Ubuntu logo. So far the Map is full of orange Ubuntu logos. It really feels great to be part of such a world wide community of Ubuntu server and cloud users and contributors
As part of helping the Ubuntu cloud community grow around open source cloud technologies, I have focused on consolidating any fragmented communication paths available. Thus far, the Ubuntu cloud community has the following #ubuntu-cloud as the official IRC channel for everything Ubuntu cloud related. And the recently created Ubuntu Cloud Forum as the official Ubuntu cloud forum. The Ubuntu Cloud mailing list is an alternate community communication venue as well
Recently I've been putting a lot of focus on creating a Ubuntu cloud portal, which aims to be a central hub for the Ubuntu cloud community. You can read all about the portal specs and give me feedback over IRC (kim0 in #ubuntu-cloud). The portal should provide a list of rolling news that relate to Ubuntu cloud, helping interested community members always be on top of all the new happenings. It also helps community new comers by becoming their one stop shop with links to all documentation and support channels. As well as guide new contributers on how to get involved
Something which I am thinking about and which will definitely take a lot of focus soon is studying potential hurdles in the way of new contributers to Ubuntu server and cloud. Basically how to make it easy and more fun for newcomers to join in, find all the information they need in place and start contributing and engaging with the community. Part of that is giving tutorials over IRC or other mediums as well as sponsorship and guidance along the way

It is definitely great being part of such a great community such as the Ubuntu one, and I hope the next period is going to be very exciting for the open source Cloud communities in general

Why our Internet2.0 is broken

2010-09-06T16:24:00.007+02:00

The modern Internet which I'll refer to as Internet2.0 is being seen as a new applications platform. It is no longer a series of "pages" that you click through, it is rather a collection of applications. I just needed this intro in case you still thought of the Internet as pages. So the Internet is now the Operating System, and different web sites (Gmail, Facebook, Twitter...etc) are the new applications if you will. I've got news for you, this Internet2.0 thing, is horribly broken! Here is why

To begin understanding the kind of problems we have to go through using the online systems of today, let us apply the same online mechanisms, to standard old and boring desktop apps. Let's imagine the following workflow, You are 3 friends working on an important report, and you all share your progress online, each person via his blog

You login to your computer, you start your email application. You find your friend has edited the report, emailed you the new copy and blogged about his progress

You check your email, find the attachment, you download that

You must copy the attachment over a USB stick in order to be able to open it in any other application. You copy it to the USB stick

You open your word processor. You authenticate to it!

You plugin your USB stick, open the document, edit it, save it

Re-copy it to the USB stick

Re-open your email application, upload the attachment, send it

You visit your first friend's blog, you add a comment that you updated the report.

Your third friend is not notified of your comment

You get the idea. If that user experience sounds horrible, it is pretty similar to what we face today with web applications especially if you want to use different applications from different providers in concert. It is just plain broken. Here is my criticizm spot on

Why do I have to login separately to each and every web application (Google, Facebook, Zoho, Twitter, MS...)? On my Linux PC I don't have to go through that

Why do I have to teach each web application my social graph. Reconnect to all my friends on facebook, then twitter, then Google Buzz...? The connectivity between me and my friend belongs to us, it does not belong to Facebook. Any other app on the Internet which I allow to access this data, should be able to. It should not be held captive by the likes of Facebook

Assuming I love to use a live.com email address (which I don't :) and like to use Google docs to edit my email attachments. Why do I have to download the attachment to my PC first, reupload to Google to edit, save, download, upload, email, yikes! The Unix architecture designed 10s of years ago, was all about sharing data between apps (pipes), why in this modern age are we unable to easily connect different webapps especially from different providers

When I write a comment on my friend's blog, the comment is MINE. I created it, I own it. My friend's blog maybe displaying it currently, but it should not own it! If my friends are interested in seeing my comments on every website I visit, and if I allow them to, they should be able to do so very easily. The 20 comments I've written today, should not die if the websites I have written them onto decide to die

The same actually goes for using online editors a la Google docs. Google should not "own" the document. The document is mine. It should be stored in a place I control. I may "Allow" Google web based editor to read/write it now, because I "choose" to. Because it maybe the best editor around. Not because I have to, and not because I need to "migrate" my data off of Google should I want to use something else. If I decide to use Zoho editor tomorrow, I should be able to allow it to access my data in-place. Just like how on the desktop one can use MS Office, OpenOffice and Mac Pages to open a presentation on one's desktop

As a new application pops up, say like Apple's new Ping service, you need to (again) rebuild your social graph teaching it all your friends. Afterwards any music you purchase will show on Ping, but won't really show up on Facebook because Apple and Facebook couldn't agree on that. Can you say that again, "I" purchased a music track, and I want to tell "MY" friends about it, and I can't because Apple and Facebook couldn't agree! Can you see how horribly broken our Internet2.0 is

How I see things could improve is as follows. Each user needs to own a certain web exposed storage space somewhere. All of my online trails (documents, friendship, Likes, comments, blogs...) should live there. I should "allow" Google docs to read/write to those document if I like it. Tomorrow I could revoke that access and allow Zoho for example or any other online editor. My connections to my friends should be stored in that storage space. Any application that I allow can see who all or some of my friends are. For example I can allow Linkedin to access my work friends, while allowing Ping to access my music friends. If I make a comment somewhere on the Internet, this is content I created, If I Facebook style "Like" something, it is again content that revolves around me. It should be stored in my storage area, and Facebook should be notified of it and allowed to display it should I want to. This architecture makes it quite easy for a new young startup to create a Facebook or Gmail killer tomorrow. Since the application from day one will have access to all my data, emails, friends, comments, blogs...etc i.e. as much content as I want to give it. We become no longer locked into different online service providers. We can switch at will. This is the way how it can and should be done. The diaspora and FreedomBox folks are doing some awesome work and designing their systems along similar lines of thought. They are however more ambitious, and are after user data confidentiality, while all I'm asking for is open-data access and portability. They want to replace all of today's web applications with distributed clones that you can run inside your home. That would be fantastic, however I still it as advantageous and easier if a user can use "closed" cloud apps like Gmail or Facebook should she want to, and switch to a different provider at will, while still owning her online digital trail and every piece of content she ever created

Ubuntu Server 10.04.1 virtual release party

2010-08-09T11:38:00.001+03:00

Ubuntu Server LTS 10.04.1 is targeting to be released this Thursday on the 12th of August. This point release is especially important for the "server" variant of Ubuntu. This is because many conservative sys-admins wait for the first point release before deploying a server OS. The idea being, any bugs the fine QA people at Canonical might have missed, would be caught by the community during the period between the GA and point releases. This means that for many, this next 12th of August is the "actual" release date for Lucid server LTS! In short, on the 12th Ubuntu Server 10.04.1 is ready to rock the world!

In order to celebrate this awesomeness, we'll be having our own little server release party in the clouds :) What a mouthful, well basically I've put up a nice little web application that tracks all cities around the world that are running Ubuntu Server 10.04. Of course you will need to hit that web app with your server box first so that your city becomes registered and shows up on the map! So if you're already running 10.04 server, be sure to hit that application. If you're not, then you should be! Go grab yourself the 10.04 server iso, install it, then hit the app with it

But wait a minute, you say you don't have a web browser on your server box. No problemo. I'll list a few cool ways (well in a serverish way at least) to hit that web app without leaving your comfortable bash shell! Hey community, if you can think of cooler CLI methods, be sure to put them in your comments. if something is very cool, I'll add it to this post. Here we go


1- elinks --dump http://maps.ubuntu.com/hit/

2- curl http://maps.ubuntu.com/hit/

3- wget http://maps.ubuntu.com/hit/

4- python -c 'import urllib; urllib.urlopen("http://maps.ubuntu.com/hit/").read()'

5- /bin/echo -e 'GET /hit/ HTTP/1.1\nHost: maps.ubuntu.com\n\n' | socat - tcp:maps.ubuntu.com:80

Well I guess that should be enough to wet your appetite. Of course "technically" you don't have to visit the web app using your server box. Hitting it from your Ubuntu desktop has the same effect, but where's the fun in that! If you're running Ubuntu Servers in multiple cities, you are encouraged to hit that app from all cities in which you operate. If you coordinate the hit with some cool stuff (maybe a puppet recipe or bounce over ec2 or something of that sort) let me know about it :)

Whichever CLI way you choose to hit the app with, you'd be missing out on seeing lots of cute little Ubuntu logos over a world-wide Google map. Now that is something you wouldn't wanna miss, so be sure to visit the app from your graphical browser as well to view the list of marked cities running Ubuntu Server. On release day (12th of Aug) it would be very cool to have every major city in the world marked on that map, so please spread the word as much as you can (yes that means retweet it, digg it, slashdot it, tell you mom about it ...etc)

bash oneliner, get GPS location + street address

2010-08-03T23:08:00.004+03:00

Hey there folks,

It's been quite some time since I last blogged, just been busy with $reallife. Today I'll show you how to an answer the eternal question of "Where am I" from the comfort of your bash shell

The following code only works if:
- You run it in a root shell (Ubuntu users do: "sudo -i" then paste it)
- You are connected via Wifi to some access point
- Your wireless adapter is called wlan0 (otherwise replace it with the correct name)
- You're using a Linux system or something similar (i.e. Windows won't really work here)

et voila


/bin/echo '{"version": "1.1.0","host": "maps.google.com","request_address": true,"address_language": "en_GB", "wifi_towers": [{"mac_address": "' $( iwlist wlan0 scan | grep Address | head -1 | awk '{print $5}' | sed -e 's/ //g' ) '","signal_strength": 8,"age": 0}]}'  | sed -e 's/" /"/' -e 's/ "/"/g' > /tmp/post.$$ && curl -X POST -d @/tmp/post.$$ http://www.google.com/loc/json | sed -e 's/{/\n/g' -e 's/,/\n/g'

I didn't really spend much time cleaning it up, since I'm busy, so that's like the first thing that worked. If any of you guys can skip writing to the file, let me know and I'll update it

PS: If the returned information is wrong, or some kind of "unknown" .. Consider yourself lucky, very lucky! That means Google does not (yet?) know where your wifi AP is. For the rest of us .. tin-foil all the way

Android map navigation in Cairo

2010-04-10T22:20:00.002+02:00

woot! Android map navigation working in Cairo now! See how it works in the following animated sequence. A few cool things to notice:

1- It can start navigation from my location, which it knows by cell tours or GPS
2- I don't write my destination, I speak it out, it travels to google servers, gets recognized and comes back as text
3- It searches local areas, to identify the exact destination
4- then provides turn-by-turn instructions .. woohoo .. the future is now

Now the negatives:
1- No voice instructions .. you actually have to read ;)
2- The choice of routes, is not always what would make sense to me. I'll try to see how can I make it change its routing decisions

Thanks Google

Taming Xen Cloud Platform Consoles

2010-01-01T19:30:00.008+02:00

At $dayjob I have been studying Xen Cloud Platform inside out, and all in all I like it a lot! The API rox, it's very extensive and after the first couple of days makes a lot of sense. We're also see'ing initial very good performance. One part that was particularly not straightforward was the getting a console for a running VM. I'll document here my findings

So, let's jump straight in, let's list the console of our centos VM


[root@xcp03 ~]# xe console-list vm-uuid=90e68b99-0408-3e4c-1dd1-ea28e98fbbad
uuid ( RO)             : e845ef9d-2075-0773-119e-08875fb61a1f
          vm-uuid ( RO): 90e68b99-0408-3e4c-1dd1-ea28e98fbbad
    vm-name-label ( RO): kamal-CentOS-5.3x64
         protocol ( RO): RFB
         location ( RO): https://10.100.170.12/console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9

The "location" you get is the location of the console for the specified VM. However, if you try visiting that with a VM, you'll get a 404 not found error! Time for some magic. In order to do any of the following, we need a "session" on the Xen Cloud. Since I play with the Xen Python API, I'll use that to get a session


In [7]: import XenAPI

In [8]: session = XenAPI.Session('http://10.100.170.12')

In [9]: session.login_with_password('root', 'woohoo')

In [10]: print session._session
-------> print(session._session)
OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb

So we now have a session cookie "OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb". Now let's try to connect to the VNC server. I'll first try to connect by hand (actually telnet) :)

Try 1: Telnet connection

We need to connect using SSL, which telnet does not support. For that we use "socat", a magical little tool


[akamal@matrix tmp]$ socat TCP4-LISTEN:31337,fork OPENSSL:10.100.170.12:443,verify=0

Let's try to telnet to the locally listening port "31337" and socat will take our connection, wrap it in SSL and connect us to the Xen-API server


[akamal@matrix ~]$ telnet localhost 31337
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
CONNECT /console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb HTTP/1.0

HTTP/1.1 200 OK
Connection: keep-alive
Cache-Control: no-cache, no-store

RFB 003.003

Et voila, all the VNC savvy readers will understand that we have been connected to the VNC server. The RFB greeting denotes the begining of the VNC protocol. A few things to note

1- The CONNECT HTTP method is one big line
2- It contains two parts, the console uuid we got first, the second part being "&session_id=" and the session uuid we got from python
3- We connected with HTTP/1.0 and got a response with HTTP/1.1

Try 2: XVP Connection

Create a custom java key store. Download and convert the certificate format and import it. This is important because if you don't the java viewer won't accept the Xen server's self signed certificate (yuck?)


cd /usr/java/jdk1.6.0_17/bin
./keytool -genkey -alias kimo -keyalg RSA -keystore /tmp/kimo.jks
[akamal@matrix bin]$ scp root@10.100.170.12:/etc/xensource/xapi-ssl.pem /var/tmp/
xapi-ssl.pem                                                                                                               100% 1108     1.1KB/s   00:00
[akamal@matrix bin]$ openssl x509 -in /var/tmp/xapi-ssl.pem -inform PEM -out /var/tmp/newxapi.crt -outform DER

[akamal@matrix bin]$ ./keytool -import -trustcacerts -file /var/tmp/newxapi.crt -alias XCP_ALIAS -keystore /tmp/kimo.jks
Enter keystore password:
Owner: CN=10.100.170.13
Issuer: CN=10.100.170.13
Serial number: bb01ad6ac4cdc83e
Valid from: Wed Dec 23 15:27:24 EET 2009 until: Sat Dec 21 15:27:24 EET 2019
Certificate fingerprints:
         MD5:  84:90:D7:CF:7F:76:95:9E:2C:52:5A:66:C7:85:DB:58
         SHA1: BB:7B:C9:B1:47:19:42:F6:75:02:84:55:A1:05:67:7B:A6:B9:4E:82
         Signature algorithm name: SHA1withRSA
         Version: 1
Trust this certificate? [no]:  yes
Certificate was added to keystore

Perfect, now let's get the XVP source code. XVP is basically tightvnc with protocol additions that the Citrix hackers have added to better control VMs


[akamal@matrix tmp]$ wget -q http://www.xvpsource.org/xvp-1.3.2.tar.gz
[akamal@matrix tmp]$ tar xzf xvp-1.3.2.tar.gz
[akamal@matrix tmp]$ cd xvp-1.3.2/
[akamal@matrix xvp-1.3.2]$ cd viewer/

Apply the following patch



diff -ur ./HTTPSConnectSocket.java /tmp/xvp/xvp-1.3.2/viewer/HTTPSConnectSocket.java
--- ./HTTPSConnectSocket.java   2009-11-30 15:37:05.000000000 +0200
+++ /tmp/xvp/xvp-1.3.2/viewer/HTTPSConnectSocket.java   2009-12-29 19:14:56.165479016 +0200
@@ -46,8 +46,10 @@
     ssl = (SSLSocket)ssf.createSocket(proxyHost, proxyPort);
     ssl.startHandshake();

+    System.out.print("CONNECT " + host +
+                             " HTTP/1.0\r\n\r\n");
     // Send the CONNECT request
-    ssl.getOutputStream().write(("CONNECT " + host + ":" + port +
+    ssl.getOutputStream().write(("CONNECT " + host +
                             " HTTP/1.0\r\n\r\n").getBytes());

     // Read the first line of the response
@@ -55,8 +57,8 @@
     String str = is.readLine();

     // Check the HTTP error code -- it should be "200" on success
-    if (!str.startsWith("HTTP/1.0 200 ")) {
-      if (str.startsWith("HTTP/1.0 "))
+    if (!str.startsWith("HTTP/1.1 200 ")) {
+      if (str.startsWith("HTTP/1.1 "))
        str = str.substring(9);
       throw new IOException("Proxy reports \"" + str + "\"");
     }

Build the code


[akamal@matrix viewer]$ make
javac -J-Xmx16m -target 1.1 -source 1.3 -nowarn -O VncViewer.java RfbProto.java AuthPanel.java VncCanvas.java VncCanvas2.java OptionsFrame.java ClipboardFrame.java ButtonPanel.java DesCipher.java CapabilityInfo.java CapsContainer.java RecordingFrame.java SessionRecorder.java SocketFactory.java ReloginPanel.java HTTPConnectSocketFactory.java HTTPConnectSocket.java HTTPSConnectSocketFactory.java HTTPSConnectSocket.java InStream.java MemInStream.java ZlibInStream.java XvpConfirmDialog.java
jar -J-Xmx16m cfm VncViewer.jar MANIFEST.MF VncViewer.class RfbProto.class AuthPanel.class VncCanvas.class VncCanvas2.class OptionsFrame.class ClipboardFrame.class ButtonPanel.class DesCipher.class CapabilityInfo.class CapsContainer.class RecordingFrame.class SessionRecorder.class SocketFactory.class ReloginPanel.class HTTPConnectSocketFactory.class HTTPConnectSocket.class HTTPSConnectSocketFactory.class HTTPSConnectSocket.class InStream.class MemInStream.class ZlibInStream.class XvpConfirmDialog.class

Now let's connect using one more magical and undocumented command line


[akamal@matrix viewer]$ java -Djavax.net.ssl.trustStore=/tmp/kimo.jks -Xmx64m -jar VncViewer.jar HOST "/console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb" PORT 443 PROXYHOST1 10.100.170.12 PROXYPORT1 443 SocketFactory "HTTPSConnectSocketFactory"
Initializing...
Connecting to /console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb, port 443...
HTTPS CONNECT via proxy 10.100.170.12 port 443
CONNECT /console?ref=OpaqueRef:11519923-ee64-21cf-654f-8cce6a7edbb9&session_id=OpaqueRef:2d600568-c2ca-3f79-b54c-d98420fea1bb HTTP/1.0

Connected to server
RFB server supports protocol version 3.3
Using RFB protocol version 3.3
No authentication needed
Desktop name is XenServer Virtual Terminal
Desktop size is 640 x 384
Using Tight/ZRLE encodings
Closing window
Disconnecting
Updates received: 2 (40 rectangles + 1 pseudo), 0.02 updates/sec
Rectangles: Tight=0(JPEG=0) ZRLE=0 Hextile=40 Raw=0 CopyRect=0 other=0
Pixel data: 983040 bytes, 4295 compressed, ratio 228.88
RFB socket closed

et voila, here's how it looks when it works

Yoohooo, hurray for undocumented ^#^&@%^#%

This post would not have been possible without great help from a friend and guru developer Ahmed Soliman. Also Abdelrahman Hussein helped with some final touches

Hoping that post is of help to some poor souls. I still like XCP though :)

Create Mega Android Playlists from PC

2009-11-30T19:25:00.008+02:00

Wanting to create a large playlist for my HTC Hero android based phone, I found out this is a fairly painful task since the obvious method involves adding songs one by one!

Fear not fellow androiders, here is how to create mega playlists easily (although it does involve typing a command OMG :) )

1- Plug in the phone over USB. My phone appears as drive G:
2- Open a terminal window, and "cd" into the Folder containing the music you would like to make a "playlist". In my case


g:
cd MP3/80sMusic

3- Here comes the magic


dir /B > 80s-best-Music.m3u
exit

4- Et voila, the terminal window exits, safely remove your USB disk, turn off the USB connection from the phone and that is it!

Addendum for Linux users
1- Mount your phone on some directory, and "cd" to it. In my case


sudo mount /dev/sdb1 /media/androidy
cd /media/androidy

2- Create the playlist file


cd MP3/80sMusic
ls > 80s-best-Music.m3u

3- Eject and Et voila. It's really almost identical to the Windows case


cd / ; sudo umount /media/androidy

I was pleasently surprised to find that in Android ALL music apps share the same playlists. So that newly created playlist is seen by the built-in Music application, as well as with the 3rd party Music player MixZing. It is also pleasantly surprising that Android seems to scan the whole folder structure for the .M3U files wherever they may be and use them as playlists! Sweet

Taking down your CPUs

2009-11-19T12:06:00.001+02:00

echo 0 > /sys/devices/system/cpu/cpu1/online

Nov 19 11:59:35 localhost kernel: CPU 1 is now offline
Nov 19 11:59:35 localhost kernel: SMP alternatives: switching to UP code

Since my CPU is/was dual core, it's now single .. and thus the kernel has switched to UP code... neat !

Google Chrome Multiple Home Pages

2009-08-29T12:07:00.003+02:00

Recently I started using Google's browser Chrome (well technically chromium) on Linux for my browsing needs. It is very fast, low on CPU and Memory, I quite like it. I am now also sure Flash plugin is teh CPU hog pig :) Anyway, one really cool feature I missed from Firefox was the ability to set multiple pages as the homepage. The thing is, I follow quite a lot of tech news sites, and I need to check them every now and then. In firefox, I'd just click the Home button, and it would open all of them, I could not find this functionality out of the box with Chrome. What is a geek to do ? Well of course write some javascript :)

So, here is how I resolved this issue:
- Open a new tab, the bookmarks toolbar appears, right click it choose add page
- Choose a Name for the bookmark
- In the URL field, paste the following javascript


javascript:(function(){ window.location.href='http://site1.org/'; window.open('http://site2.com/'); window.open('http://site3.com/'); window.open('http://site4.com/'); })();

That is it :) Keep adding more sites as much as you want. Pretty straightforward, but I thought I'd blog it since it might not be all that obvious

8 sockets, 64 cores, 128 threads, Yummy!

2009-07-21T22:10:00.001+03:00

The Windows task manager almost looks funny! enjoy

English/Arabic Voice/Voice iPhone translator

2009-07-02T19:47:00.000+03:00

Amazing video! The future is now!

KDE 4.3 coming

2009-07-01T21:07:00.003+03:00

Network Scan for Conficker

2009-06-30T14:15:00.005+03:00

The conficker worm is a pretty nasty one! How do you scan if a machine on your network is infected ?

In this article we'll be using the awesome open-source nmap network scanning engine. In this example I will be preforming my scan from a Linux box. However you can use other OSs.


wget http://nmap.org/dist/nmap-4.90RC1-1.i386.rpm
rpm -Uvh nmap-4.90RC1-1.i386.rpm

It is recommended that you get the latest nmap from their download page. Next, launch nmap on the local network IP range, in this example it is 192.168.96.0/22, but YMMV. The interesting argument here is the --script which utilizes the NSE (Nmap scripting engine) in order to assess whether or not the current windows machine nmap is connected to is vulnerable. So let's launch the scan


nmap -PN -T4 -p139,445 -n -v --script smb-check-vulns,smb-os-discovery --script-args safe=1 192.168.96.0/22

Once the scan has completed Identify the potentially infected machine on the LAN. An infected machine would result in nmap output similar to


Host script results:
|  smb-os-discovery: Windows XP
|  LAN Manager: Windows 2000 LAN Manager
|  Name: WORKGROUP\LAME-WinXP
|_ System time: 2009-06-30 10:47:12 UTC+3
|  smb-check-vulns:
|  MS08-067: CHECK DISABLED (remove 'safe=1' argument to run)
|  Conficker: Likely INFECTED (by Conficker.C or lower)
|_ regsvc DoS: CHECK DISABLED (add '--script-args=unsafe=1' to run)

Once identified, you can use the great kkiller tool from kaspersky labs to clean the infected machine

You can find the latest kkiller from the following page page as well

Have fun, patch your broken windows, or better yet, learn some *nix foo or bar :)

OpenSolaris 2009.06 PV under Xen Centos5 dom0

2009-06-03T23:23:00.008+03:00

OpenSolaris 2009.06 has been recently released. Since ZFS offers a lot of flexibility and advantages as a storage platform, many would be interested in running the opensolaris stack. In this article I'll be installing OpenSolaris 2009.06 para-virtualized under a centos-5-x64 dom0. This setup has the advatange of running OpenSolaris on storage controllers that may not have native solaris driver support. Not that that would be recommended, however, it might be useful for testing. So, let's begin

- Install a minimal Centos-5-x64 installation. I installed the OS on a 20G partition, while the rest of the disk was formatted as LVM partition for creating logical volumes for xen guests to live on

- Install the latest Xen stack. I will be installing Xen-3.4. This is highly recommended since the bundled boot-loader pygrub has the ability to mount ZFS block devices.


[root@xen2 yum.repos.d]# cat > /etc/yum.repos.d/gitco.repo << EOF
> [gitco]
> name = Red Hat Enterprise $releasever - gitco
> baseurl = http://www.gitco.de/repo/xen3.4.0
> enabled = 1
> protect = 0
> gpgcheck = 0
> EOF

- Now let's install the required virtualization stack


[root@xen2 yum.repos.d]# yum install xen-libs.x86_64 bridge-utils.x86_64 qemu.x86_64 gtkglext-libs.x86_64 gtk-vnc.x86_64 gtk-vnc-python.x86_64 cyrus-sasl-md5.x86_64 libvirt.x86_64 libvirt-python.x86_64 python-virtinst virt-manager.x86_64 kernel-xen.x86_64 xen.x86_64 virt-viewer.x86_64 libvirt.x86_64 gnome-applet-vm.x86_64

Check that your menu.lst file has been updated to point to the newly installed Xen


[root@xen2 yum.repos.d]# cat /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You do not have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /, eg.
#          root (hd0,0)
#          kernel /boot/vmlinuz-version ro root=/dev/sda1
#          initrd /boot/initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-128.1.10.el5xen)
        root (hd0,0)
        kernel /boot/xen.gz-3.4.0
        module /boot/vmlinuz-2.6.18-128.1.10.el5xen ro root=LABEL=/ rhgb quiet
        module /boot/initrd-2.6.18-128.1.10.el5xen.img
title CentOS (2.6.18-128.el5)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.18-128.el5 ro root=LABEL=/ rhgb quiet
        initrd /boot/initrd-2.6.18-128.el5.img

- Reboot into your new environment, and check you are booted on Xen-3.4


[root@xen2 ~]# xm dmesg | head
 __  __            _____ _  _    ___
 \ \/ /___ _ __   |___ /| || |  / _ \
  \  // _ \ '_ \    |_ \| || |_| | | |
  /  \  __/ | | |  ___) |__   _| |_| |
 /_/\_\___|_| |_| |____(_) |_|(_)___/

(XEN) Xen version 3.4.0 (root@gitco.tld) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri May 29 21:39:26 CEST 2009
(XEN) Latest ChangeSet: unavailable
(XEN) Command line:
(XEN) Video information:

- Now we're ready to begin :) Make a folder and download the ISO in it


[root@xen2 ~]# mkdir -p /xen/osol/ && cd /xen/osol && wget http://genunix.org/distributions/indiana/osol-0906-x86.iso

- Let's loop mount the iso, extract the paravirt kernel and micro-root


[root@xen2 osol]# mkdir -p /mnt/loop
[root@xen2 osol]# mount -o ro,loop osol-0906-x86.iso /mnt/loop
[root@xen2 osol]# cp /mnt/loop/boot/amd64/x86.microroot /mnt/loop/platform/i86xpv/kernel/amd64/unix .

- Construct the Xen installation-time configuration file


[root@xen2 osol]# cat > osol0906install << EOF
> name = "osol0906install"
> memory = 1024
> vfb = [ 'type=vnc,vnclisten=127.0.0.1,vncdisplay=0' ]
> disk = ['file:/xen/osol/osol-0906-x86.iso,6:cdrom,r',
>  "phy:/dev/vgxen/osol,xvda,w" ]
> vif = [ "mac=00:16:36:10:f5:69,bridge=eth1" ]
> on_reboot = "destroy" # comment out this after installation
> kernel = "/xen/osol/unix"
> ramdisk = "/xen/osol/x86.microroot"
> extra = "/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom"
> EOF

- Create the logical volume for the OpenSolaris installtion


[root@xen2 osol]# lvcreate -L 15G -n osol vgxen
  Logical volume "osol" created
[root@xen2 osol]# ls -l /dev/vgxen/osol
lrwxrwxrwx 1 root root 22 Jun  3 22:14 /dev/vgxen/osol -> /dev/mapper/vgxen-osol

- Start the Xen VM, and hit enter a few times for the defaults


[root@xen2 osol]# xm create ./osol0906install -c
Using config file "././osol0906install".
Started domain osol0906install (id=1)
                                     v3.4.0 chgset 'unavailable'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: opensolaris
Remounting root read/write
Probing for device nodes ...
NOTICE: xdf@6: failed to read feature-barrier
NOTICE: xdf@51712: failed to read feature-barrier
Preparing live image for use
Done mounting Live image
USB keyboard
 1. Albanian                      23. Lithuanian
 2. Belarusian                    24. Latvian
 3. Belgian                       25. Macedonian
 4. Brazilian                     26. Malta_UK
 5. Bulgarian                     27. Malta_US
 6. Canadian-Bilingual            28. Norwegian
 7. Croatian                      29. Polish
 8. Czech                         30. Portuguese
 9. Danish                        31. Russian
10. Dutch                         32. Serbia-And-Montenegro
11. Finnish                       33. Slovenian
12. French                        34. Slovakian
13. French-Canadian               35. Spanish
14. Hungarian                     36. Swedish
15. German                        37. Swiss-French
16. Greek                         38. Swiss-German
17. Icelandic                     39. Traditional-Chinese
18. Italian                       40. TurkishQ
19. Japanese-type6                41. TurkishF
20. Japanese                      42. UK-English
21. Korean                        43. US-English
22. Latin-American
To select the keyboard layout, enter a number [default 43]:
 1. Arabic
 2. Chinese - Simplified
 3. Chinese - Traditional
 4. Czech
 5. Dutch
 6. English
 7. French
 8. German
 9. Greek
10. Hebrew
11. Hungarian
12. Indonesian
13. Italian
14. Japanese
15. Korean
16. Polish
17. Portuguese - Brazil
18. Russian
19. Slovak
20. Spanish
21. Swedish
To select desktop language, enter a number [default is 6]:
User selected: English
Configuring devices.
Mounting cdroms
Reading ZFS config: done.

opensolaris console login: jack
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
jack@opensolaris:~$  pfexec uname -a
SunOS opensolaris 5.11 snv_111b i86pc i386 i86xpv Solaris
jack@opensolaris:~$
jack@opensolaris:~$ ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
xnf0: flags=1004843 mtu 1500 index 2
        inet 192.168.96.121 netmask fffffc00 broadcast 192.168.99.255
lo0: flags=2002000849 mtu 8252 index 1
        inet6 ::1/128
xnf0: flags=2004841 mtu 1500 index 2
        inet6 fe80::216:36ff:fe10:f569/10

- Note that once the installation system boots up, we login with username/password = jack/jack. We use "ifconfig -a" to check the system's IP address (was configured by DHCP, VM NIC is bridged to physical host's eth1). The IP in this case is 192.168.96.121

- Now here comes a bit of a tricky part, the Xen console is text only, the OpenSolaris installtion is GUI only (yuck?) In order to get GUI access we will use a built in VNC service in the OpenSolaris Live CD. To get the password for the default vnc session, perform the following command on the Centos Host machine



[root@xen2 osol]# xenstore-ls | grep pass
     passwd = "guzc/VCg"

- Connect to the live CD and launch the installer


vncviewer 192.168.96.121:0

Continue with the installer GUI

- Once done, click Reboot on the installer. The VM actually powers off, which is what we've configured and what we need.

- Create the run-time Xen config file


[root@xen2 osol]# cat > osol0906pv << EOF
> name = "osol0906pv"
> memory = 2048
> vfb = [ 'type=vnc,vnclisten=127.0.0.1,vncdisplay=0' ]
> disk = [ "phy:/dev/vgxen/osol,xvda,w" ]
> vif = [ "mac=00:16:36:10:f5:69,bridge=eth1" ]
> bootloader = '/usr/bin/pygrub'
> EOF

- Start the VM and enjoy :)



[root@xen2 osol]# xm create -c ./osol0906pv
Using config file "././osol0906pv".
Started domain osol0906pv (id=2)
                                v3.4.0 chgset 'unavailable'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
NOTICE: xdf@51712: failed to read feature-barrier
Hostname: osol0906pv
Configuring devices.
Loading smf(5) service descriptions: 150/150
svccfg import warnings. See /var/svc/log/system-manifest-import:default.log .
Reading ZFS config: done.
Mounting ZFS filesystems: (6/6)
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair

osol0906pv console login: dev
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
dev@osol0906pv:~$ pfexec uname -a
SunOS osol0906pv 5.11 snv_111b i86pc i386 i86xpv Solaris
dev@osol0906pv:~$

Hope that's useful to someone out there

Traffic accounting with SNMP and Python on Windows

2009-05-30T14:00:00.018+03:00

If you're not on a flat-rate ISP data-plan, and your ISP made you pay extra last month such as myself :) You might be interested in monitoring your home bandwidth. A nifty little features of most routers or ADSL modems, is support for SNMP. Since both me and my old folks use the same ADSL line, any host based monitoring solution wouldn't really work. I had to monitor the usage using SNMP. Since my folks are using Windows, the solution had to work on that. The solution overview is:

1- Enable SNMP on ADSL modem
2- Use a windows scheduled task to run a windows Batch file, that uses SNMP-tools for windows to pull data from the router
3- Use a windows scheduled task to run a python script that processes the batch file output and computes the total traffic

Data from the batch files are written in a file signifying the current month (example 05.txt for May). Here is the code for the batch file




cd e:\traffic

e:\

FOR /F "TOKENS=1* DELIMS= " %%A IN ('DATE/T') DO SET CDATE=%%B

FOR /F "TOKENS=1,2 eol=/ DELIMS=/ " %%A IN ('DATE/T') DO SET mm=%%B



echo %mm%

snmpget.exe -c public -O v -v 1 192.168.11.1 .1.3.6.1.2.1.2.2.1.10.12 >> %mm%.txt

Note that the IP 192.168.11.1 should be replaced with the internal IP of your ADSL router. Also, the SNMP OID .1.3.6.1.2.1.2.2.1.10.12 (symbolically: IF-MIB::ifInOctets.12) is the incoming octets on network interface "12". You might need to change that last "12" to represent your network topology. You can use the "snmpwalk" command to list your interface names like:

snmpwalk -c public -v1 192.168.11.1
this yields
...
IF-MIB::ifDescr.12 = STRING: ppp0
...

I know that my ppp0 is the link to my ISP, so that's the interface I used

And here is the code for the python script



# -*- coding: utf-8 -*-

import glob

import locale

import os

locale.setlocale(locale.LC_ALL, '')

reports = sorted(glob.glob('??.txt'))

totalsReport = open('totals.txt','w')

totalsReport.write(' Month      Traffic \n')

totalsReport.write('====================\n')

for report in reports:

        datastring = open(report,'r').readlines()

        data = [ int(datastring[i].strip().split()[1]) for i in range(len(datastring)) ]

        diff = [ data[i+1] - data[i] for i in range(len(data) - 1) ]

        trafficDeltas = [ (data[i+1],diff[i])[diff[i] > 0] for i in range(len(diff)) ]

        totalTraffic = sum(trafficDeltas)

        totalsReport.write(' %s => %s \n'% (report.replace('.txt','') , locale.format('%d', totalTraffic, True)) )



totalsReport.close()

Glancing over the Python code, you can already see I'm a list comprehension addict :) The code reads the snmp bytes values from ??.txt (example: 05.txt), computes bandwidth deltas, handles counter resets (32bit overflow, or router loosing power) and computes totals. Then writes the total to a file named "totals.txt"

This solution is resistant to the router loosing power and resetting its internal traffic counter back to zero. This is because the data is continuously being saved to non-volatile medium (PC's hard-disk). Also, the Windows PC running those two scripts, does NOT need to be running 24x7 .. Just most of the time. If however the PC is powered off, AND the modem power cycles or hangs, you might loose some accuracy, the results will still be fairly valid though. Hope that's helpful to anyone out there

Google Wave, me likes

2009-05-30T01:55:00.001+03:00

Check this out! Email+IM+Wiki+RealTimeCollaboration+Documents+Workflows+ServerSideBots+GoogleMagic+OpenStandards = Google Wave

This is the "Wave" of the Future ;)

Waiting for Fedora 11

2009-05-27T20:17:00.001+03:00

Nexenta 2.0 is out

2009-05-25T23:18:00.000+03:00

Check out the googley style toons at
http://www.nexenta.org/os

VMs over ZFS rock

2009-05-25T23:12:00.002+03:00

when configuring some complex task over a server, sometimes things go bad (corrupt files, wrong versions, bad data in DB ... etc) and sometimes it's really not easy to undo everything such that you're sure the machine is in a known-good-n-clean state. This is where VMs over ZFS rock!

while self.isPatient():
ssh vm poweroff;
zfs rollback vm@someTimeWhenThingsWhereGood
VBoxManage startvm vm
self.letsTryAgain()

Hyper cool! Snapshots and ease/reliability of backups/restores are about one of the most important things why I love virtulization.

Killing Virii with Gentoo and Kaspersky

2009-05-20T19:24:00.006+03:00

Sorry but if you pronounce kaspersky to rhyme with whisky, the title isn't going to sound as jiggly as it should ;) I used to pronounce it to rhyme with "sky" but apparently that's incorrect. Anyway, I've been a long time admirer of the Kaspersky AV engine. And today it has saved someone's a$$ one more time ;)

A friend of mine, who is pretty technical, well he maintains windows drivers for a living, so that sure makes him a hot-shot techy in anyone's book. That friend's laptop caught a nasty virus. In his own words, he was only downloading some power point presentations (ugh), when the miserable closed source proprietary OS he's running (euhm Vista) became infected. He was using Google's chrome browser, so the possibility of having been infected through a browser exploit remains pretty low in my opinion. Especially that Chrome auto-updates itself, thus fixing any potential security holes. My first impression was that he got infected through an exploit in MS Office 2007 (yuck). Anyway, with me trying to help him clean up the laptop, we tried the following

- Tried installing Symantec's AV suite. That totally fails to even install. What a piece of crap. Symantec's ware is highly over-rated IMO. I used to really like Norton stuff, back in the days of Norton's DiskDoctor .. those were the days :D

- Tried installing the tried and true MalwareBytes, which did detect and clean a whole bunch of malware, however, much to my surprise, the problem persisted. MalwareBytes is a cool piece of anti-malware, it has worked fantastically for me multiple times, but this time it wasn't enough!

- Having wasted a couple of hours on this already, I wanted to fire some Kaspersky power on the problem. I visited http://devbuilds.kaspersky-labs.com/devbuilds/RescueDisk/ and downloaded a Live CD image Burned the iso, and booted!

Now this Live CD is absolutely cool, it's a customized build of Gentoo linux (w00t!) that automagically detected the hardware, connected to the network, started an X server, launched a customized icewm environment with Kaspersky's "K" logo as the "start" button down below. I was impressed, and through that GUI I could launch Kaspersky's AntiVirus tool.

The first thing it did was to auto-update itself over the internet. Most definitely needed. Afterwards it located and mounted all Windows NTFS partitions, and I was presented with options to scan them. I chose to scan the c: drive. Scan has begun, the scan tool sports a nice looking GUI, although it can be a bit confusing. Anyway scanning has started churning on the hard-disk. It was a bit slow, took around 3 hours for a 100G c: drive!

But I'll sure take slow and reliable over anything else every time! At the end, Kaspersky has located hundreds of infected executable files. I chose to disinfect them. It started disinfection one by one. This took around 20 minutes or so as well. Rebooting after that, windows came up finally clean . The system is working normally again, sigh!

All in all, Kaspersky proved to be a reliable tool. Kudos to their team for providing a top notch Linux based Live CD for free, that updates itself and provides adequate disinfection for free. Thank you Kaspersky. I will surely recommend you guys in the future. This is one AntiVirus tool I will be sure to remember, when a friend comes knocking on my door. Note however, that they're not the only game in town, others like Avira, and BitDefender and others as well offer Live CD "rescue-disks" as they are called. Hope this post helps anyone out there.

Why We Hate Micro$oft

2009-04-22T23:06:00.003+02:00

Did you ever wonder, why "those people" hate Microsoft
If you have, and if you really honestly truly want to gain a better understanding, put down your MS lover hat, and feast your eyes on the following Master-Piece document produced by "The European Committee for Interoperable Systems". The document named "Microsoft - A History of Anticompetitive Behavior and Consumer Harm"

You can read the an introductory article + the full document (scroll down a couple of pages) at

http://www.groklaw.net/article.php?story=20090421111327711

Beware though, it's not a short read. It's more of a week-end type of thing. However, to wet your appetite, here are a couple of quotes

Killing Dr-Dos: The OS monopoly

"The approach we will take is to detect dr [DOS] ⁶ and refuse to load. The error
message should be something like 'Invalid device driver interface.'"
--Phillip Barrett, Microsoft Windows Development Manager

Killing Word-Perfect: The Office Monopoly

"I have decided that we should not publish these [Windows 95 user interface]
extensions. We should wait until we have a way to do a high level of integration
that will be harder for likes of Notes, WordPerfect to achieve, and which will give
Office a real advantage.... We can't compete with Lotus and WordPerfect/Novell without this."
--Bill Gates, Microsoft founder and then-CEO

Killing Java: The language Monopoly

"[W]e should just quietly grow j++ share and assume that people will take more
advantage of our classes without ever realizing they are building win32-only java
apps."
--Microsoft's Thomas Reardon

That's enough for a teaser :) Enjoy reading the rest of the document

Magical apt-clone, broken system recovery

2008-12-08T23:19:00.009+02:00

I have received some very nice feedback regarding my last blog on D-Bus, thanks guys. This week I'm going to play once more with my Nexenta installation that we completed in part1. This time around I am going to basically demo the integration the Nexenta folks have pulled mixing apt-get with ZFS file-system capabilities, resulting in transactional system upgrades, with multi-snapshots that makes system recovery a snap!

We start off booting our nexenta NCP2-alpha installation, which we'll be upgrading to beta-1 as we speak. Grub shows the default boot kernel

Once booted, and we log in, let's update apt's cache

I have also used the command "apt-clone" to list the currently available boot environments. Basically, using ZFS snapshots the whole machine state is snapshot'ed resulting in a instantly bootable environment. Currently we only have the default environment "rootfs-nmu-000". Now I am going to use apt-clone to "dist-upgrade" the whole system. If this is new to you (who has never seen Ubuntu :) it basically means you're upgrading the whole system to the latest tasty bits

Note that I have used verbose mode "-v" in order to get more information. After a lot of downloading and dpkg grinding, I now have the latest packages installed. apt-clone has created a new system snapshot "001" and the process ends like so

I have simply hit 'n' to discard any automatic changes, then I used apt-clone to activate the upgraded snapshot as instructed by the on-screen output

Now let's reboot and see what kind of coolness we have

Oh cool, Grub now displays two bootable environments! One before the upgrades and one after. The dates besides each entry make that clear. Great, now let's reboot into the new envrionment and list the available clones (cloned file systems)

Note that the "A" and "C" in the output mean "Active" and "Current". Great, now creating instant bootable system snapshots like that is so useful that it does not relate to software installations only. Say you're running some third party closed source installation application and you're not really sure what's it going to do to your system. Or if the new clueless administrator needs to install something to the server but you think he's probably going to destroy it! In such situations, it's very helpful to manually create a system snapshot. So, let's do just that to create the environment 002 and let's activate it

Now we have our own little safety net. Let's now boot into that new environment. Use apt-clone to make sure "002" is the active and current environment. Then's let's simulate complete system failure. I will not be too creative and will simply delete the kernel and reboot!

Once in Grub, boot the new and destroyed system. As expected Grub errors out and we're staring at

So right now we have a broken system. Now do we run to tape, dump | restore, and use some black magic, nope, we simply reboot and choose our safe snapshot 001, boot into that and destroy the corrupt environment 002 and activate the safe one 001 as so

Perfect! Once rebooted, our system is back fully functional. Mind you that taking system snapshots, destroying corrupt ones all take a couple of seconds, and of course does not consume disk space except for the changed disk blocks. Now that we have our server back, let's do something useful with it. Let's "apt-clone install apache2".

Once done, the SMF could not start apache because I had chosen nwam network auto-configuration (think NetworkManager), which apache2 does not like. Anyway, I simply started the SysV script manually, and verified "it works" with a direct telnet connection. You can basically start installing and configuring various server software mostly just like an Ubuntu system. All in all, the Nexenta-2 system is a very promising concept. The integration of advanced Solaris kernel features with GNU userland and attempting to port Ubuntu repos, would basically be paradise. It is however a lot of work. The system is not perfect yet, for example in this beta-1, you cannot install X yet (broken deps). That will surely be resolved by release time. However, rebuilding the huge debian/Ubuntu repos, needs a lot of work. If you have free time to push this interesting FOSS project forward, please do so. Visit nexenta and join their development team. Meanwhile and separately from nexenta, I have been interacting with various open source developers working with opensolaris projects. Unfortunately it seems that Sun is not really making it easy for "outsiders" to work on opensolaris projects. I have heard such comments multiple times, from different people. One would think that after so much time, Sun would start to understand what it takes to build a community around any piece of software! Let me use the moment to shout out: Sun, please cut the community some slack! :)

Linus explains how Linux started

2008-12-08T23:15:00.002+02:00

Nice vid

Ride the D-Bus, Control your Linux desktop from the shell

2008-11-29T23:06:00.009+02:00

What is the D-Bus?

From the D-Bus documentation: "D-Bus is an inter-process communication mechanism—a medium for local communication between processes running on the same host. (Inter-host connects may be added in the future, but that is not what D-Bus is meant for). D-Bus is meant to be fast and lightweight, and is designed for use as a unified middleware layer underneath the main free desktop environments"

So for the non technically inclined (why are you reading this anyway) ;) basically a simple way to think of D-Bus, is that it's a way for processes on an operating system to communicate with each other. That's probably a simple and non accurate enough wording, but it should help grasp the concept. Many utilities nowadays are implementing D-Bus connectivity. It is especially interesting (well for me anyway) to script some GUI elements like for example my KDE-4 desktop from the command line. This would help automate some tasks and is cool anyways. Let's see how to begin talking on the D-Bus

In order to communicate with running applications on the D-Bus, we need a front end CLI application, or we could use a language binding (say python bindings). However, we will choose a CLI application. It is possible to use "dbus-launch", or the "qdbus" command part of Qt4. I will be using qdbus since it demonstrates the point and is faster to use. So, let's launch qdbus, and list the running communications buses.

# qdbus
:1.1
org.kde.klauncher
:1.10
org.freedesktop.ScreenSaver
org.kde.krunner
org.kde.screensaver
:1.12
org.kde.plasma
:1.1249
... List continues

So basically, qdbus now contacts the running session D-Bus and lists available buses. The buses are named in a reverse DNS style names as you can see. Some buses do not have names, but rather numbers, as far as I can see those represent specific sessions of some applications. Now we need to pick an interesting bus to talk on. I will run the following command to list only freedesktop dbuses, as I believe those should work whether you're running Gnome or KDE. So, it should help any reader follow along

#qdbus | grep freedesktop
org.freedesktop.ScreenSaver
org.freedesktop.Notifications
org.freedesktop.PowerManagement
org.freedesktop.DBus

so, let's pick the ScreenSaver bus. In order to list the available objects on any specific bus, we call it like so:

#qdbus org.freedesktop.ScreenSaver
/
/App
/Interface
/KBookmarkManager
/KBookmarkManager/konqueror
/KDebug
/MainApplication
/ManagerIface_contact
/ScreenSaver

I mostly find only one "useful" object, in this case "/ScreenSaver". Let's contact that object, and list its avialable "methods". The methods of an object for those who have not done any object oriented coding before, is basically the list of actions or "things" this object can "do". So listing the methods:

# qdbus org.freedesktop.ScreenSaver /ScreenSaver
signal void org.freedesktop.ScreenSaver.ActiveChanged(bool)
method bool org.freedesktop.ScreenSaver.GetActive()
method uint org.freedesktop.ScreenSaver.GetActiveTime()
method uint org.freedesktop.ScreenSaver.GetSessionIdleTime()
method uint org.freedesktop.ScreenSaver.Inhibit(QString application_name, QString reason_for_inhibit)
method void org.freedesktop.ScreenSaver.Lock()
method bool org.freedesktop.ScreenSaver.SetActive(bool e)
method void org.freedesktop.ScreenSaver.SimulateUserActivity()
method uint org.freedesktop.ScreenSaver.Throttle(QString application_name, QString reason_for_inhibit)
method void org.freedesktop.ScreenSaver.UnInhibit(uint cookie)
method void org.freedesktop.ScreenSaver.UnThrottle(uint cookie)
method void org.kde.screensaver.configure()
method void org.kde.screensaver.saverLockReady()
method QDBusVariant org.freedesktop.DBus.Properties.Get(QString interface_name, QString property_name)
method void org.freedesktop.DBus.Properties.Set(QString interface_name, QString property_name, QDBusVariant value)
method QString org.freedesktop.DBus.Introspectable.Introspect()

Hmm, interesting stuff. Can you see the following method:
org.freedesktop.ScreenSaver.GetSessionIdleTime
Obviously it gives you the session idle time, i.e. how long has the interactive user not touched his keyboard or mouse. This piece of information is interesting if you want your shell script to only do certain things when the user is not interactively using his machine. Without D-Bus, getting this kind of information would be almost impossible, or too tricky. Now how do we call this method you ask? easy! just append it to the command line:

#qdbus org.freedesktop.ScreenSaver /ScreenSaver org.freedesktop.ScreenSaver.GetSessionIdleTime
0

So, the output from this command is a "0". Meaning the session idle time is zero! Everytime you execute this command, you get the same answer. Starting to guess why ? :) Basically, as you hit "enter" to run this command, you reset the session idle time back to zero! So, what do we do to test the functionality, try this:

# sleep 5 ; qdbus org.freedesktop.ScreenSaver /ScreenSaver org.freedesktop.ScreenSaver.GetSessionIdleTime
4

So, that previous combo command sleeps for 5 seconds .. then executes the D-Bus query without touching any further keyboard keys, so this time we get an answer of "4" seconds. Don't know where that extra second slipped though :)

Ok, as a second example let's focus on the following method:
method bool org.freedesktop.ScreenSaver.SetActive(bool e)

This method sets the screen saver active, i.e. it launches the screen saver from your shell command or script. Let's test it out:

# qdbus org.freedesktop.ScreenSaver /ScreenSaver org.freedesktop.ScreenSaver.SetActive
Error: org.freedesktop.DBus.Error.UnknownMethod
No such method 'SetActive' in interface 'org.freedesktop.ScreenSaver' at object path '/ScreenSaver' (signature '')

Oops, this time we got an error. Why? Basically the error is telling us that it cannot find that method we're calling with that "signature". A method signature usually means the arguments you pass to it and the return type. This simply means we're not calling the method the way it's meant to be called. The sharp shooters are going to instantly know this is because we missed the (bool e) at the end of the method. This means the method call is expecting a boolean (true/false) argument. So, let's call the screensaver method correctly this time:

# qdbus org.freedesktop.ScreenSaver /ScreenSaver org.freedesktop.ScreenSaver.SetActive True
true

Cool! We're now launching the screensaver from the command line. Let's see what else we can do:

# qdbus org.freedesktop.ScreenSaver /ScreenSaver org.freedesktop.ScreenSaver.Lock

Yep, that locks the screen on demand, cool! I'm hungry for more. Let's see. Let's call that inhibit method. This will basically inhibit the screensaver from kicking in, even if your computer is idle. Why would that be useful ? Say you're watching a You-tube longish video, and AFAIK, Flash doesn't yet communicate that to the system, and thus the screensaver will kick in to interrupt the video playback. So, let's do it:

# qdbus org.freedesktop.ScreenSaver /ScreenSaver org.freedesktop.ScreenSaver.Inhibit "$$" "Testing D-Bus Interface"
5822

Let's explain those arguments we passed, the first one should be the application name, in my case, the special shell variable $$ is the PID for the currently executing shell. The second string is the "reason for inhibit"! The method call returns a cookie, i.e. a magic number that identifies my request. This is useful when you want to turn off the inhibit, i.e. return the system to its normal state. You would need to pass back that magic cookie. This is used by the system to identify the different inhibit requests. So finally let's uninhibit our screensaver using that cookie number

# qdbus org.freedesktop.ScreenSaver /ScreenSaver org.freedesktop.ScreenSaver.UnInhibit 5822

Cool! Are you starting to see the possibilities just yet? Hope you had fun on this D-Bus tour

ZFS a smash hit

2008-11-24T18:58:00.001+02:00

Pretty freaking amazing