Bash script self-updates from github

Recently the systems engineering team @ Cloud9ers ($dayjob) has been busy building the infrastructure for a very ambitious and large scale project for one of our customers. The project involves tons of distributed programming besides lots of systems work as well. I'd like to share with the community a small tidbit of information. It's often some small things the one sometimes gives a moment to admire, and hereby share!

The problem

• We are building a private cloud. Ubuntu server instances running on that are cloned from a master template
• While most clouds provide what I call "Instance Identity" information through some pre-known web service URI, in our case we utilize VMware's "Invoke-VMScript" API to run scripts inside the cloned template, thus customizing it and giving it its identity, then puppet takes it from there. Note that we're using a simple vCenter+ESXi (no vCloud stuff here) no shared storage even!

Ok the real problem

• Editing the template, and publishing it across the cloud (a topic for another post) takes considerable time! I wanted a way to be able to quickly update my identity scripts without having to re-build and re-publish images

Solution

• A script with a trivially simple (thus mostly fixed) "bootstrap" section, which auto-updates itself from github and relaunches its-new-self!

Code

Nothing ground breaking, but still a cool trick eh! Yeah I could have added a broken this into multiple scripts and maybe run-parts them. Of-course this script is quite basic, but it's meant to remain that way since the heavy lifting is puppet's responsibility!
Note: To run this successfully the user this is run under, should have passwordless sudo rights to run "ip" and to run "itself" :) Got cool ideas, thoughts or comments? Leave me a message

LXC lies about /.. inode number making FlexLM unhappy

One of our customers wanted to run a flexlm licensed tool on the Amazon EC2 cloud. He turned to us over here at Cloud Niners and we started doing the work. The approach we chose was to start an ec2 Ubuntu precise machine, host a CentOS LXC container on it and run the license daemon inside that. The reason for the CentOS choice is that the tools and license daemons are only certified on RHEL and CentOS is the closest next thing

Trying to start the license daemon, I hit the following error

 8:59:27 (TOOLNAME) Cannot open daemon lock file

Checking the usual suspicions of permissions and the like led no where. Google immediately led me to similar problems for people wanting to run Flexlm tools under Solaris zones (which are quite similar to Linux LXC containers). I was reading about a guy facing a similar problem, and how Solaris' legend Brendan Gregg wrote dtrace scripts to run-time patch memory structures to resolve the issue.

At first I was dismissive that this was what I was actually see'ing. A quick "ls -lai /" just to confirm /. and /.. actually had the same inode number, and I was sure it was something else

# ls -lai / | grep '\.$'

  256 dr-xr-xr-x   1 root root   212 Dec 26 09:08 .

  256 dr-xr-xr-x   1 root root   212 Dec 26 09:08 ..

but I had one of those mmm moments after stracing the binaries and confirming they are actually failing immediately after calling getdents on "/". This sounds suspiciously close to what the Solaris folks were see'ing. I grabbed gcc and built the sample code in getdents kernel doc page (Thankful!) And much to my surprise, inode numbers for /. and /.. were actually different!

# ./a.out / | grep '\.$'

     256  directory    24            1  .

   42669  directory    24          2  ..

Of course this makes sense since the LXC guest was just another directory on the host, but I didn't suspect ls -i was actually lying inside guests!

At this point, I'm not exactly sure how to resolve my issue apart from reinstalling the LXC guest on a separate block device (like lvm) which I think should resolve the issue. This blog post is simply to confirm the issue, and to gather feedback and potential solutions from smart people reading this. Shoot me a comment

Celebrating Cloud9ers is MEA's leading Amazon Consulting partner

It's been quite a while since I last blogged, I think it was over a year ago! I've been taking a break from blogging and focusing on Cloud Niners my baby startup that I co-founded with a bunch of life long friends.

While many things are interesting over here at Cloud Niners, this blog post is specifically to celebrate the fact that we've just been upgraded to become an Amazon Consulting Partner. I believe this makes us the first active middle-east Amazon AWS cloud consulting partner and certainly the best :) I do hear a lot from Amazon that we are the cloud leader in the middle-east! I am quite happy that we've achieved this milestone. I'm a big believer in cloud computing, and I simply enjoy helping our clients understand how the cloud (and Ubuntu server) can help them achieve more!

My team and I do custom cloud-application development, where we write top notch html5/CSS3/JS web applications that run and make use of the Amazon AWS scalable cloud infrastructure. We also do mobile applications backends and front-end (Android, iOS). We usually use the frankly excellent Ubuntu server, unless the customer forces us otherwise! Ubuntu server is simply the best OS for the cloud IMO. Our DevOps team helps our customers migrate and run their apps on the cloud (and on Ubuntu server) at top quality!

If you've read so far, here's a Gift for you! I'm offering my (and my team's) help (up-to-24hrs) for free in the next 30-days. If you're thinking about coding a custom cloud app, or have questions about any Amazon services, I'll give you free consultation and answer any questions. Shoot me a comment over here, or contact the team. Oh yeah, feel free to share this blog post with friends, to let them have the free goodies!

Oneiric server, Deploy Server fleets p2

Welcome to the second installment of this article series. In the first part of this article we installed an Ubuntu server instance, made sure it became an orchestra installation server. If this is new to you, Orchestra is a new Oneiric server feature that enables admins to very easily deploy fleets of Ubuntu servers. Let's pick up where the first article stopped

First, let's check where we are. You see installing the orchestra server, it automatically downloads and imports various Ubuntu server ISOs and creates all the needed structure (distros, profiles ...etc) in the underlying cobbler system. Let's see what have we

$ sudo cobbler list
distros:
   hardy-i386
   hardy-x86_64  
   lucid-i386
   lucid-x86_64  
   maverick-i386 
   maverick-x86_64
   natty-i386
   natty-x86_64  
   oneiric-i386  
   oneiric-x86_64

profiles:
   hardy-i386
   hardy-i386-juju
   hardy-x86_64
   hardy-x86_64-juju
   lucid-i386
   lucid-i386-juju
   lucid-x86_64
   lucid-x86_64-juju
   maverick-i386
   maverick-i386-juju
   maverick-x86_64
   maverick-x86_64-juju
   natty-i386
   natty-i386-juju
   natty-x86_64
   natty-x86_64-juju
   oneiric-i386
   oneiric-i386-juju
   oneiric-x86_64
   oneiric-x86_64-juju

systems:

repos:
   hardy-i386
   hardy-i386-security
   hardy-x86_64
   hardy-x86_64-security
   lucid-i386
   lucid-i386-security
   lucid-x86_64
   lucid-x86_64-security
   maverick-i386
   maverick-i386-security
   maverick-x86_64
   maverick-x86_64-security
   natty-i386
   natty-i386-security
   natty-x86_64
   natty-x86_64-security
   oneiric-i386
   oneiric-i386-security
   oneiric-x86_64
   oneiric-x86_64-security

images:

mgmtclasses:
   orchestra-juju-acquired
   orchestra-juju-available

woah! that sure makes my life easier. If you're interested to see where the isos were downloaded (like I was) here you are

ls /var/lib/cobbler/isos/
hardy-i386-mini.iso    lucid-i386-mini.iso    maverick-i386-mini.iso    natty-i386-mini.iso    oneiric-i386-mini.iso
hardy-x86_64-mini.iso  lucid-x86_64-mini.iso  maverick-x86_64-mini.iso  natty-x86_64-mini.iso  oneiric-x86_64-mini.iso

Let's create a new virtual box VM, to serve as our new "server" that needs to be installed. Here's how it looks for me


One thing is worth noting however, it's that the NIC is placed on the "intnet" network, which has the IP range 192.168.77.0/24 that we configured in the first part of this article


now the only "real" thing you have to do, is to add a profile on the orchestra server for your new bare server. The profile binds its mac address, to a name and an installation profile (think OS to install, kickstart ..etc)

sudo cobbler system add --name="oneiric01.ubuntu.lan" --mac-address="08:00:27:B7:76:2A" --ip-address="192.168.77.33" --dns-name="oneiric01.ubuntu.lan" --hostname="oneiric01.ubuntu.lan" --profile="oneiric-x86_64-juju" --mgmt-classes="orchestra-juju-available" --kopts=" DEBCONF_DEBUG=developer netcfg/dhcp_timeout=120 netcfg/choose_interface=eth0"

Boot the server, choose PXE (For vbox that's F12 then "l" that's an L)



Watch the installer fly by (look ma hands free)



and your box is ready!


That's how easy it is to install a fresh server off your orchestra box! So basically the only thing you need to do per server, is to attach it to a profile and that's it. Boot it and it installs whatever you provisioned for it. Of course any good admin already did that manually before, but it took effort and it wasn't standardized. Now you can count on Ubuntu server covering your back when you're tasked with installing a hundred servers

How cool was that! Got thoughts, comments or rotten tomatoes ? Shoot me a comment

Oneiric server, Deploy Server fleets p1

I'm gonna be posting a series of articles on new features and cool technology bits that are landing in Ubuntu Oneiric (11.10) server. Why? I like servers, I like cloud, I like Ubuntu, it all mixes well, what's not to like :)

During this first article, I'll be demoing (in a graphically intensive way :) what it takes (hint: not much!) to deploy a server fleet with Oneiric server. Orchestra is the name of a wonderful piece of technology that lands in Oneiric, that's been created on top of the open-source cobbler project. Orchestra is super easy to install and get started with, and enables you to very rapidly deploy tens or hundreds of physical servers. I'll be using virtualbox to build a small test "lab" on my laptop for purposes of this article. I did actually try KVM first, but faced some trouble getting PXE booting reliably, so I opted for virtualbox which worked flawlessly (kudos vbox guys, you rock!)

Let's get started, I created a VM to represent the very first "head node", that will install the rest of all nodes. Here is a summary of its configuration

Pop in the virtual CD, boot it, press F6, add "priority=critical locale=en_US url=http://bit.ly/uquick" (Thanks Dustin!) so it looks like

The uquick profile answers all the installer questions, such that the installation is fully automatic. Since the VM contains two NICs however, we'll need to select a primary one (eth0 in my case)

The installation runs like a champ, fully automated, give it a few minutes till it finishes everything and reboots into the server OS (oh that was easy!)

Now I configure eth1 to have a static IP address of 192.168.77.1/24 (I made any address up), here is a snapshot of /etc/network/interfaces and I started eth1 using ifup

At this stage, I rebooted the server but you definitely don't have to. Let's start actually installing Orchestra

sudo apt-get update
sudo apt-get install ubuntu-orchestra-server -y

Everything proceeds automatically, for any question you get during package installation, I'll provide a picture with the answer :)







That's it! You've just installed and configured your first Ubuntu Orchestra server, and you're now ready to install a fleet of Ubuntu servers the easy way! In part 2 of this article, I'll go through creating a second server, PXE booting and installing it from the orchestra server. (Extra credit: If you can't wait, try PXE booting a fresh server right now. Note that after installation, orchestra actually downloads and auto-imports a few Ubuntu mini ISOs, thus will need a few minutes depending on your internet connection speed)

So, what do you think of this coolness? Is this easier than the last time you tried building yourself an automated network installer? Shoot me a comment, let me know what you think

Ensemble meets Hadoop on the cloud

So you wanted to play with hadoop to crunch on some big-data problems, except that, well getting a hadoop cluster up and running in not exactly a one minute thing! Let me show you how to make it "a one minute thing" using Ensemble! Since Ensemble now has formulas for creating hadoop master and slave nodes, spinning up a hadoop cluster could not be easier! Check this video out


If you can't see the embedded video, here's a direct link http://youtu.be/e8IKkWJj7bA

Yep that's how simple it is! If you want to scale-out the cluster, you only need to ask Ensemble to do it for you:
$ ensemble add-unit hadoop-slave

So is this easier than configuring a hadoop cluster manually? Leave me a comment, let me know your thoughts! Also let me know what you'd like to see deployed next with Ensemble

Ubuntu takes UFOs to the cloud

I've always believed in UFOs as a kid, and while I've never seen one (yet?) I am still more on the believer side! I was interested to stumble upon a database of UFO sightings at http://www.infochimps.com/tags/ufo# A shout-out at infochimps (you guys are great!). Downloading the sightings DB (around 80MB), I found a listing of 60,000 documented sightings, hmm interesting! I started thinking I could crunch on this data in some useful and fun way, what about finding the most commonly spotted UFO shape?! Sounds like I could use hadoop for that, just for the coolness factor really, the data is not that large anyway, but hey why not! I had no-idea how to get started with hadoop though and wasn't really interested in learning up all the gory details!

Well Ensemble to the rescue, hadoop master and slave formulas exist, which means someone else packaged the knowledge needed to setup and run a hadoop cluster for me. All I needed to do was ask Ensemble to deploy me a couple of cloud instances and start playing. Let's see how you can do that for yourself

I won't repeat the instructions to get started with Ensemble, since the documentation is a good place for that (and it's so easy anyway!). If you feel you need more help there, this little video should be helpful. If you're still stuck, you can always drop by on irc/freenode at #ubuntu-ensemble and ask your questions

Hadoop node, with an extra slave please

So, let's start ensembling

bzr branch lp:~negronjl/+junk/hadoop-master
bzr branch lp:~negronjl/+junk/hadoop-slave
ensemble bootstrap

wait a minute or two for EC2 to spin up the instance, then

ensemble status

which’ll give you output like

$ ensemble status
2011-07-12 15:20:54,978 INFO Connecting to environment.
The authenticity of host 'ec2-50-17-28-19.compute-1.amazonaws.com (50.17.28.19)' can't be established.
RSA key fingerprint is c5:21:62:f0:ac:bd:9c:0f:99:59:12:ec:4d:41:48:c8.
Are you sure you want to continue connecting (yes/no)? yes
machines:
  0: {dns-name: ec2-50-17-28-19.compute-1.amazonaws.com, instance-id: i-8bc034ea}
services: {}
2011-07-12 15:21:01,205 INFO 'status' command finished successfully

Now let's deploy a two node hadoop cluster

ensemble deploy --repository . hadoop-master
ensemble deploy --repository . hadoop-slave
ensemble add-relation hadoop-master hadoop-slave

Yeah it's that easy! Ensemble formulas manage all the kung-fu for you. The hadoop cluster is ready, let's ssh into the master node and switch to user hdfs

ensemble ssh hadoop-master/0
sudo -su hdfs

Downloading UFOs

Download the infochimps sightings database here, unzip it and locate the TSV file (tab separated values) file. Note that you can download the file from infochimps without registering on their website (didn't I say these guys were great :)

Upload the TSV DB to hadoop's distributed filesystem

hadoop dfs -copyFromLocal ufo_awesome.tsv ufo_awesome.tsv

Almost ready, the corpus has been uploaded. Now we need to write some map/reduce jobs to do the actual crunching. Not being a pro developer, the thought of writing that in java was like (oh no ew), so python to the rescue! Thanks to the great instructions at Michael Noll's blog, I was able to massage some of that code to get it to do what I wanted. I pushed my code to launchpad, so that you can grab it directly from the hadoop master node

cd /tmp
bzr branch lp:~kim0/+junk/ufo-ensemble-cruncher
cd ufo-ensemble-cruncher

Unleashing the elephant

Now for the big moment, let's launch the elephant

hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-*.jar -file ./mapper.py -mapper mapper.py -file ./reducer.py -reducer reducer.py -input ufo_awesome.tsv -output ufo-output
packageJobJar: [./mapper.py, ./reducer.py, /tmp/hadoop-hdfs/hadoop-unjar1418682529553378062/] [] /tmp/streamjob5701745574334998473.jar tmpDir=null
11/07/29 12:27:52 INFO mapred.FileInputFormat: Total input paths to process : 1
11/07/29 12:27:53 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-hdfs/mapred/local]
11/07/29 12:27:53 INFO streaming.StreamJob: Running job: job_201107290935_0010
11/07/29 12:27:53 INFO streaming.StreamJob: To kill this job, run:
11/07/29 12:27:53 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=domU-12-31-39-10-81-8E.compute-1.internal:8021 -kill job_201107290935_0010
11/07/29 12:27:53 INFO streaming.StreamJob: Tracking URL: http://domU-12-31-39-10-81-8E.compute-1.internal:50030/jobdetails.jsp?jobid=job_201107290935_0010
11/07/29 12:27:54 INFO streaming.StreamJob:  map 0%  reduce 0%
11/07/29 12:28:11 INFO streaming.StreamJob:  map 10%  reduce 0%
11/07/29 12:28:12 INFO streaming.StreamJob:  map 19%  reduce 0%
11/07/29 12:28:14 INFO streaming.StreamJob:  map 72%  reduce 0%
11/07/29 12:28:16 INFO streaming.StreamJob:  map 100%  reduce 0%
11/07/29 12:28:33 INFO streaming.StreamJob:  map 100%  reduce 100%
11/07/29 12:28:37 INFO streaming.StreamJob: Job complete: job_201107290935_0010
11/07/29 12:28:37 INFO streaming.StreamJob: Output: ufo-output

Woohoo success! Now let's grab the results, sorting it to easily see the most popular sighting shape

Is the answer really 42

hadoop dfs -cat ufo-output/part-00000 | sort -k 2,2 -nr
light   12202
triangle        6082
circle  5271
disk    4825
other   4593
unknown 4490
sphere  3637
fireball        3452
oval    2869
formation       1788
cigar   1782
changing        1546
flash   990
cylinder        982
rectangle       966
diamond 915
chevron 760
egg     664
teardrop        595
cone    265
cross   177
delta   8
round   2
crescent        2
pyramid 1
hexagon 1
flare   1
dome    1
changed 1

The answer is "light" then! Wow that was a blast! I had fun doing this exercise. Now I am no hadoop expert in any way (so direct those hadoopy questions to someone who can actually answer them), however I was quite pleased Ensemble could help me get up and running that fast. The Ensemble community is doing a great job wrapping many free software with formulas, such that you can always get up and running with any app you need in seconds rather than days (months?). You too can write Ensemble formulas for your favorite (server?) application. Hop on to #ubuntu-ensemble and grab me (kim0) or any of the dev team and ask any questions on your mind! We're a happy community

So was that fun? Can you think of something cooler you want to see done? Leave me a comment, let me know about it

Cloud Portal one-click launch

Announcing two nice little additions to the cloud portal AMI tool that should make everyone's cloud life a little easier

• Amazon now allows passing parameters to the AWS console to basically be able to choose the region and AMI-ID to launch. This is now integrated in the cloud portal so that you search for the ami, click the link and it's pre-selected, ready to be launched in AWS console. Here's a screenshot

• As that image shows as well, you can search now for "cluster" to get Ubuntu cloud cluster-ready images!

Interested to add any features? Have any thoughts or comments, let me know, leave me a word

Help Squash Natty Server Bugs

Mirroring a post from Ubuntu Server developer Dave Walker, If you're capable and willing to help squash some server bugs, your effort and contribution is really needed now! Join us on irc #ubuntu-server, ask around if you need help. Thanks in advance

Here is a list of current bugs that should hopefully be fixed for Natty release.  The bugs with "Not assigned to anyone :(", could really do  with someone volunteering to take.  Please feel free to assign yourself, if you are willing to take on working towards a fix. 

If you are an assignee, please make sure that the current status (and comments) accurately represent the status before tomorrows server team meeting. 

== Bugs for the server team == 

[655533] [likewise-open] [master] package likewise-open 5.4.0.42111-2ubuntu2 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1 (http://pad.lv/655533) 

 * Bug Status: Confirmed 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-18 

[726769] [eucalyptus] package eucalyptus-common 2.0.1 bzr1255-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1 (http://pad.lv/726769) 

 * Bug Status: Confirmed 

 * Assigned to: Dave Walker 

 * Last updated: 2011-04-18 

[661453] [dovecot] dovecot.conf always shows as having been locally modified on update (http://pad.lv/661453) 

 * Bug Status: Confirmed 

 * Assigned to: Clint Byrum 

 * Last updated: 2011-04-18 

[745946] [cloud-init] cloud-final job did not run in ec2-automated-tests (http://pad.lv/745946) 

 * Bug Status: Confirmed 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-18 

[759943] [mod-wsgi] mod_wsgi.so-3.2 gives errors (http://pad.lv/759943) 

 * Bug Status: Confirmed 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-15 

[764391] [cobbler] cobbler fails to manage bind9  (http://pad.lv/764391) 

 * Bug Status: Confirmed 

 * Assigned to: James Page 

 * Last updated: 2011-04-18 

[745930] [cloud-init] cloud-init timeout waiting for metadata service on EC2 (http://pad.lv/745930) 

 * Bug Status: New 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-15 

[760288] [Ubuntu] JeOS is oversized (http://pad.lv/760288) 

 * Bug Status: Confirmed 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-15 

 == Bugs being worked on in other areas, more for information == 

[728088] [debian-installer] iscsi root with or without auth fails to boot (http://pad.lv/728088) 

 * Bug Status: Confirmed 

 * Assigned to: Colin Watson 

 * Last updated: 2011-04-14 

[747090] [linux] wrong return address sometimes pushed for INT in kvm (not qemu) (http://pad.lv/747090) 

 * Bug Status: Fix Committed 

 * Assigned to: Andy Whitcroft 

 * Last updated: 2011-04-18 

[566818] [plymouth] Cryptsetup passphrase prompt during boot: every character typed repeats the prompt (http://pad.lv/566818) 

 * Bug Status: Confirmed 

 * Assigned to: Surbhi Palande 

 * Last updated: 2011-04-16 

[580319] [upstart] dhcp3-server launches before upstart brings all interface, thus failing to start (http://pad.lv/580319) 

 * Bug Status: Triaged 

 * Assigned to: James Hunt 

 * Last updated: 2011-04-18 

[759545] [grub2] user prompted to update unmodified grub configuration during Ubuntu server upgrade (http://pad.lv/759545) 

 * Bug Status: New 

 * Last updated: 2011-04-18 

Ensemble Cloud Community Meeting

Ensemble is an exciting new cloud technology innovated by Ubuntu. If you’re curious what it does (I bet you are) it is a service orchestration framework with focus on rapid cloud deployment. That is to say, it allows one to rapidly deploy cloud instances, bootstrap them, install software on them, configure them as well as link them all together to provide you (as a sys-admin or DevOp) with an end-user facing high-level “service” (Load Balanced CMS with sharded Database, Multinode email service installation…etc). Ensemble has been compared to being APT for the cloud! Now does that sound interesting, it sure is if you ask me! Welcome to next generation Linux infrastructure automation.

If that got you interested, be sure to attend the weekly IRC meeting on Wed 18:00-UTC on #ubuntu-cloud. 

Read More at: http://cloud.ubuntu.com/2011/04/ensemble-cloud-community-meeting/

Feel free to tweet/dent and spread this around the globe. Got feedback? Leave me comments on cloud.ubuntu.com or on my blog

Help Shape Maps.Ubuntu.com

About either months ago, I had started on a fun little project to raise awareness of how widely used Ubuntu Server is around the world. The result was http://maps.ubuntu.com/ a web tool allowing anyone using Ubuntu Server to "mark" his location on a global map. This was launched with celebrations for 10.04.1, and thousands of cities were marked (woohoo). The project code was open-sourced and lives at https://launchpad.net/mapuntu/trunk

It's been a while since the project was last updated, a few days ago, an Ubuntu community member ( hey Ronnie ) stepped up to re-shape the mapuntu project and relaunch it into an inspiring map that helps anyone record and locate Ubuntu events around the globe! The first part of that effort is creating a proper back-end API, everyone is encouraged to click that link and leave comments. If you are inspired by that project, and would like to help, please get involved! Things you can do to help include

• Review and enhance the proposed API
• Help write python code to implement the backend api
• Write html/css/js for the front-end
• Play with the code, test it, report bugs, adapt to your own website!

If you want to talk more about this, Please join #ubuntu-locoteams on IRC and ping Ronnie or myself (kim0, also on #ubuntu-cloud).

Creating Customized UEC and EC2 Images

Want to create a customized Ubuntu cloud image that you can deploy to EC2 or UEC ? No problem! Check out this 5 min video where I demo everything needed to download, customize, rebundle and publish the new image to EC2! Isn't this just great :)



If you can't see the embedded player, here's a direct link: http://www.youtube.com/watch?v=K1IwTGLWbu0

Ubuntu Cloud Days, Day-2 Summary

Yesterday was the second and final day of cloud days. It was such a blast! Very interesting stuff, and very cool community to be part of! In case you missed it, you can still read up the logs, and you can reach me and others for questions and comments on #ubuntu-cloud .. Let's quickly zip through yesterday's sessions

16.00 UTC smoser gave a very good introduction to the different ways with which you can rebundle UEC or EC2 images for Ubuntu, how you can customize them and the different pros and cons. Scott also answered lots of questions
17.00 UTC tetet started by presenting some interesting new features and tricks that can be used with UEC namely persistency, which means kexec'ing the kernel to load from an EBS volume resulting in persistent UEC instances. Thorsten zipped through the session quickly, but as a pro instructor, he was very well prepared with links to graphics, links and documentation
18.00 UTC Daviey wowed the crowd with a puppet introductory session, he showed how to configure apache with puppet and what it takes and deploy a basic system to get running fast. It was a great session
19.00 UTC edulix took over to present an awesome session about using hadoop and its map/reduce programming model to crunch on large (we're talking terabytes) of data on lots of machines. Very interesting and useful session
20.00 UTC obino had the final say on cloud days. The session was about UEC/Eucalyptus. He presented a detailed architecture of the system, what each node does, how things work. He also explained how to get started contributing to the eucalyptus community. 

It was a great event. Big Thank-Yous to everyone who presented in UCD. Till next time o/

Day-2 Ubuntu Cloud Days

Yesterday was great! It was Ubuntu Cloud Days, the event was packed with very interesting sessions, hundreds of attendees, lots of questions got answered, tons of ideas and comments were made. I personally had lots of fun, and was literally stuck to my keyboard the full time of the event


Let's zip through what happened yesterday (If you missed it, Click the time to read the logs!)
16.00 UTC kim0 (me) took-over, gave a quick introduction about cloud computing, differences between public and private clouds and what the cloud fuss is all about
17.00 UTC semiosis started a great session about building scalable distributed storage on the EC2 Cloud using Glusterfs for improving redundancy, and performance
18.00 UTC SpamapS gave a cool presentation and hands-on demo of a new Ubuntu ground breaking "service" management framework called Ensemble. While in early development phase, it's already working well and is changing how we look at server management
19.00 UTC hallyn took over presenting a technically deep discussion and hands-on demo about LXC Linux containers technology. This is a light weight Linux virtualization (or container) technology
20.00 UTC soren took over to talk about OpenStack, one of the cloud stacks that Ubuntu ships with 11.04 that's been getting lots of cloud people excited lately. Soren explained how the whole thing got started, explained the architecture, answered lots of questions and discussed what we should be seeing in the future

Here's a sneak peak of today's sessions

Thu 24th Mar
16.00 UTC rebundling/re-using Ubuntu’s UEC images — smoser
17.00 UTC UEC persistency — tetet
18.00 UTC Puppet Configuration Management — Daviey
19.00 UTC Using hadoop, divide and conquer — edulix
20.00 UTC UEC/Eucalyptus Private Cloud — obino

Let's get crunching on even more cool sessions today! If you’re new to IRC, you can simply use this web page to join. For more information, check out the wiki page, also feel free to ping “kim0″ on IRC!

Ubuntu Cloud Days approaching

Just a reminder, Ubuntu Cloud Days is approaching, it's only two days away! Yes it's this Wed/Thu. I just posted the full details on the Ubuntu Cloud Portal. Check it out, and leave me a comment if you have any questions

Cloud Python hackers wanted

A while back I had blogged (yes the one with the big uncle Sam pic) asking for people interested in working on a python tool to help migrate ebs AMIs across ec2 regions. Since then the project has progressed a bit:

- Project page is now: https://launchpad.net/ec2-migrate-ebs-ami
- Project is now able to launch two utility instances in the two specified regions, and mount the source volume in the source instance
- What remains is attaching a destination volume, and sync'ing over contents. More details can be found on the blueprint: https://blueprints.launchpad.net/ubuntu/+spec/cloud-server-n-ec2-migrate-region

Development pace has now slowed down significantly, as such I am renewing the call. This is a great way to start getting involved with cloud development, and to join the Ubuntu Cloud and server communities! joining in is "easy", you don't even have to know lots of python to start hacking! The steps are very well defined, and I am happy to help with all steps along the way. If you are thinking about it, go ahead and get in touch with me right now (kim0 on irc, kim0 AT ubuntu.com) or join today's irc meeting (6pm-UTC #ubuntu-cloud https://wiki.ubuntu.com/UbuntuCloudMeeting) and start discussing how you can get involved

FAST Win7 KVM VirtIO{Disk-Net} Install

Everyone who has to use Windows, please let's at least keep it contained inside a virtual machine! In this article I'll demo how to "Install Windows7 over Ubuntu 11.04 Natty, using KVM with System Disk over VirtIO". Quoting the libvirt wiki "Virtio is a Linux standard for network and disk device drivers where just the guest's device driver "knows" it is running in a virtual environment, and cooperates with the hypervisor. This enables guests to get high performance network and disk operations, and gives most of the performance benefits of paravirtualization"

Here's a little convenience script to download virtio Windows drivers and launch virt-install to Install Windows. Copy/Paste this script into an empty folder, chmod +x it, and run it.

#!/bin/sh
WINISO=/path/to/win7.iso    #Windows ISO
INSTALLDISK=win7virtio.img  #Disk location. Can be LVM LV
VFD=http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-1.1.16.vfd
DRVRISO=http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-1.1.16.iso

[ -e $(basename $VFD) ]     || wget $VFD
[ -e $(basename $DRVRISO) ] || wget $DRVRISO
[ -e $INSTALLDISK ]         || qemu-img create $INSTALLDISK 30G

sudo virt-install -c qemu:///system --virt-type kvm --name win7virtio --ram 1024 --disk path="$INSTALLDISK",bus=virtio \
--disk $(basename $VFD),device=floppy --os-variant win7 --cdrom $(basename $DRVRISO) --cdrom "$WINISO" --vcpus 2

et voila Windows installer kicks in, we're greeted with the usual



Next the welcome screen, and agree on the license agreement (if you want!)


In this next screenshot, we see how Windows Installer does not see any "disks". This is because Windows now does not have any drivers for the virtio disk that we attached!


Let's fix that, clicking "Load Driver" and clicking OK, Windows starts reading the VFD virtual floppy image we attached, and detects the driver


Clicking Next, Windows can now see the virtio disk

Then it's business like usual

After installation completes, be sure to check the device manager and confirm the disk (and network) are both using optimized virtio drivers


et voila, Windows in its full glory running over optimized virtio drivers. The only part that sucks is desktop VNC performance, something that I'll probably write about very soon :)

First Ever Ubuntu Cloud Days

I am very excited to announce the very first Ubuntu Cloud Days! UCD is an online event that is designed for everyone interested in cloud computing, as well as someone who is already using Ubuntu server and who's thinking about "that cloud thing" everyone is talking about. If the words "virtualization", "cloud", "server", "scalability" or "automation" intrigue and amuse you, then you should definitely attend! It's a chance to learn a lot, get your questions answered as well as share your experiences and have fun with the rest of the cloud community!

In fact I'll go one bit further, not only should you attend, you should even deliver a session! Ubuntu as usual is all about community, learning from others, and sharing your knowledge to others. It's quite easy to deliver a session! My first session, I was quite surprised how I didn't have to prepare huge content to "fill" an hour, just answering questions and discussing with people, easily gets you by and you get a warm fuzzy feeling that you just helped lots of people and shared your knowledge.

So, Ladies and Gentlemen, mark your calendars! March 23rd and March 24th

Fore more information: https://wiki.ubuntu.com/UbuntuCloudDays/

To contribute a session, (or if you're thinking about it) talk to me (kim0) on irc #ubuntu-cloud or email [ kim0 =AT= ubuntu.com ]

Volunteer QA to Ubuntu Server

A great way to help contribute to Ubuntu and ensure Ubuntu Server remains the rock solid platform that it is, is by committing to helping perform specific QA tests in predefined scenarios. If you're interested, the following test cases specifically need someone to step up and commit to them

• Install (JeOS on ESX) QA Testcase
• Install (JeOS on KVM) QA Testcase
• iSCSI Authenticated Root Installation QA Testcase
• iSCSI Unauthenticated Root Installation QA Testcase

These tests are being marked as NOT required for ISO release exit criteria for Ubuntu Server. Quoting Robbie Williamson "Our decision was based on the lack of consistently available resources to verify these configurations. With that said, we are keeping them optional"

Robbie adds "Anyone wishing to reverse this decision, please contact me directly.  I have no problem adding them back along with your agreement to run them for each ISO release"

Again that's a great way to contribute to Ubuntu. If interested please contact Robbie directly, or contact me (leave a comment right here) and I will help get you on track

Cloud Backup LAMP Stack like a pro

Running Ubuntu server in the cloud to run your own LAMP environment is extremely popular. Backing up that environment is definitely something that almost everyone should think about, unless it's a test env. In this screencast I'll demo using the ec2-consistent-snapshot tool to perform consistent database backups over a XFS formatted ebs volume

If you can't see the embedded player, here's a direct link
http://www.youtube.com/watch?v=SPVqJWWiLVI



Shoot me a comment if there's something specific you'd like to see, or if you'd like to contribute one such screencast! Till next time