tag:blogger.com,1999:blog-3429214981094250042024-02-07T06:09:17.018+02:00FOSS BossEgyptian geek's Blog about Linux Solaris Unix open-source and everything hot and cool in technologyKamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.comBlogger48125tag:blogger.com,1999:blog-342921498109425004.post-32382949037339209782013-03-12T00:07:00.000+02:002013-03-12T00:35:25.374+02:00Bash script self-updates from github<div dir="ltr" style="text-align: left;" trbidi="on">
Recently the systems engineering team @ <a href="http://www.cloud9ers.com/" target="_blank">Cloud9ers</a> ($dayjob) has been busy building the infrastructure for a very ambitious and large scale project for one of our customers. The project involves tons of distributed programming besides lots of systems work as well. I'd like to share with the community a small tidbit of information. It's often some small things the one sometimes gives a moment to admire, and hereby share!<br />
<h4 style="text-align: left;">
The problem</h4>
<div>
<ul style="text-align: left;">
<li>We are building a private cloud. Ubuntu server instances running on that are cloned from a master template</li>
<li>While most clouds provide what I call "Instance Identity" information through some pre-known web service URI, in our case we utilize VMware's "<a href="https://www.vmware.com/support/developer/PowerCLI/PowerCLI501/html/Invoke-VMScript.html" target="_blank">Invoke-VMScript</a>" API to run scripts inside the cloned template, thus customizing it and giving it its identity, then <a href="https://puppetlabs.com/" target="_blank">puppet</a> takes it from there. Note that we're using a simple vCenter+ESXi (no vCloud stuff here) no shared storage even!</li>
</ul>
<h4 style="text-align: left;">
Ok the real problem</h4>
</div>
<div>
<ul style="text-align: left;">
<li>Editing the template, and publishing it across the cloud (a topic for another post) takes considerable time! I wanted a way to be able to quickly update my identity scripts without having to re-build and re-publish images</li>
</ul>
<h4 style="text-align: left;">
Solution</h4>
</div>
<div>
<ul style="text-align: left;">
<li>A script with a trivially simple (thus mostly fixed) "bootstrap" section, which auto-updates itself from github and relaunches its-new-self!</li>
</ul>
<h4 style="text-align: left;">
Code</h4>
</div>
<div>
Nothing ground breaking, but still a cool trick eh! Yeah I could have added a broken this into multiple scripts and maybe run-parts them. Of-course this script is quite basic, but it's meant to remain that way since the heavy lifting is puppet's responsibility!<br />
Note: To run this successfully the user this is run under, should have passwordless sudo rights to run "ip" and to run "itself" :) Got cool ideas, thoughts or comments? Leave me a message<br />
<br /></div>
<script src="http://gist-it.appspot.com/github/kim0/vmware-bootstrap-puppet/blob/master/customization.sh?footer=0"></script>
</div>
Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0Cloud Niners30.105163797485403 31.37691201383063330.103446797485404 31.374390513830633 30.106880797485402 31.379433513830634tag:blogger.com,1999:blog-342921498109425004.post-39031299804253891742012-12-26T17:12:00.000+02:002012-12-26T17:12:27.017+02:00LXC lies about /.. inode number making FlexLM unhappy<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
One of our customers wanted to run a flexlm licensed tool on the Amazon EC2 cloud. He turned to us over here at <a href="http://www.cloud9ers.com/" target="_blank">Cloud Niners</a> and we started doing the work. The approach we chose was to start an ec2 Ubuntu precise machine, host a CentOS LXC container on it and run the license daemon inside that. The reason for the CentOS choice is that the tools and license daemons are only certified on RHEL and CentOS is the closest next thing<br />
<br />
Trying to start the license daemon, I hit the following error<br />
<br /></div>
<pre>
8:59:27 (TOOLNAME) Cannot open daemon lock file
<br />
</pre>
Checking the usual suspicions of permissions and the like led no where. Google immediately led me to similar problems for people wanting to run Flexlm tools under Solaris zones (which are quite similar to Linux LXC containers). I was reading about a guy <a href="http://coffeeortech.wordpress.com/2011/07/05/dtrace-saves-the-day/" target="_blank">facing a similar problem</a>, and how Solaris' legend Brendan Gregg wrote <a href="http://dtrace.org/blogs/brendan/2011/06/30/tweaking-memory-on-the-fly/" target="_blank">dtrace scripts to run-time patch memory structures</a> to resolve the issue.<br />
<br />
At first I was dismissive that this was what I was actually see'ing. A quick "ls -lai /" just to confirm /. and /.. actually had the same inode number, and I was sure it was something else<br />
<br />
<pre>
# ls -lai / | grep '\.$'<br />
256 dr-xr-xr-x 1 root root 212 Dec 26 09:08 .<br />
256 dr-xr-xr-x 1 root root 212 Dec 26 09:08 ..<br />
</pre>
<br />
<br />
<br />
but I had one of those mmm moments after stracing the binaries and confirming they are actually failing immediately after calling getdents on "/". This sounds suspiciously close to what the Solaris folks were see'ing. I grabbed gcc and built the sample code in <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/getdents.2.html" target="_blank">getdents kernel doc page</a> (Thankful!) And much to my surprise, inode numbers for /. and /.. were actually different!<br />
<br />
<pre>
# ./a.out / | grep '\.$'<br />
256 directory 24 1 .<br />
42669 directory 24 2 ..<br />
</pre>
<br />
<br />
<br />
Of course this makes sense since the LXC guest was just another directory on the host, but I didn't suspect ls -i was actually lying inside guests!<br />
<br />
At this point, I'm not exactly sure how to resolve my issue apart from reinstalling the LXC guest on a separate block device (like lvm) which I think should resolve the issue. This blog post is simply to confirm the issue, and to gather feedback and potential solutions from smart people reading this. <b>Shoot</b> me a comment</div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com4tag:blogger.com,1999:blog-342921498109425004.post-72283357588276004282012-10-11T11:20:00.001+02:002012-10-11T13:25:00.347+02:00Celebrating Cloud9ers is MEA's leading Amazon Consulting partner<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://aws.amazon.com/solution-providers/si/cloud-niners" target="_blank"><img border="0" height="110" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTSd6oMJr21Izhw_IkHh9vmOejX2BFHRNZgHbQwMi5vmflW7LubJXxk69A2fcNcvaW7sqoppiFcJdqfq4Vb_rD608O4A-iwB8XW2gMZW8_aKxoU1vkimgCJc3gCgg4Tv2GxDk0jVBR_9o/s640/cloud-niners-cloud9ers-amzon-cloud-consulting-partner.jpg" width="640" /></a></div>
<br />
It's been quite a while since I last blogged, I think it was over a year ago! I've been taking a break from blogging and focusing on <a href="http://cloud9ers.com/" target="_blank">Cloud Niners</a> my baby startup that I co-founded with a bunch of life long friends.</div>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
While many things are interesting over here at <a href="http://cloud9ers.com/" target="_blank">Cloud Niners</a>, this blog post is specifically to <b>celebrate</b> the fact that we've just been upgraded to become an <a href="https://aws.amazon.com/solution-providers/si/cloud-niners" target="_blank">Amazon Consulting Partner</a>. I believe this makes us the first active middle-east <a href="http://aws.amazon.com/" target="_blank">Amazon AWS cloud</a> consulting partner and certainly the best :) I do hear a lot from Amazon that we are the <b>cloud leader</b> in the middle-east! I am quite happy that we've achieved this milestone. I'm a big believer in cloud computing, and I simply enjoy helping our clients understand how the cloud (and <a href="http://www.ubuntu.com/download/server" target="_blank">Ubuntu server</a>) can help them achieve more!</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.ubuntu.com/download/server" target="_blank"><img border="0" height="86" src="http://www.ubuntu.com/sites/www.ubuntu.com/files/active/02_ubuntu/U_business/pictograms-server.png" width="320" /></a></div>
My team and I do <a href="http://cloud9ers.com/services/cloud-computing/cloud-development" target="_blank">custom cloud-application development</a>, where we write top notch <a href="http://en.wikipedia.org/wiki/HTML5" target="_blank">html5</a>/CSS3/JS web applications that run and make use of the Amazon AWS scalable cloud infrastructure. We also do mobile applications backends and front-end (<a href="http://cloud9ers.com/services/mobile-development/android-development" target="_blank">Android</a>, <a href="http://cloud9ers.com/services/mobile-development/ios-development" target="_blank">iOS</a>). We usually use the frankly excellent <a href="http://www.ubuntu.com/business/server/overview" target="_blank">Ubuntu server</a>, unless the customer forces us otherwise! Ubuntu server is simply the best OS for the cloud IMO. Our <a href="http://en.wikipedia.org/wiki/Devops" target="_blank">DevOps</a> team helps our customers migrate and run their apps on the cloud (and on Ubuntu server) at top quality!<br />
<br />
If you've read so far, here's a <b>Gift</b> for you! I'm offering my (and my team's) help (up-to-24hrs) for free in the next 30-days. If you're thinking about coding a custom cloud app, or have questions about any <a href="http://cloud9ers.com/services/cloud-computing/amazon-services" target="_blank">Amazon services</a>, I'll give you free consultation and answer any questions. Shoot me a comment over here, or <a href="http://cloud9ers.com/contact" target="_blank">contact</a> the team. Oh yeah, feel free to share this blog post with friends, to let them have the free goodies!</div>
</div>
Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-44455883247718077482011-09-23T20:53:00.001+02:002011-09-23T21:06:39.830+02:00Oneiric server, Deploy Server fleets p2Welcome to the second installment of this article series. In the first part of this article we installed an Ubuntu server instance, made sure it became an orchestra installation server. If this is new to you, Orchestra is a new Oneiric server feature that enables admins to very easily deploy fleets of Ubuntu servers. Let's pick up where the first article stopped<br />
<br />
First, let's check where we are. You see installing the orchestra server, it automatically downloads and imports various Ubuntu server ISOs and creates all the needed structure (distros, profiles ...etc) in the underlying cobbler system. Let's see what have we<br />
<br />
<pre>$ sudo cobbler list
distros:
hardy-i386
hardy-x86_64
lucid-i386
lucid-x86_64
maverick-i386
maverick-x86_64
natty-i386
natty-x86_64
oneiric-i386
oneiric-x86_64
profiles:
hardy-i386
hardy-i386-juju
hardy-x86_64
hardy-x86_64-juju
lucid-i386
lucid-i386-juju
lucid-x86_64
lucid-x86_64-juju
maverick-i386
maverick-i386-juju
maverick-x86_64
maverick-x86_64-juju
natty-i386
natty-i386-juju
natty-x86_64
natty-x86_64-juju
oneiric-i386
oneiric-i386-juju
oneiric-x86_64
oneiric-x86_64-juju
systems:
repos:
hardy-i386
hardy-i386-security
hardy-x86_64
hardy-x86_64-security
lucid-i386
lucid-i386-security
lucid-x86_64
lucid-x86_64-security
maverick-i386
maverick-i386-security
maverick-x86_64
maverick-x86_64-security
natty-i386
natty-i386-security
natty-x86_64
natty-x86_64-security
oneiric-i386
oneiric-i386-security
oneiric-x86_64
oneiric-x86_64-security
images:
mgmtclasses:
orchestra-juju-acquired
orchestra-juju-available
</pre>woah! that sure makes my life easier. If you're interested to see where the isos were downloaded (like I was) here you are<br />
<pre>ls /var/lib/cobbler/isos/
hardy-i386-mini.iso lucid-i386-mini.iso maverick-i386-mini.iso natty-i386-mini.iso oneiric-i386-mini.iso
hardy-x86_64-mini.iso lucid-x86_64-mini.iso maverick-x86_64-mini.iso natty-x86_64-mini.iso oneiric-x86_64-mini.iso
</pre><br />
Let's create a new virtual box VM, to serve as our new "server" that needs to be installed. Here's how it looks for me<br />
<a href="http://www.flickr.com/photos/90048824@N00/6175453989/" title="12-oneiric01-vboxsettings by Ahmed-Kamal, on Flickr"><img alt="12-oneiric01-vboxsettings" height="500" src="http://farm7.static.flickr.com/6179/6175453989_eddc3b516a.jpg" width="384" /></a><br />
<br />
One thing is worth noting however, it's that the NIC is placed on the "intnet" network, which has the IP range 192.168.77.0/24 that we configured in the first part of this article<br />
<a href="http://www.flickr.com/photos/90048824@N00/6175453857/" title="13-vbox-natty01-netsettings by Ahmed-Kamal, on Flickr"><img alt="13-vbox-natty01-netsettings" height="425" src="http://farm7.static.flickr.com/6166/6175453857_6edd623a70.jpg" width="500" /></a><br />
<br />
now the only "real" thing you have to do, is to add a profile on the orchestra server for your new bare server. The profile binds its mac address, to a name and an installation profile (think OS to install, kickstart ..etc)<br />
<br />
<pre>sudo cobbler system add --name="oneiric01.ubuntu.lan" --mac-address="08:00:27:B7:76:2A" --ip-address="192.168.77.33" --dns-name="oneiric01.ubuntu.lan" --hostname="oneiric01.ubuntu.lan" --profile="oneiric-x86_64-juju" --mgmt-classes="orchestra-juju-available" --kopts=" DEBCONF_DEBUG=developer netcfg/dhcp_timeout=120 netcfg/choose_interface=eth0"
</pre>Boot the server, choose PXE (For vbox that's F12 then "l" that's an L)<br />
<br />
<a href="http://www.flickr.com/photos/90048824@N00/6175981526/" title="14-natty-PXEbooting by Ahmed-Kamal, on Flickr"><img alt="14-natty-PXEbooting" height="317" src="http://farm7.static.flickr.com/6153/6175981526_e4d49e3341.jpg" width="500" /></a><br />
<br />
Watch the installer fly by (look ma hands free)<br />
<br />
<a href="http://www.flickr.com/photos/90048824@N00/6175981638/" title="15-installer-running by Ahmed-Kamal, on Flickr"><img alt="15-installer-running" height="445" src="http://farm7.static.flickr.com/6151/6175981638_432079c8d3.jpg" width="500" /></a><br />
<br />
and your box is ready!<br />
<a href="http://www.flickr.com/photos/90048824@N00/6175981698/" title="16-Oneiric01-ready by Ahmed-Kamal, on Flickr"><img alt="16-Oneiric01-ready" height="416" src="http://farm7.static.flickr.com/6178/6175981698_501492aaa2.jpg" width="500" /></a><br />
<br />
That's how easy it is to install a fresh server off your orchestra box! So basically the only thing you need to do per server, is to attach it to a profile and that's it. Boot it and it installs whatever you provisioned for it. Of course any good admin already did that manually before, but it took effort and it wasn't standardized. Now you can count on Ubuntu server covering your back when you're tasked with installing a hundred servers<br />
<br />
How cool was that! Got thoughts, comments or rotten tomatoes ? Shoot me a commentKamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com7tag:blogger.com,1999:blog-342921498109425004.post-32897148498358728422011-09-21T01:11:00.000+02:002011-09-21T01:11:07.862+02:00Oneiric server, Deploy Server fleets p1<div dir="ltr" style="text-align: left;" trbidi="on">I'm gonna be posting a series of articles on new features and cool technology bits that are landing in Ubuntu Oneiric (11.10) server. Why? I like servers, I like cloud, I like Ubuntu, it all mixes well, what's not to like :)<br />
<br />
During this first article, I'll be demoing (in a graphically intensive way :) what it takes (hint: not much!) to deploy a server fleet with Oneiric server. <a href="https://launchpad.net/orchestra">Orchestra</a> is the name of a wonderful piece of technology that lands in Oneiric, that's been created on top of the open-source <a href="https://fedorahosted.org/cobbler/">cobbler</a> project. Orchestra is super easy to install and get started with, and enables you to very rapidly deploy tens or hundreds of physical servers. I'll be using virtualbox to build a small test "lab" on my laptop for purposes of this article. I did actually try KVM first, but faced some trouble getting PXE booting reliably, so I opted for <a href="http://www.virtualbox.org/">virtualbox</a> which worked flawlessly (kudos vbox guys, you rock!)<br />
<br />
Let's get started, I created a VM to represent the very first "head node", that will install the rest of all nodes. Here is a summary of its configuration<br />
<a href="http://www.flickr.com/photos/90048824@N00/6167776998/" title="1-orchestra by Ahmed-Kamal, on Flickr"><img alt="1-orchestra" height="500" src="http://farm7.static.flickr.com/6159/6167776998_a1b0965c27.jpg" width="364" /></a><br />
Pop in the virtual CD, boot it, press F6, add "priority=critical locale=en_US url=http://bit.ly/uquick" (Thanks <a href="http://blog.dustinkirkland.com/2011/03/ubuntu-server-quick-install-no.html">Dustin</a>!) so it looks like<br />
<a href="http://www.flickr.com/photos/90048824@N00/6167777042/" title="2-orchestra-bootoptions by Ahmed-Kamal, on Flickr"><img alt="2-orchestra-bootoptions" height="416" src="http://farm7.static.flickr.com/6166/6167777042_3a046d47a8.jpg" width="500" /></a><br />
The uquick profile answers all the installer questions, such that the installation is fully automatic. Since the VM contains two NICs however, we'll need to select a primary one (eth0 in my case)<br />
<a href="http://www.flickr.com/photos/90048824@N00/6167242023/" title="3-orchestra-whicheth by Ahmed-Kamal, on Flickr"><img alt="3-orchestra-whicheth" height="408" src="http://farm7.static.flickr.com/6178/6167242023_f6d68d1a1a.jpg" width="500" /></a><br />
The installation runs like a champ, fully automated, give it a few minutes till it finishes everything and reboots into the server OS (oh that was easy!)<br />
<a href="http://www.flickr.com/photos/90048824@N00/6167242047/" title="4-orchestra-login by Ahmed-Kamal, on Flickr"><img alt="4-orchestra-login" height="416" src="http://farm7.static.flickr.com/6151/6167242047_6bd5e2f693.jpg" width="500" /></a><br />
Now I configure eth1 to have a static IP address of 192.168.77.1/24 (I made any address up), here is a snapshot of /etc/network/interfaces and I started eth1 using ifup<br />
<a href="http://www.flickr.com/photos/90048824@N00/6167777170/" title="5-orchestra-eth1up by Ahmed-Kamal, on Flickr"><img alt="5-orchestra-eth1up" height="416" src="http://farm7.static.flickr.com/6167/6167777170_6717546dce.jpg" width="500" /></a><br />
At this stage, I rebooted the server but you definitely don't have to. Let's start actually installing Orchestra<br />
<br />
<pre>sudo apt-get update
sudo apt-get install ubuntu-orchestra-server -y
</pre><br />
Everything proceeds automatically, for any question you get during package installation, I'll provide a picture with the answer :)<br />
<a href="http://www.flickr.com/photos/90048824@N00/6167242113/" title="6-cobbler-password by Ahmed-Kamal, on Flickr"><img alt="6-cobbler-password" height="416" src="http://farm7.static.flickr.com/6174/6167242113_0553b191da.jpg" width="500" /></a><br />
<a href="http://www.flickr.com/photos/90048824@N00/6167242157/" title="7-nextserver by Ahmed-Kamal, on Flickr"><img alt="7-nextserver" height="416" src="http://farm7.static.flickr.com/6168/6167242157_f4587a6af6.jpg" width="500" /></a><br />
<a href="http://www.flickr.com/photos/90048824@N00/6167777288/" title="8-enable-dns-dhcp by Ahmed-Kamal, on Flickr"><img alt="8-enable-dns-dhcp" height="416" src="http://farm7.static.flickr.com/6166/6167777288_4a69c2a9d1.jpg" width="500" /></a><br />
<a href="http://www.flickr.com/photos/90048824@N00/6167777326/" title="9-dhcp-range by Ahmed-Kamal, on Flickr"><img alt="9-dhcp-range" height="416" src="http://farm7.static.flickr.com/6172/6167777326_8fae8c2bd1.jpg" width="500" /></a><br />
<a href="http://www.flickr.com/photos/90048824@N00/6167242275/" title="10-dhcp-gw by Ahmed-Kamal, on Flickr"><img alt="10-dhcp-gw" height="416" src="http://farm7.static.flickr.com/6152/6167242275_ae23aaac7a.jpg" width="500" /></a><br />
<a href="http://www.flickr.com/photos/90048824@N00/6167242329/" title="11-domain-name by Ahmed-Kamal, on Flickr"><img alt="11-domain-name" height="416" src="http://farm7.static.flickr.com/6171/6167242329_dbac14a4e7.jpg" width="500" /></a><br />
<br />
That's it! You've just installed and configured your first Ubuntu Orchestra server, and you're now ready to install a fleet of Ubuntu servers the easy way! In part 2 of this article, I'll go through creating a second server, PXE booting and installing it from the orchestra server. (Extra credit: If you can't wait, try PXE booting a fresh server right now. Note that after installation, orchestra actually downloads and auto-imports a few Ubuntu mini ISOs, thus will need a few minutes depending on your internet connection speed)<br />
<br />
So, what do you think of this coolness? Is this easier than the last time you tried building yourself an automated network installer? Shoot me a comment, let me know what you think</div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com1tag:blogger.com,1999:blog-342921498109425004.post-58659657939227395322011-08-08T16:39:00.000+02:002011-08-08T16:39:47.342+02:00Ensemble meets Hadoop on the cloud<a href="http://hadoop.apache.org/"><img alt="Hadoop" src="http://hadoop.apache.org/images/hadoop-logo.jpg" title="Hadoop" class="aligncenter" width="300" height="71" /></a><br />
<br />
So you wanted to play with hadoop to crunch on some big-data problems, except that, well getting a hadoop cluster up and running in not exactly a one minute thing! Let me show you how to make it "a one minute thing" using Ensemble! Since Ensemble now has formulas for creating hadoop <a href="https://code.launchpad.net/~negronjl/+junk/hadoop-master">master</a> and <a href="https://code.launchpad.net/~negronjl/+junk/hadoop-slave">slave</a> nodes, spinning up a hadoop cluster could not be easier! Check this video out<br />
<br />
<object width="425" height="349"><param name="movie" value="http://www.youtube.com/v/e8IKkWJj7bA?version=3&hl=en_US&rel=0"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/e8IKkWJj7bA?version=3&hl=en_US&rel=0" type="application/x-shockwave-flash" width="425" height="349" allowscriptaccess="always" allowfullscreen="true"></embed></object><br />
If you can't see the embedded video, here's a direct link <a href="http://youtu.be/e8IKkWJj7bA">http://youtu.be/e8IKkWJj7bA</a><br />
<br />
Yep that's how simple it is! If you want to scale-out the cluster, you only need to ask Ensemble to do it for you:<br />
$ ensemble add-unit hadoop-slave<br />
<br />
So is this easier than configuring a hadoop cluster manually? Leave me a comment, let me know your thoughts! Also let me know what you'd like to see deployed next with EnsembleKamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-74340027302799032382011-07-29T16:44:00.002+02:002011-07-29T19:01:57.372+02:00Ubuntu takes UFOs to the cloud<div dir="ltr" style="text-align: left;" trbidi="on"><div class="separator" style="float: right; text-align: right;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQNPxazevQtPxCkDmggMpiTITZy6qPHSIFDYg5j0XSFZeJb1RsKMJk3bgyW9ecejCiuXNa5dcg7aWMP5b03aCHXmZAvJ1zoTVc_st-tPcEuhKBr1hIMil4OE_Gjuc35QHsSVRPUW2O1HA/s1600/UFOold.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="313" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQNPxazevQtPxCkDmggMpiTITZy6qPHSIFDYg5j0XSFZeJb1RsKMJk3bgyW9ecejCiuXNa5dcg7aWMP5b03aCHXmZAvJ1zoTVc_st-tPcEuhKBr1hIMil4OE_Gjuc35QHsSVRPUW2O1HA/s400/UFOold.jpg" width="400" /></a></div><br />
<div dir="ltr" style="text-align: left;" trbidi="on">I've always believed in <a href="http://en.wikipedia.org/wiki/UFO">UFOs</a> as a kid, and while I've never seen one (yet?) I am still more on the believer side! I was interested to stumble upon a database of UFO sightings at <a href="http://www.infochimps.com/tags/ufo#">http://www.infochimps.com/tags/ufo#</a> A shout-out at infochimps (you guys are great!). Downloading the sightings DB (around 80MB), I found a listing of 60,000 documented sightings, hmm interesting! I started thinking I could crunch on this data in some useful and fun way, what about finding the most commonly spotted UFO shape?! Sounds like I could use hadoop for that, just for the coolness factor really, the data is not that large anyway, but hey why not! I had no-idea how to get started with hadoop though and wasn't really interested in learning up all the gory details!<br />
<br />
Well <a href="https://ensemble.ubuntu.com/">Ensemble</a> to the rescue, hadoop master and slave formulas exist, which means someone else packaged the knowledge needed to setup and run a hadoop cluster for me. All I needed to do was ask Ensemble to deploy me a couple of cloud instances and start playing. Let's see how you can do that for yourself<br />
<br />
I won't repeat the instructions to <a href="https://ensemble.ubuntu.com/docs/getting-started.html">get started with Ensemble</a>, since the documentation is a good place for that (and it's so easy anyway!). If you feel you need more help there, this little <a href="http://www.youtube.com/user/ubuntucloud#p/a/u/1/qxMhKbDSbOw">video</a> should be helpful. If you're still stuck, you can always drop by on irc/freenode at #ubuntu-ensemble and ask your questions<br />
<br />
<h2>Hadoop node, with an extra slave please</h2><br />
So, let's start ensembling<br />
<pre>bzr branch lp:~negronjl/+junk/hadoop-master
bzr branch lp:~negronjl/+junk/hadoop-slave
ensemble bootstrap
</pre>wait a minute or two for EC2 to spin up the instance, then<br />
<pre>ensemble status
</pre>which’ll give you output like<br />
<pre>$ ensemble status
2011-07-12 15:20:54,978 INFO Connecting to environment.
The authenticity of host 'ec2-50-17-28-19.compute-1.amazonaws.com (50.17.28.19)' can't be established.
RSA key fingerprint is c5:21:62:f0:ac:bd:9c:0f:99:59:12:ec:4d:41:48:c8.
Are you sure you want to continue connecting (yes/no)? yes
machines:
0: {dns-name: ec2-50-17-28-19.compute-1.amazonaws.com, instance-id: i-8bc034ea}
services: {}
2011-07-12 15:21:01,205 INFO 'status' command finished successfully
</pre><br />
Now let's deploy a two node hadoop cluster<br />
<pre>ensemble deploy --repository . hadoop-master
ensemble deploy --repository . hadoop-slave
ensemble add-relation hadoop-master hadoop-slave
</pre><br />
Yeah it's that easy! Ensemble formulas manage all the kung-fu for you. The hadoop cluster is ready, let's ssh into the master node and switch to user hdfs<br />
<pre>ensemble ssh hadoop-master/0
sudo -su hdfs
</pre><br />
<h2>Downloading UFOs</h2><br />
Download the infochimps sightings database here, unzip it and locate the TSV file (tab separated values) file. Note that you can download the file from infochimps without registering on their website (didn't I say these guys were great :)<br />
<br />
Upload the TSV DB to hadoop's distributed filesystem<br />
<pre>hadoop dfs -copyFromLocal ufo_awesome.tsv ufo_awesome.tsv
</pre><br />
Almost ready, the corpus has been uploaded. Now we need to write some map/reduce jobs to do the actual crunching. Not being a pro developer, the thought of writing that in java was like (oh no ew), so python to the rescue! Thanks to the great instructions at <a href="http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/">Michael Noll</a>'s blog, I was able to massage some of that code to get it to do what I wanted. I pushed my code to <a href="https://code.launchpad.net/~kim0/+junk/ufo-ensemble-cruncher">launchpad</a>, so that you can grab it directly from the hadoop master node<br />
<br />
<pre>cd /tmp
bzr branch lp:~kim0/+junk/ufo-ensemble-cruncher
cd ufo-ensemble-cruncher
</pre><br />
<h2>Unleashing the elephant</h2><br />
Now for the big moment, let's launch the elephant<br />
<pre>hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-*.jar -file ./mapper.py -mapper mapper.py -file ./reducer.py -reducer reducer.py -input ufo_awesome.tsv -output ufo-output
packageJobJar: [./mapper.py, ./reducer.py, /tmp/hadoop-hdfs/hadoop-unjar1418682529553378062/] [] /tmp/streamjob5701745574334998473.jar tmpDir=null
11/07/29 12:27:52 INFO mapred.FileInputFormat: Total input paths to process : 1
11/07/29 12:27:53 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-hdfs/mapred/local]
11/07/29 12:27:53 INFO streaming.StreamJob: Running job: job_201107290935_0010
11/07/29 12:27:53 INFO streaming.StreamJob: To kill this job, run:
11/07/29 12:27:53 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=domU-12-31-39-10-81-8E.compute-1.internal:8021 -kill job_201107290935_0010
11/07/29 12:27:53 INFO streaming.StreamJob: Tracking URL: http://domU-12-31-39-10-81-8E.compute-1.internal:50030/jobdetails.jsp?jobid=job_201107290935_0010
11/07/29 12:27:54 INFO streaming.StreamJob: map 0% reduce 0%
11/07/29 12:28:11 INFO streaming.StreamJob: map 10% reduce 0%
11/07/29 12:28:12 INFO streaming.StreamJob: map 19% reduce 0%
11/07/29 12:28:14 INFO streaming.StreamJob: map 72% reduce 0%
11/07/29 12:28:16 INFO streaming.StreamJob: map 100% reduce 0%
11/07/29 12:28:33 INFO streaming.StreamJob: map 100% reduce 100%
11/07/29 12:28:37 INFO streaming.StreamJob: Job complete: job_201107290935_0010
11/07/29 12:28:37 INFO streaming.StreamJob: Output: ufo-output
</pre><br />
Woohoo success! Now let's grab the results, sorting it to easily see the most popular sighting shape<br />
<br />
<h2>Is the answer really 42</h2><br />
<pre>hadoop dfs -cat ufo-output/part-00000 | sort -k 2,2 -nr
light 12202
triangle 6082
circle 5271
disk 4825
other 4593
unknown 4490
sphere 3637
fireball 3452
oval 2869
formation 1788
cigar 1782
changing 1546
flash 990
cylinder 982
rectangle 966
diamond 915
chevron 760
egg 664
teardrop 595
cone 265
cross 177
delta 8
round 2
crescent 2
pyramid 1
hexagon 1
flare 1
dome 1
changed 1
</pre><br />
The answer is "light" then! Wow that was a blast! I had fun doing this exercise. Now I am no hadoop expert in any way (so direct those hadoopy questions to someone who can actually answer them), however I was quite pleased Ensemble could help me get up and running that fast. The Ensemble community is doing a great job wrapping many free software with formulas, such that you can always get up and running with any app you need in seconds rather than days (months?). <b>You</b> too can write Ensemble formulas for your favorite (server?) application. Hop on to #ubuntu-ensemble and grab me (kim0) or any of the dev team and ask any questions on your mind! We're a happy community<br />
<br />
So was that fun? Can you think of something cooler you want to see done? Leave me a comment, let me know about it<br />
<br />
</div></div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com4tag:blogger.com,1999:blog-342921498109425004.post-26968168736853228492011-05-11T11:33:00.000+02:002011-05-11T11:33:52.293+02:00Cloud Portal one-click launch<div dir="ltr" style="text-align: left;" trbidi="on">Announcing two nice little additions to the <a href="http://cloud.ubuntu.com/ami/">cloud portal AMI tool</a> that should make everyone's cloud life a little easier<br />
<br />
<ul style="text-align: left;"><li>Amazon now allows passing parameters to the AWS console to basically be able to choose the region and AMI-ID to launch. This is now integrated in the cloud portal so that you search for the ami, click the link and it's pre-selected, ready to be launched in AWS console. Here's a screenshot</li>
</ul><br />
<a href="http://www.flickr.com/photos/90048824@N00/5709230381/" title="cloud-portal-one-click-launch by Ahmed-Kamal, on Flickr"><img alt="cloud-portal-one-click-launch" height="94" src="http://farm3.static.flickr.com/2245/5709230381_1d27f334a3_z.jpg" width="640" /></a><br />
<br />
<br />
<ul style="text-align: left;"><li>As that image shows as well, you can search now for "cluster" to get Ubuntu cloud cluster-ready images!</li>
</ul><div>Interested to add any features? Have any thoughts or comments, let me know, leave me a word</div></div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-90982854481676439102011-04-18T14:14:00.000+02:002011-04-18T14:14:48.867+02:00Help Squash Natty Server Bugs<div dir="ltr" style="text-align: left;" trbidi="on"><div>Mirroring a post from Ubuntu Server developer <a href="http://blog.daviey.com/">Dave Walker</a>, If you're capable and willing to help squash some server bugs, your effort and contribution is really needed now! Join us on irc <b>#ubuntu-server</b>, ask around if you need help. Thanks in advance</div><div><br />
</div><div>Here is a list of current bugs that should hopefully be fixed for Natty release. The bugs with "Not assigned to anyone :(", could really do with someone volunteering to take. Please feel free to assign yourself, if you are willing to take on working towards a fix. </div><div><br />
</div><div>If you are an assignee, please make sure that the current status (and comments) accurately represent the status before tomorrows server team meeting. </div><div><br />
</div><div>== Bugs for the server team == </div><div>[655533] [likewise-open] [master] package likewise-open 5.4.0.42111-2ubuntu2 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1 (<a href="http://pad.lv/655533">http://pad.lv/655533</a>) </div><div> * Bug Status: Confirmed </div><div> * Not assigned to anyone :( </div><div> * Last updated: 2011-04-18 </div><div>[726769] [eucalyptus] package eucalyptus-common 2.0.1 bzr1255-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1 (<a href="http://pad.lv/726769">http://pad.lv/726769</a>) </div><div> * Bug Status: Confirmed </div><div> * Assigned to: Dave Walker </div><div> * Last updated: 2011-04-18 </div><div>[661453] [dovecot] dovecot.conf always shows as having been locally modified on update (<a href="http://pad.lv/661453">http://pad.lv/661453</a>) </div><div> * Bug Status: Confirmed </div><div> * Assigned to: Clint Byrum </div><div> * Last updated: 2011-04-18 </div><div>[745946] [cloud-init] cloud-final job did not run in ec2-automated-tests (<a href="http://pad.lv/745946">http://pad.lv/745946</a>) </div><div> * Bug Status: Confirmed </div><div> * Not assigned to anyone :( </div><div> * Last updated: 2011-04-18 </div><div>[759943] [mod-wsgi] mod_wsgi.so-3.2 gives errors (<a href="http://pad.lv/759943">http://pad.lv/759943</a>) </div><div> * Bug Status: Confirmed </div><div> * Not assigned to anyone :( </div><div> * Last updated: 2011-04-15 </div><div>[764391] [cobbler] cobbler fails to manage bind9 (<a href="http://pad.lv/764391">http://pad.lv/764391</a>) </div><div> * Bug Status: Confirmed </div><div> * Assigned to: James Page </div><div> * Last updated: 2011-04-18 </div><div>[745930] [cloud-init] cloud-init timeout waiting for metadata service on EC2 (<a href="http://pad.lv/745930">http://pad.lv/745930</a>) </div><div> * Bug Status: New </div><div> * Not assigned to anyone :( </div><div> * Last updated: 2011-04-15 </div><div>[760288] [Ubuntu] JeOS is oversized (<a href="http://pad.lv/760288">http://pad.lv/760288</a>) </div><div> * Bug Status: Confirmed </div><div> * Not assigned to anyone :( </div><div> * Last updated: 2011-04-15 </div><div><br />
</div><div> == Bugs being worked on in other areas, more for information == </div><div>[728088] [debian-installer] iscsi root with or without auth fails to boot (<a href="http://pad.lv/728088">http://pad.lv/728088</a>) </div><div> * Bug Status: Confirmed </div><div> * Assigned to: Colin Watson </div><div> * Last updated: 2011-04-14 </div><div>[747090] [linux] wrong return address sometimes pushed for INT in kvm (not qemu) (<a href="http://pad.lv/747090">http://pad.lv/747090</a>) </div><div> * Bug Status: Fix Committed </div><div> * Assigned to: Andy Whitcroft </div><div> * Last updated: 2011-04-18 </div><div>[566818] [plymouth] Cryptsetup passphrase prompt during boot: every character typed repeats the prompt (<a href="http://pad.lv/566818">http://pad.lv/566818</a>) </div><div> * Bug Status: Confirmed </div><div> * Assigned to: Surbhi Palande </div><div> * Last updated: 2011-04-16 </div><div>[580319] [upstart] dhcp3-server launches before upstart brings all interface, thus failing to start (<a href="http://pad.lv/580319">http://pad.lv/580319</a>) </div><div> * Bug Status: Triaged </div><div> * Assigned to: James Hunt </div><div> * Last updated: 2011-04-18 </div><div>[759545] [grub2] user prompted to update unmodified grub configuration during Ubuntu server upgrade (<a href="http://pad.lv/759545">http://pad.lv/759545</a>) </div><div> * Bug Status: New </div><div> * Last updated: 2011-04-18 </div></div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-24840509040275963422011-04-12T18:38:00.000+02:002011-04-12T18:38:49.392+02:00Ensemble Cloud Community Meeting<div dir="ltr" style="text-align: left;" trbidi="on"><span class="Apple-style-span" style="color: #333333; font-family: Ubuntu, 'Ubuntu Beta', UbuntuBeta, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; line-height: 16px;"><a href="https://wiki.ubuntu.com/Ensemble" style="clear: none; color: #dd4814; font-family: Ubuntu, 'Ubuntu Beta', UbuntuBeta, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; outline-color: initial; outline-style: none; outline-width: initial; text-decoration: none;">Ensemble</a> is an exciting new cloud technology innovated by Ubuntu. If you’re curious what it does (I bet you are) it is a service orchestration framework with focus on rapid cloud deployment. That is to say, it allows one to rapidly deploy cloud instances, bootstrap them, install software on them, configure them as well as link them all together to provide you (as a sys-admin or DevOp) with an end-user facing high-level “service” (Load Balanced CMS with sharded Database, Multinode email service installation…etc). Ensemble has been compared to being APT for the cloud! Now does that sound interesting, it sure is if you ask me! Welcome to next generation Linux infrastructure automation.</span><div><span class="Apple-style-span" style="color: #333333; font-family: Ubuntu, 'Ubuntu Beta', UbuntuBeta, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; line-height: 16px;"><br />
</span></div><div><span class="Apple-style-span" style="color: #333333; font-family: Ubuntu, 'Ubuntu Beta', UbuntuBeta, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif;"><span class="Apple-style-span" style="line-height: 16px;">If that got you interested, be sure to attend the <b>weekly IRC meeting on Wed 18:00-UTC on #ubuntu-cloud</b>. </span></span></div><div><span class="Apple-style-span" style="color: #333333; font-family: Ubuntu, 'Ubuntu Beta', UbuntuBeta, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif;"><span class="Apple-style-span" style="line-height: 16px;">Read More at: </span></span><a href="http://cloud.ubuntu.com/2011/04/ensemble-cloud-community-meeting/">http://cloud.ubuntu.com/2011/04/ensemble-cloud-community-meeting/</a></div><div>Feel free to tweet/dent and spread this around the globe. Got feedback? Leave me comments on cloud.ubuntu.com or on my blog</div></div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-22195845516726311962011-04-11T14:31:00.000+02:002011-04-11T14:31:52.943+02:00Help Shape Maps.Ubuntu.com<div dir="ltr" style="text-align: left;" trbidi="on">About either months ago, I had started on a fun little project to raise awareness of how widely used Ubuntu Server is around the world. The result was <a href="http://maps.ubuntu.com/">http://maps.ubuntu.com/</a> a web tool allowing anyone using Ubuntu Server to "mark" his location on a global map. This was launched with celebrations for 10.04.1, and thousands of cities were marked (woohoo). The project code was open-sourced and lives at <a href="https://launchpad.net/mapuntu/trunk">https://launchpad.net/mapuntu/trunk</a><br />
<br />
It's been a while since the project was last updated, a few days ago, an Ubuntu community member ( hey <a href="https://wiki.ubuntu.com/ronnie.vd.c">Ronnie</a> ) stepped up to re-shape the <a href="https://launchpad.net/mapuntu/trunk">mapuntu</a> project and relaunch it into an <b>inspiring map that helps anyone record and locate Ubuntu events around the globe!</b> The first part of that effort is creating a proper <a href="https://wiki.ubuntu.com/mapuntu/API">back-end API</a>, everyone is encouraged to click that link and leave comments. If you are inspired by that project, and would like to help, please get involved! Things you can do to help include<br />
<br />
<ul style="text-align: left;"><li>Review and enhance the proposed API</li>
<li>Help write python code to implement the backend api</li>
<li>Write html/css/js for the front-end</li>
<li>Play with the code, test it, report bugs, adapt to your own website!</li>
</ul><br />
<br />
If you want to talk more about this, Please join #ubuntu-locoteams on IRC and ping Ronnie or myself (kim0, also on #ubuntu-cloud).<br />
</div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com8tag:blogger.com,1999:blog-342921498109425004.post-18511286205806535872011-04-01T18:35:00.000+02:002011-04-01T18:35:59.187+02:00Creating Customized UEC and EC2 Images<div dir="ltr" style="text-align: left;" trbidi="on">Want to create a customized Ubuntu cloud image that you can deploy to EC2 or UEC ? No problem! Check out this 5 min video where I demo everything needed to download, customize, rebundle and publish the new image to EC2! Isn't this just great :)<br />
<br />
<object height="390" width="640"><param name="movie" value="http://www.youtube.com/v/K1IwTGLWbu0?fs=1&hl=en_US&rel=0&hd=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/K1IwTGLWbu0?fs=1&hl=en_US&rel=0&hd=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="390"></embed></object><br />
<br />
If you can't see the embedded player, here's a direct link: <a href="http://www.youtube.com/watch?v=K1IwTGLWbu0">http://www.youtube.com/watch?v=K1IwTGLWbu0</a><br />
</div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-40396148615436462042011-03-25T15:03:00.000+02:002011-03-25T15:03:42.992+02:00Ubuntu Cloud Days, Day-2 Summary<div dir="ltr" style="text-align: left;" trbidi="on">Yesterday was the second and final day of cloud days. It was such a blast! Very interesting stuff, and very cool community to be part of! In case you missed it, you can still read up the logs, and you can reach me and others for questions and comments on #ubuntu-cloud .. Let's quickly zip through yesterday's sessions<br />
<br />
<a href="http://irclogs.ubuntu.com/2011/03/24/%23ubuntu-classroom.html#t16:00">16.00 UTC smoser</a> gave a very good introduction to the different ways with which you can rebundle UEC or EC2 images for Ubuntu, how you can customize them and the different pros and cons. Scott also answered lots of questions<br />
<a href="http://irclogs.ubuntu.com/2011/03/24/%23ubuntu-classroom.html#t17:00">17.00 UTC tetet</a> started by presenting some interesting new features and tricks that can be used with UEC namely persistency, which means kexec'ing the kernel to load from an EBS volume resulting in persistent UEC instances. Thorsten zipped through the session quickly, but as a pro instructor, he was very well prepared with links to graphics, links and documentation<br />
<a href="http://irclogs.ubuntu.com/2011/03/24/%23ubuntu-classroom.html#t18:00">18.00 UTC Daviey</a> wowed the crowd with a puppet introductory session, he showed how to configure apache with puppet and what it takes and deploy a basic system to get running fast. It was a great session<br />
<a href="http://irclogs.ubuntu.com/2011/03/24/%23ubuntu-classroom.html#t19:00">19.00 UTC edulix</a> took over to present an awesome session about using hadoop and its map/reduce programming model to crunch on large (we're talking terabytes) of data on lots of machines. Very interesting and useful session<br />
<a href="http://irclogs.ubuntu.com/2011/03/24/%23ubuntu-classroom.html#t20:00">20.00 UTC obino</a> had the final say on cloud days. The session was about UEC/Eucalyptus. He presented a detailed architecture of the system, what each node does, how things work. He also explained how to get started contributing to the eucalyptus community. <div><br />
</div><div>It was a great event. Big Thank-Yous to everyone who presented in UCD. Till next time o/</div></div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-62692852515926648832011-03-24T15:23:00.000+02:002011-03-24T15:23:55.470+02:00Day-2 Ubuntu Cloud Days<div dir="ltr" style="text-align: left;" trbidi="on"><span class="Apple-style-span" style="font-family: inherit;">Yesterday was great! It was Ubuntu Cloud Days, the event was packed with very interesting sessions, hundreds of attendees, lots of questions got answered, tons of ideas and comments were made. I personally had lots of fun, and was </span>literally<span class="Apple-style-span" style="font-family: inherit;"> stuck to my keyboard the full time of the event</span><br />
<span class="Apple-style-span" style="font-family: inherit;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: inherit;">Let's zip through what happened yesterday (If you missed it, Click the time to read the logs!)</span><br />
<a href="http://irclogs.ubuntu.com/2011/03/23/%23ubuntu-classroom.html#t16:00">16.00 UTC kim0</a><span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="line-height: 16px;"> (me) took-over, gave a quick introduction about cloud computing, differences between public and private clouds and what the cloud fuss is all about</span></span><br />
<span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="line-height: 16px;"></span></span><a href="http://irclogs.ubuntu.com/2011/03/23/%23ubuntu-classroom.html#t17:00">17.00 UTC semiosis</a><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="line-height: 16px;"> started a great session about building scalable distributed storage on the EC2 Cloud using Glusterfs for improving redundancy, and performance</span></span></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="line-height: 16px;"></span></span></span><a href="http://irclogs.ubuntu.com/2011/03/23/%23ubuntu-classroom.html#t18:00">18.00 UTC SpamapS</a><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="line-height: 16px;"> gave a cool presentation and hands-on demo of a new Ubuntu ground breaking "service" management framework called Ensemble. While in early development phase, it's already working well and is changing how we look at server management</span></span></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="line-height: 16px;"></span></span></span><a href="http://irclogs.ubuntu.com/2011/03/23/%23ubuntu-classroom.html#t19:00">19.00 UTC hallyn</a><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="line-height: 16px;"> took over presenting a technically deep discussion and hands-on demo about LXC Linux containers technology. This is a light weight Linux virtualization (or container) technology</span></span></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="line-height: 16px;"></span></span></span><a href="http://irclogs.ubuntu.com/2011/03/23/%23ubuntu-classroom.html#t19:59">20.00 UTC soren</a><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="line-height: 16px;"> took over to talk about OpenStack, one of the cloud stacks that Ubuntu ships with 11.04 that's been getting lots of cloud people excited lately. Soren explained how the whole thing got started, explained the architecture, answered lots of questions and discussed what we should be seeing in the future</span></span></span><br />
<div style="clear: none; color: #333333; outline-color: initial; outline-style: none; outline-width: initial;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="line-height: 16px;"><br />
</span></span></div><div style="clear: none; color: #333333; outline-color: initial; outline-style: none; outline-width: initial;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333; line-height: 16px;">Here's a sneak peak of today's sessions</span></span></div><div style="clear: none; color: #333333; line-height: 25px; outline-color: initial; outline-style: none; outline-width: initial;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333; line-height: 16px;"><strong style="clear: none; color: #333333; outline-color: initial; outline-style: none; outline-width: initial;"><br />
</strong></span><span class="Apple-style-span" style="color: #333333; line-height: 16px;"><strong style="clear: none; color: #333333; outline-color: initial; outline-style: none; outline-width: initial;">Thu 24th Mar</strong></span><br />
<span class="Apple-style-span" style="color: #333333; line-height: 16px;"><strong style="clear: none; color: #333333; outline-color: initial; outline-style: none; outline-width: initial;">16.00 UTC</strong> rebundling/re-using Ubuntu’s UEC images — smoser</span><br />
<span class="Apple-style-span" style="color: #333333; line-height: 16px;"><strong style="clear: none; color: #333333; outline-color: initial; outline-style: none; outline-width: initial;">17.00 UTC</strong> UEC persistency — tetet</span><br />
<span class="Apple-style-span" style="color: #333333; line-height: 16px;"><strong style="clear: none; color: #333333; outline-color: initial; outline-style: none; outline-width: initial;">18.00 UTC</strong> Puppet Configuration Management — Daviey</span><br />
<span class="Apple-style-span" style="color: #333333; line-height: 16px;"><strong style="clear: none; color: #333333; outline-color: initial; outline-style: none; outline-width: initial;">19.00 UTC</strong> Using hadoop, divide and conquer — edulix</span><br />
<span class="Apple-style-span" style="color: #333333; line-height: 16px;"><strong style="clear: none; color: #333333; outline-color: initial; outline-style: none; outline-width: initial;">20.00 UTC</strong> UEC/Eucalyptus Private Cloud — obino</span></span></div><div><br />
</div><div>Let's get crunching on even more cool sessions today! If you’re new to IRC, you can simply use this <a href="http://webchat.freenode.net/?channels=ubuntu-classroom%2Cubuntu-classroom-chat">web page</a> to join. For more information, check out the <a href="https://wiki.ubuntu.com/UbuntuCloudDays">wiki page</a>, also feel free to ping “kim0″ on IRC!</div></div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-3294999778009540072011-03-21T14:10:00.000+02:002011-03-21T14:10:02.824+02:00Ubuntu Cloud Days approaching<div dir="ltr" style="text-align: left;" trbidi="on">Just a reminder, Ubuntu Cloud Days is approaching, it's only two days away! Yes it's this Wed/Thu. I just posted the full details on the <a href="http://cloud.ubuntu.com/2011/03/ubuntu-cloud-days-approaching/">Ubuntu Cloud Portal</a>. Check it out, and leave me a comment if you have any questions</div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-63844059209666566412011-03-09T14:57:00.000+02:002011-03-09T14:57:41.571+02:00Cloud Python hackers wanted<div dir="ltr" style="text-align: left;" trbidi="on">A while back I had <a href="http://foss-boss.blogspot.com/2011/01/hack-on-ubuntu-cloud-utils.html">blogged</a> (yes the one with the big uncle Sam pic) asking for people interested in working on a python tool to help migrate ebs AMIs across ec2 regions. Since then the project has progressed a bit:<br />
<br />
- Project page is now: <a href="https://launchpad.net/ec2-migrate-ebs-ami">https://launchpad.net/ec2-migrate-ebs-ami</a><br />
- Project is now able to launch two utility instances in the two specified regions, and mount the source volume in the source instance<br />
- What remains is attaching a destination volume, and sync'ing over contents. More details can be found on the blueprint: <a href="https://blueprints.launchpad.net/ubuntu/+spec/cloud-server-n-ec2-migrate-region">https://blueprints.launchpad.net/ubuntu/+spec/cloud-server-n-ec2-migrate-region</a><br />
<br />
Development pace has now slowed down significantly, as such I am renewing the call. This is a great way to start getting involved with cloud development, and to join the Ubuntu Cloud and server communities! joining in is "easy", you don't even have to know lots of python to start hacking! The steps are very well defined, and I am happy to help with all steps along the way. If you are thinking about it, go ahead and get in touch with me right now (kim0 on irc, kim0 AT ubuntu.com) or join today's irc meeting (6pm-UTC #ubuntu-cloud <a href="https://wiki.ubuntu.com/UbuntuCloudMeeting">https://wiki.ubuntu.com/UbuntuCloudMeeting</a>) and start discussing how you can get involved</div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com4tag:blogger.com,1999:blog-342921498109425004.post-88089707694131360632011-03-02T18:44:00.000+02:002011-03-02T18:44:15.620+02:00FAST Win7 KVM VirtIO{Disk-Net} InstallEveryone who has to use Windows, please let's at least keep it contained inside a virtual machine! In this article I'll demo how to "Install Windows7 over Ubuntu 11.04 Natty, using KVM with System Disk over VirtIO". Quoting the libvirt wiki "<i>Virtio is a Linux standard for network and disk device drivers where just the guest's device driver "knows" it is running in a virtual environment, and cooperates with the hypervisor. This enables guests to get high performance network and disk operations, and gives most of the performance benefits of paravirtualization</i>"<br />
<br />
Here's a little convenience script to download virtio Windows drivers and launch virt-install to Install Windows. Copy/Paste this script into an empty folder, chmod +x it, and run it. <br />
<style type="text/css">
<!--
pre { font-family: monospace; color: #000000; background-color: #ffffff; }
body { font-family: monospace; color: #000000; background-color: #ffffff; }
.PreProc { color: #a020f0; }
.Constant { color: #ff00ff; }
.Identifier { color: #008b8b; }
.Comment { color: #0000ff; }
-->
</style><br />
<pre><span class="Comment">#!/bin/sh</span>
<span class="Identifier">WINISO</span>=/path/to/win7.iso <span class="Comment">#Windows ISO</span>
<span class="Identifier">INSTALLDISK</span>=win7virtio.img <span class="Comment">#Disk location. Can be LVM LV</span>
<span class="Identifier">VFD</span>=<a href="http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win">http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win</a><span class="Constant">-1</span>.<span class="Constant">1</span>.<span class="Constant">16</span>.vfd
<span class="Identifier">DRVRISO</span>=<a href="http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win">http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win</a><span class="Constant">-1</span>.<span class="Constant">1</span>.<span class="Constant">16</span>.iso
<span class="Statement">[</span> <span class="Statement">-e</span> <span class="Error">$(</span><span class="Special">basename </span><span class="PreProc">$VFD</span><span class="Error">)</span> <span class="Statement">]</span> || wget <span class="PreProc">$VFD</span>
<span class="Statement">[</span> <span class="Statement">-e</span> <span class="Error">$(</span><span class="Special">basename </span><span class="PreProc">$DRVRISO</span><span class="Error">)</span> <span class="Statement">]</span> || wget <span class="PreProc">$DRVRISO</span>
<span class="Statement">[</span> <span class="Statement">-e</span> <span class="PreProc">$INSTALLDISK</span> <span class="Statement">]</span> || qemu-img create <span class="PreProc">$INSTALLDISK</span> 30G
sudo virt-install <span class="Special">-c</span> qemu:///system <span class="Special">--virt-type</span> kvm <span class="Special">--name</span> win7virtio <span class="Special">--ram</span> <span class="Constant">1024</span> <span class="Special">--disk</span> <span class="Identifier">path</span>=<span class="Statement">"</span><span class="PreProc">$INSTALLDISK</span><span class="Statement">"</span>,<span class="Identifier">bus</span>=virtio <span class="Statement">\</span>
--disk <span class="Error">$(</span><span class="Special">basename </span><span class="PreProc">$VFD</span><span class="Error">)</span>,<span class="Identifier">device</span>=floppy <span class="Special">--os-variant</span> win7 <span class="Special">--cdrom</span> <span class="Error">$(</span><span class="Special">basename </span><span class="PreProc">$DRVRISO</span><span class="Error">)</span> <span class="Special">--cdrom</span> <span class="Statement">"</span><span class="PreProc">$WINISO</span><span class="Statement">"</span> <span class="Special">--vcpus</span> <span class="Constant">2</span>
</pre><br />
et voila Windows installer kicks in, we're greeted with the usual<br />
<br />
<a href="http://www.flickr.com/photos/90048824@N00/5491828910/" title="Screenshot-win7virtio - Virt Viewer by Ahmed-Kamal, on Flickr"><img src="http://farm6.static.flickr.com/5214/5491828910_668a552b19.jpg" width="500" height="402" alt="Screenshot-win7virtio - Virt Viewer" /></a><br />
<a href="http://www.flickr.com/photos/90048824@N00/5491828774/" title="Screenshot-win7virtio - Virt Viewer-1 by Ahmed-Kamal, on Flickr"><img src="http://farm6.static.flickr.com/5178/5491828774_b3877c374b.jpg" width="500" height="402" alt="Screenshot-win7virtio - Virt Viewer-1" /></a><br />
Next the welcome screen, and agree on the license agreement (if you want!)<br />
<a href="http://www.flickr.com/photos/90048824@N00/5491234647/" title="Screenshot-win7virtio - Virt Viewer-2 by Ahmed-Kamal, on Flickr"><img src="http://farm6.static.flickr.com/5014/5491234647_478f19d938.jpg" width="500" height="410" alt="Screenshot-win7virtio - Virt Viewer-2" /></a><br />
<br />
In this next screenshot, we see how Windows Installer does not see any "disks". This is because Windows now does not have any drivers for the virtio disk that we attached!<br />
<a href="http://www.flickr.com/photos/90048824@N00/5491234391/" title="Screenshot-win7virtio - Virt Viewer-3 by Ahmed-Kamal, on Flickr"><img src="http://farm6.static.flickr.com/5135/5491234391_5b8d5cf02a.jpg" width="500" height="410" alt="Screenshot-win7virtio - Virt Viewer-3" /></a><br />
<br />
Let's fix that, clicking "Load Driver" and clicking OK, Windows starts reading the VFD virtual floppy image we attached, and detects the driver<br />
<a href="http://www.flickr.com/photos/90048824@N00/5491234391/" title="Screenshot-win7virtio - Virt Viewer-3 by Ahmed-Kamal, on Flickr"><img src="http://farm6.static.flickr.com/5135/5491234391_5b8d5cf02a.jpg" width="500" height="410" alt="Screenshot-win7virtio - Virt Viewer-3" /></a><br />
<a href="http://www.flickr.com/photos/90048824@N00/5491233895/" title="Screenshot-win7virtio - Virt Viewer-5 by Ahmed-Kamal, on Flickr"><img src="http://farm6.static.flickr.com/5139/5491233895_aa542d5538.jpg" width="500" height="410" alt="Screenshot-win7virtio - Virt Viewer-5" /></a><br />
Clicking Next, Windows can now see the virtio disk<br />
<a href="http://www.flickr.com/photos/90048824@N00/5491233895/" title="Screenshot-win7virtio - Virt Viewer-5 by Ahmed-Kamal, on Flickr"><img src="http://farm6.static.flickr.com/5139/5491233895_aa542d5538.jpg" width="500" height="410" alt="Screenshot-win7virtio - Virt Viewer-5" /></a><br />
Then it's business like usual<br />
<a href="http://www.flickr.com/photos/90048824@N00/5491233339/" title="Screenshot-win7virtio - Virt Viewer-7 by Ahmed-Kamal, on Flickr"><img src="http://farm6.static.flickr.com/5254/5491233339_b5aff1050c.jpg" width="500" height="410" alt="Screenshot-win7virtio - Virt Viewer-7" /></a><br />
After installation completes, be sure to check the device manager and confirm the disk (and network) are both using optimized virtio drivers<br />
<a href="http://www.flickr.com/photos/90048824@N00/5491826916/" title="DeviceManager by Ahmed-Kamal, on Flickr"><img src="http://farm6.static.flickr.com/5017/5491826916_175c79dc2e.jpg" width="385" height="496" alt="DeviceManager" /></a><br />
<br />
et voila, Windows in its full glory running over optimized virtio drivers. The only part that sucks is desktop VNC performance, something that I'll probably write about very soon :)Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com8tag:blogger.com,1999:blog-342921498109425004.post-69253760443724870372011-03-01T19:31:00.001+02:002011-03-01T19:32:11.534+02:00First Ever Ubuntu Cloud Days<div dir="ltr" style="text-align: left;" trbidi="on"><div class="separator" style="clear: both; text-align: center;"><a href="https://wiki.ubuntu.com/UbuntuCloudDays?action=AttachFile&do=get&target=UCD.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="49" src="https://wiki.ubuntu.com/UbuntuCloudDays?action=AttachFile&do=get&target=UCD.png" width="320" /></a></div><br />
I am very excited to announce the very first <b>Ubuntu Cloud Days</b>! UCD is an online event that is designed for everyone interested in cloud computing, as well as someone who is already using Ubuntu server and who's thinking about "that cloud thing" everyone is talking about. If the words "virtualization", "cloud", "server", "scalability" or "automation" intrigue and amuse you, then you should definitely attend! It's a chance to learn a lot, get your questions answered as well as share your experiences and have fun with the rest of the cloud community!<br />
<br />
In fact I'll go one bit further, not only should you attend, you should even deliver a session! Ubuntu as usual is all about community, learning from others, and sharing your knowledge to others. It's quite easy to deliver a session! My first session, I was quite surprised how I didn't have to prepare huge content to "fill" an hour, just answering questions and discussing with people, easily gets you by and you get a warm fuzzy feeling that you just helped lots of people and shared your knowledge.<br />
<br />
So, Ladies and Gentlemen, mark your calendars! March 23rd and March 24th<br />
<br />
Fore more information: <a href="https://wiki.ubuntu.com/UbuntuCloudDays/">https://wiki.ubuntu.com/UbuntuCloudDays/</a><br />
<br />
To contribute a session, (or if you're thinking about it) talk to me (kim0) on irc #ubuntu-cloud or email [ kim0 =AT= ubuntu.com ]</div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-43752145438854952892011-02-18T14:23:00.000+02:002011-02-18T14:23:22.834+02:00Volunteer QA to Ubuntu Server<div dir="ltr" style="text-align: left;" trbidi="on">A great way to help contribute to Ubuntu and ensure Ubuntu Server remains the rock solid platform that it is, is by committing to helping perform specific QA tests in predefined scenarios. If you're interested, the following test cases specifically need someone to step up and commit to them<br />
<ul style="text-align: left;"><li>Install (JeOS on ESX) <a href="http://testcases.qa.ubuntu.com/Install/ServerMinimalVirtualInstall">QA Testcase</a></li>
<li>Install (JeOS on KVM) <a href="http://testcases.qa.ubuntu.com/Install/ServerMinimalVirtualInstall">QA Testcase</a></li>
<li>iSCSI Authenticated Root Installation <a href="http://testcases.qa.ubuntu.com/Install/ServeriSCSIRoot#Authenticated%20Login">QA Testcase</a></li>
<li>iSCSI Unauthenticated Root Installation <a href="http://testcases.qa.ubuntu.com/Install/ServeriSCSIRoot#Unauthenticated%20Login">QA Testcase</a></li>
</ul>These tests are being marked as <b>NOT</b> required for ISO release exit criteria for Ubuntu Server. Quoting Robbie Williamson "<i>Our decision was based on the lack of consistently available resources to verify these configurations. With that said, we are keeping them optional</i>"<br />
<br />
Robbie adds "<i>Anyone wishing to reverse this decision, please contact me directly. I have no problem adding them back along with your agreement to run them for each ISO release</i>"<br />
<br />
Again that's a great way to contribute to Ubuntu. If interested please contact Robbie directly, or contact me (leave a comment right here) and I will help get you on track</div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com4tag:blogger.com,1999:blog-342921498109425004.post-56539937092818338512011-02-09T21:38:00.000+02:002011-02-09T21:38:16.572+02:00Cloud Backup LAMP Stack like a proRunning Ubuntu server in the cloud to run your own LAMP environment is extremely popular. Backing up that environment is definitely something that almost everyone should think about, unless it's a test env. In this screencast I'll demo using the ec2-consistent-snapshot tool to perform consistent database backups over a XFS formatted ebs volume<br />
<br />
If you can't see the embedded player, here's a direct link<br />
<a href="http://www.youtube.com/watch?v=SPVqJWWiLVI">http://www.youtube.com/watch?v=SPVqJWWiLVI</a><br />
<br />
<object width="640" height="385"><param name="movie" value="http://www.youtube-nocookie.com/v/SPVqJWWiLVI?fs=1&hl=en_US"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube-nocookie.com/v/SPVqJWWiLVI?fs=1&hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="385"></embed></object><br />
<br />
Shoot me a comment if there's something specific you'd like to see, or if you'd like to contribute one such screencast! Till next timeKamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com3tag:blogger.com,1999:blog-342921498109425004.post-71594326582785083992011-01-19T15:31:00.000+02:002011-01-19T15:31:27.785+02:00Hack on Ubuntu Cloud Utils<div class="separator" style="clear: both; text-align: center;"><a href="http://www.flickr.com/photos/90048824@N00/5369373619/" style="margin-left: 1em; margin-right: 1em;" title="hack-cloud-small by Ahmed-Kamal, on Flickr"><img alt="hack-cloud-small" src="http://farm6.static.flickr.com/5204/5369373619_1619a8236b.jpg" /></a></div>Want a chance to hack on the cloud? Yes Ubuntu cloud code? What could possibly be cooler! For us cloud geeks, almost nothing! So, here's the deal. Ubuntu aims to be the best operating system for the cloud, period. Part of that is delivering code and utilities to ease common tasks for people working with Ubuntu as a guest cloud OS. For that I've suggested a utility to migrate ec2 ebs based images across different Amazon regions. Conceptually, what's needed is simple, a tool needs to be written to spin up two virtual servers in the two regions and sync the disks across them then terminate. The Ubuntu server team agrees it's a needed utility, and if done right, it should end up being shipped with Ubuntu (coolness!)<br />
<br />
I have written step by step, what needs to be done to accomplish this task in a <a href="https://blueprints.launchpad.net/ubuntu/+spec/cloud-server-n-ec2-migrate-region">BluePrint</a>. For those new to that term, BluePrints are Ubuntu's way of writing software specs in the open, where everyone can read and comment on them (don't you just love open!). So, if you're ready to start contributing, click that blueprint link, decide whether you want to hack on this tool, ping me and let's start working on it! For any questions or discussions, please drop by on Freenode IRC #ubuntu-cloud (if you're new to IRC, use this web UI instead <a href="http://cloud.ubuntu.com/community/irc-chat/">http://cloud.ubuntu.com/community/irc-chat/</a>). A great time to discuss this, would be today (Wed) 6pm-UTC during the weekly cloud community meeting. If you miss the meeting no problemo, just talk in the channel anytime, most people are there all the time anyway. I'm already hyper excited. Any questions or feedback? let me know in comments belowKamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-72219820869903174592011-01-17T17:36:00.000+02:002011-01-17T17:36:28.420+02:00Help Ubuntu Server, Voice your Opinion<div>Hello Ubuntu Cloud and Server communities. Now is your chance to voice your opinion about Ubuntu server, as well as help make it better! The Ubuntu marketing team would like more insight into why your organization has chosen to deploy Ubuntu on the server/cloud sides (or why they did not) as well as your opinion as to how Ubuntu server can improve. If you're a pro-IT person deploying or supporting Ubuntu server or cloud in any capacity, please help make Ubuntu better by helping us better understand your needs. Here's a survey designed for just that</div><div><a href="http://survey.ubuntu.com/">http://survey.ubuntu.com/</a></div><div>Thanks!</div>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-39583047338122132522011-01-13T20:23:00.000+02:002011-01-13T20:23:27.332+02:00Ubuntu Server news stormCheck out my interview with Dustin Kirkland. Take a sneak peak into Ubuntu Server's future directions. The server team is working on some pretty exciting technologies for the natty release and later. Things touched on<br />
<br />
<ul><li>Virtualization and LXC improvements</li>
<li>Higher quality Java packaging</li>
<li>Automated Server ISO testing, ensuring rock solid releases</li>
<li>Even more Upstart polish</li>
<li>Improving Ubuntu server's Cluster stack</li>
<li>Datacenter power efficiency with powernap</li>
<li>UEC and EC2 image improvements, pv-grub support</li>
<li>Cloud "desktop" images! wow!</li>
<li>HUGE project for cloud/data-center automatic deployment, management, monitoring, debugging. IMO this takes Ubuntu server way above anything out there. The technology is all about picking and integrating the best of breed open-source server technologies. While you could always do that work yourself, having a tightly integrated stack that is known to work, is what makes Ubuntu server an easy and fun platform to build upon</li>
</ul><div>Exciting times indeed, check out the full interview at</div><br />
Video link: <a href="http://www.youtube.com/watch?v=CWNjzwci5e8">http://www.youtube.com/watch?v=CWNjzwci5e8</a><br />
<br />
<object height="344" width="425"><param name="movie" value="http://www.youtube.com/v/CWNjzwci5e8?fs=1&hl=en_US&rel=0&hd=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/CWNjzwci5e8?fs=1&hl=en_US&rel=0&hd=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="344"></embed></object>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-29263795072093886832011-01-13T00:52:00.001+02:002011-01-13T01:04:37.701+02:00Upgrade Lucid to Maverick on EC2So you're running Ubuntu server Lucid on EC2 cloud EBS root, and decide you don't really want to wait till the next LTS release (12.04)! You want to upgrade your server to Maverick, well, wait no further. Here's how to do it<br />
<br />
Let's download the list of ec2 images, get a list lucid ebs i386 amis<br />
<pre>ec2-describe-images --all > /tmp/ec2-images
$ grep '099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386' /tmp/ec2-images
IMAGE ami-714ba518 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20100427.1 099720109477 available public i386machine aki-754aa41c ebs paravirtual
IMAGE ami-1234de7b 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20100827 099720109477 available public i386 machine aki-5037dd39 ebs paravirtual
IMAGE ami-6c06f305 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20100923 099720109477 available public i386 machine aki-3204f15b ebs paravirtual
IMAGE ami-480df921 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20101020 099720109477 available public i386 machine aki-6603f70f ebs paravirtual
IMAGE ami-a2f405cb 099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20101228 099720109477 available public i386 machine aki-3af50453 ebs paravirtual
</pre><br />
Start our instance, if you were already running a lucid server, of course you wouldn't have to do that, but for our demo, we'll create a lucid instance to upgrade it to maverick<br />
<pre>$ ec2-run-instances ami-a2f405cb --instance-type t1.micro -k default
RESERVATION r-33271c59 553172479171 default
INSTANCE i-05cd0169 ami-a2f405cb pending default 0 t1.micro 2011-01-12T20:42:12+0000 us-east-1d aki-3af50453 monitoring-disabled ebs paravirtual
$ ec2-describe-instances | awk '-F\t' '$1 == "INSTANCE" { print $4 }'
ec2-67-202-26-253.compute-1.amazonaws.com
</pre><br />
Let's ssh into our to-be-upgraded server, install latest lucid upgrades<br />
<pre>ssh ubuntu@ec2-67-202-26-253.compute-1.amazonaws.com
screen -S upgrade
sudo apt-get update
sudo apt-get dist-upgrade
</pre><br />
Normally, you wouldn't be able to update a LTS release, except to another LTS release. We'll change that to upgrade to maverick, and start upgrading<br />
<pre>$ sudo sed -i.bak -e 's@lts$@normal@' /etc/update-manager/release-upgrades
$ do-release-upgrade
Checking for a new ubuntu release
Done Upgrade tool signature
Done Upgrade tool
Done downloading
extracting 'maverick.tar.gz'
authenticate 'maverick.tar.gz' against 'maverick.tar.gz.gpg'
tar: Removing leading `/' from member names
Reading cache
Checking package manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Building data structures... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
Building data structures... Done
Updating repository information
WARNING: Failed to read mirror file
100% [Working]
Checking package manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Building data structures... Done
Calculating the changes
Calculating the changes
Do you want to start the upgrade?
2 installed packages are no longer supported by Canonical. You can
still get support from the community.
3 packages are going to be removed. 29 new packages are going to be
installed. 274 packages are going to be upgraded.
You have to download a total of 124M. This download will take about
15 minutes with a 1Mbit DSL connection and about 4 hours with a 56k
modem.
Fetching and installing the upgrade can take several hours. Once the
download has finished, the process cannot be cancelled.
Continue [yN] Details [d]
</pre><br />
Agree to any prompts while upgrading, till it actually finishes. Let's not reboot the server, instead we power it off<br />
<pre>System upgrade is complete.
Restart required
To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.
Continue [yN]n
$ sudo poweroff
</pre><br />
Now what happened is the server has actually been updated from lucid to maverick. However, Amazon would still boot the server using a lucid kernel. That's because for ec2 the kernel is stored outside of the ebs image itself. However, recently amazon added the ability to use pv-grub for booting, that's basically a way to chainload the kernel from inside the ebs image and boot that. That actually makes the cloud server behave just like a bare metal server, which is always a good thing :) To get the aki ID for the pv-grub kernel, we simply grep for the latest maverick images, since those already have pv-grub attached to them<br />
<pre>$ grep '099720109477/ebs/ubuntu-images/ubuntu-maverick-10.10-i386' /tmp/ec2-images
IMAGE ami-508c7839 099720109477/ebs/ubuntu-images/ubuntu-maverick-10.10-i386-server-20101007.1 099720109477 available public i386machine aki-407d9529 ebs paravirtual
IMAGE ami-ccf405a5 099720109477/ebs/ubuntu-images/ubuntu-maverick-10.10-i386-server-20101225 099720109477 available public i386machine aki-407d9529 ebs paravirtual
</pre><br />
Now let's switch the upgraded image to use the pv-grub kernel<br />
<pre>ec2-modify-instance-attribute i-05cd0169 --kernel aki-407d9529
kernel i-05cd0169 aki-407d9529
$ ec2-start-instances i-05cd0169
INSTANCE i-05cd0169 stopped pending
$ ec2-describe-instances
RESERVATION r-33271c59 553172479171 default
INSTANCE i-05cd0169 ami-a2f405cb ec2-50-16-133-253.compute-1.amazonaws.com ip-10-112-53-225.ec2.internal running default 0 t1.micro 2011-01-12T21:27:51+0000 us-east-1d aki-407d9529 monitoring-disabled 50.16.133.253 10.112.53.225 ebs paravirtual
BLOCKDEVICE /dev/sda1 vol-dc7bc9b4 2011-01-12T20:42:22.000Z
</pre><br />
We're done, let's ssh into the new instance, and make sure it's running maverick kernel<br />
<pre>ssh ubuntu@ec2-50-16-133-253.compute-1.amazonaws.com
$ uname -r
2.6.35-24-virtual
</pre><br />
voila, mission accomplished. Don't forget to terminate your instance<br />
<pre>$ ec2-terminate-instances i-05cd0169
INSTANCE i-05cd0169 running shutting-down
</pre>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com0tag:blogger.com,1999:blog-342921498109425004.post-71955401553666644412011-01-11T20:51:00.000+02:002011-01-11T20:51:39.903+02:00Jono'ed at Dallas sprintI'm attending Ubuntu's sprint at Dallas, and I've just been Jono'ed. Check out this video where Jono asks me about everything cloud communitish. If you wanna know how to get involved with the ubuntu cloud community, there are some good pointers in <a href="http://www.youtube.com/user/ubuntudevelopers#p/a/u/1/Xs4HC1PDvl4">there</a><br />
<br />
<object style="height: 390px; width: 640px"><param name="movie" value="http://www.youtube.com/v/Xs4HC1PDvl4?version=3"><param name="allowFullScreen" value="true"><param name="allowScriptAccess" value="always"><embed src="http://www.youtube.com/v/Xs4HC1PDvl4?version=3" type="application/x-shockwave-flash" allowfullscreen="true" allowScriptAccess="always" width="640" height="390"></object>Kamalhttp://www.blogger.com/profile/02280645950384593949noreply@blogger.com1