Ubuntu takes UFOs to the cloud

I've always believed in UFOs as a kid, and while I've never seen one (yet?) I am still more on the believer side! I was interested to stumble upon a database of UFO sightings at http://www.infochimps.com/tags/ufo# A shout-out at infochimps (you guys are great!). Downloading the sightings DB (around 80MB), I found a listing of 60,000 documented sightings, hmm interesting! I started thinking I could crunch on this data in some useful and fun way, what about finding the most commonly spotted UFO shape?! Sounds like I could use hadoop for that, just for the coolness factor really, the data is not that large anyway, but hey why not! I had no-idea how to get started with hadoop though and wasn't really interested in learning up all the gory details!

Well Ensemble to the rescue, hadoop master and slave formulas exist, which means someone else packaged the knowledge needed to setup and run a hadoop cluster for me. All I needed to do was ask Ensemble to deploy me a couple of cloud instances and start playing. Let's see how you can do that for yourself

I won't repeat the instructions to get started with Ensemble, since the documentation is a good place for that (and it's so easy anyway!). If you feel you need more help there, this little video should be helpful. If you're still stuck, you can always drop by on irc/freenode at #ubuntu-ensemble and ask your questions

Hadoop node, with an extra slave please

So, let's start ensembling

bzr branch lp:~negronjl/+junk/hadoop-master
bzr branch lp:~negronjl/+junk/hadoop-slave
ensemble bootstrap

wait a minute or two for EC2 to spin up the instance, then

ensemble status

which’ll give you output like

$ ensemble status
2011-07-12 15:20:54,978 INFO Connecting to environment.
The authenticity of host 'ec2-50-17-28-19.compute-1.amazonaws.com (50.17.28.19)' can't be established.
RSA key fingerprint is c5:21:62:f0:ac:bd:9c:0f:99:59:12:ec:4d:41:48:c8.
Are you sure you want to continue connecting (yes/no)? yes
machines:
  0: {dns-name: ec2-50-17-28-19.compute-1.amazonaws.com, instance-id: i-8bc034ea}
services: {}
2011-07-12 15:21:01,205 INFO 'status' command finished successfully

Now let's deploy a two node hadoop cluster

ensemble deploy --repository . hadoop-master
ensemble deploy --repository . hadoop-slave
ensemble add-relation hadoop-master hadoop-slave

Yeah it's that easy! Ensemble formulas manage all the kung-fu for you. The hadoop cluster is ready, let's ssh into the master node and switch to user hdfs

ensemble ssh hadoop-master/0
sudo -su hdfs

Downloading UFOs

Download the infochimps sightings database here, unzip it and locate the TSV file (tab separated values) file. Note that you can download the file from infochimps without registering on their website (didn't I say these guys were great :)

Upload the TSV DB to hadoop's distributed filesystem

hadoop dfs -copyFromLocal ufo_awesome.tsv ufo_awesome.tsv

Almost ready, the corpus has been uploaded. Now we need to write some map/reduce jobs to do the actual crunching. Not being a pro developer, the thought of writing that in java was like (oh no ew), so python to the rescue! Thanks to the great instructions at Michael Noll's blog, I was able to massage some of that code to get it to do what I wanted. I pushed my code to launchpad, so that you can grab it directly from the hadoop master node

cd /tmp
bzr branch lp:~kim0/+junk/ufo-ensemble-cruncher
cd ufo-ensemble-cruncher

Unleashing the elephant

Now for the big moment, let's launch the elephant

hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-*.jar -file ./mapper.py -mapper mapper.py -file ./reducer.py -reducer reducer.py -input ufo_awesome.tsv -output ufo-output
packageJobJar: [./mapper.py, ./reducer.py, /tmp/hadoop-hdfs/hadoop-unjar1418682529553378062/] [] /tmp/streamjob5701745574334998473.jar tmpDir=null
11/07/29 12:27:52 INFO mapred.FileInputFormat: Total input paths to process : 1
11/07/29 12:27:53 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-hdfs/mapred/local]
11/07/29 12:27:53 INFO streaming.StreamJob: Running job: job_201107290935_0010
11/07/29 12:27:53 INFO streaming.StreamJob: To kill this job, run:
11/07/29 12:27:53 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=domU-12-31-39-10-81-8E.compute-1.internal:8021 -kill job_201107290935_0010
11/07/29 12:27:53 INFO streaming.StreamJob: Tracking URL: http://domU-12-31-39-10-81-8E.compute-1.internal:50030/jobdetails.jsp?jobid=job_201107290935_0010
11/07/29 12:27:54 INFO streaming.StreamJob:  map 0%  reduce 0%
11/07/29 12:28:11 INFO streaming.StreamJob:  map 10%  reduce 0%
11/07/29 12:28:12 INFO streaming.StreamJob:  map 19%  reduce 0%
11/07/29 12:28:14 INFO streaming.StreamJob:  map 72%  reduce 0%
11/07/29 12:28:16 INFO streaming.StreamJob:  map 100%  reduce 0%
11/07/29 12:28:33 INFO streaming.StreamJob:  map 100%  reduce 100%
11/07/29 12:28:37 INFO streaming.StreamJob: Job complete: job_201107290935_0010
11/07/29 12:28:37 INFO streaming.StreamJob: Output: ufo-output

Woohoo success! Now let's grab the results, sorting it to easily see the most popular sighting shape

Is the answer really 42

hadoop dfs -cat ufo-output/part-00000 | sort -k 2,2 -nr
light   12202
triangle        6082
circle  5271
disk    4825
other   4593
unknown 4490
sphere  3637
fireball        3452
oval    2869
formation       1788
cigar   1782
changing        1546
flash   990
cylinder        982
rectangle       966
diamond 915
chevron 760
egg     664
teardrop        595
cone    265
cross   177
delta   8
round   2
crescent        2
pyramid 1
hexagon 1
flare   1
dome    1
changed 1

The answer is "light" then! Wow that was a blast! I had fun doing this exercise. Now I am no hadoop expert in any way (so direct those hadoopy questions to someone who can actually answer them), however I was quite pleased Ensemble could help me get up and running that fast. The Ensemble community is doing a great job wrapping many free software with formulas, such that you can always get up and running with any app you need in seconds rather than days (months?). You too can write Ensemble formulas for your favorite (server?) application. Hop on to #ubuntu-ensemble and grab me (kim0) or any of the dev team and ask any questions on your mind! We're a happy community

So was that fun? Can you think of something cooler you want to see done? Leave me a comment, let me know about it

Ensemble deploy and scale cloud apps

Think deploying and scaling your cloud application is hard ? Think again!



Check out this video demo, where I deploy a multi-tiered cloud application. I'm deploying

• HAproxy load balancer
• Memcached caching server
• MediaWiki application server
• MySQL DB

connecting them together and getting it working. After which, I'm scaling the whole thing from two application servers, to four

How long it takes me ? Well basically 5 minutes for the whole thing!

The secret: Ensemble !

Here's the video

Here is a direct link if you can't see the player

So what do you guys think, like the demo ? Leave me a comment, let me know which demo you'd like to see next. Also if you'd like to see your favorite application deployed with Ensemble, leave me a comment or ping me (kim0) on freenode irc

Ensemble user tutorial p2

Welcome to the second part of this Ensemble user tutorial. In part 1 we bootstrapped an Ensemble environment, deployed a sample wordpress service and a MySQL service. Related the two services together and got ourselves a working wordpress installation. In this second part, let's check out viewing the debug-log output to understand the asynchronous nature of hook execution. You'll see how easy it is to "scale-up" a service deployment

Tracing hook execution 
An Ensemble user should never have to trace the execution order of hooks, however if you are the kind of person who enjoys looking under the hood, this section is for you. Understanding hook order execution, the parallel nature of hook execution across instances, and how relation-set in a hook can trigger the execution of another hook is quite interesting and provides insight into Ensemble internals

Here are a few important messages from the debug-log of this Ensemble run. The date field has been deliberately left in this log, in order to understand the parallel nature of hook execution.

Things to consider while reading the log include:

• The time the log message was generated
• Which service unit is causing the log message (for example mysql/0)
• The message logging level. In this run DEBUG messages are generated by the Ensemble core engine, while WARNING messages are generated by calling ensemble-log from inside formulas (which you can read in the examples folder)

Let’s view select debug-log messages which can help understand the execution order:

14:29:43,625 unit:mysql/0: hook.scheduler DEBUG: executing hook for wordpress/0:joined
14:29:43,626 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
14:29:43,660 unit:wordpress/0: hook.scheduler DEBUG: executing hook for mysql/0:joined
14:29:43,660 unit:wordpress/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
14:29:43,661 unit:wordpress/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:29:43,789 unit:mysql/0: unit.hook.api WARNING: Creating new database and corresponding security settings
14:29:43,813 unit:wordpress/0: unit.hook.api WARNING: Retrieved hostname: ec2-184-72-156-54.compute-1.amazonaws.com
14:29:43,976 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:29:43,997 unit:wordpress/0: hook.scheduler DEBUG: executing hook for mysql/0:modified
14:29:43,997 unit:wordpress/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:29:44,143 unit:wordpress/0: unit.hook.api WARNING: Retrieved hostname: ec2-184-72-156-54.compute-1.amazonaws.com
14:29:44,849 unit:wordpress/0: unit.hook.api WARNING: Creating appropriate upload paths and directories
14:29:44,992 unit:wordpress/0: unit.hook.api WARNING: Writing wordpress config file /etc/wordpress/config-ec2-184-72-156-54.compute-1.amazonaws.com.php
14:29:45,130 unit:wordpress/0: unit.hook.api WARNING: Writing apache config file /etc/apache2/sites-available/ec2-184-72-156-54.compute-1.amazonaws.com
14:29:45,301 unit:wordpress/0: unit.hook.api WARNING: Enabling apache modules: rewrite, vhost_alias
14:29:45,512 unit:wordpress/0: unit.hook.api WARNING: Enabling apache site: ec2-184-72-156-54.compute-1.amazonaws.com
14:29:45,688 unit:wordpress/0: unit.hook.api WARNING: Restarting apache2 service

Scaling the ensemble 
Assuming your blog got really popular, is having high load and you decided to scale it up (it’s a cloud deployment after all). Ensemble makes this magically easy. All what is needed is:

$ bin/ensemble add-unit wordpress
INFO Connecting to environment.
INFO Unit 'wordpress/1' added to service 'wordpress'
INFO 'add_unit' command finished successfully
$ bin/ensemble status
machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
  2: {dns-name: ec2-184-72-156-54.compute-1.amazonaws.com, instance-id: i-9c7e29f3}
  3: {dns-name: ec2-50-16-156-106.compute-1.amazonaws.com, instance-id: i-ba6532d5}
services:
  mysql:
    formula: local:mysql-11
    relations: {db: wordpress}
    units:
      mysql/0:
        machine: 1
        relations:
          db: {state: up}
        state: started
  wordpress:
    formula: local:wordpress-29
    relations: {db: mysql}
    units:
      wordpress/0:
        machine: 2
        relations:
          db: {state: up}
        state: started
      wordpress/1:
        machine: 3
        relations:
          db: {state: up}
        state: started

The add-unit command starts a new wordpress instance (wordpress/1), which then joins the relation with the already existing mysql/0 instance. mysql/0 notices the database required has already been created and thus decides all needed configuration has already been done. On the other hand wordpress/1 reads service settings from mysql/0 and starts configuring itself and joining the ensemble. Let’s review a short version of debug-log for adding wordpress/1:

14:36:19,755 unit:mysql/0: hook.scheduler DEBUG: executing hook for wordpress/1:joined
14:36:19,755 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
14:36:19,810 unit:wordpress/1: hook.scheduler DEBUG: executing hook for mysql/0:joined
14:36:19,811 unit:wordpress/1: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
14:36:19,811 unit:wordpress/1: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:36:19,918 unit:mysql/0: unit.hook.api WARNING: Database already exists, exiting
14:36:19,938 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
14:36:19,990 unit:wordpress/1: unit.hook.api WARNING: Retrieved hostname: ec2-50-16-156-106.compute-1.amazonaws.com
14:36:20,757 unit:wordpress/1: unit.hook.api WARNING: Creating appropriate upload paths and directories
14:36:20,916 unit:wordpress/1: unit.hook.api WARNING: Writing wordpress config file /etc/wordpress/config-ec2-50-16-156-106.compute-1.amazonaws.com.php
14:36:21,088 unit:wordpress/1: unit.hook.api WARNING: Writing apache config file /etc/apache2/sites-available/ec2-50-16-156-106.compute-1.amazonaws.com
14:36:21,236 unit:wordpress/1: unit.hook.api WARNING: Enabling apache modules: rewrite, vhost_alias
14:36:21,476 unit:wordpress/1: unit.hook.api WARNING: Enabling apache site: ec2-50-16-156-106.compute-1.amazonaws.com
14:36:21,682 unit:wordpress/1: unit.hook.api WARNING: Restarting apache2 service

Destroying the environment 
Once you are done with an Ensemble deployment, you need to terminate all running instances in order to stop paying for them. The shutdown command helps terminate all running instances:

$ bin/ensemble shutdown

Ensemble will ask for user confirmation of shutdown before proceeding as this will destroy service data in the environment as well.

Hope you enjoyed the article! If you find Ensemble interesting, please do visit us in #ubuntu-ensemble. We're a friendly bunch :) Do you want to write Ensemble formulas? Want to get the satisfaction of Ensemblizing your favorite application, grab me (kim0) on irc, and I will help you do it

Ensemble user tutorial p1

I've been adding lots of documentation to help a new Ensemble user find her way around Ensemble. I thought it would nice to share this documentation as a series of blog posts, to raise its exposure and help interested users find their way quickly. Here is the first of a series of posts, which I hope you'll enjoy. If you think we can improve the docs in any way, please do let me know

Introduction

This tutorial demonstrates basic features of Ensemble from a user perspective. An Ensemble user would typically be a devops or a sys-admin who is interested in automated deployment and management of servers and services.

Bootstrapping

The first step for deploying an Ensemble system is to perform bootstrapping. Bootstrapping launches a utility instance that is used in all subsequent operations to launch and orchestrate other instances:

$ bin/ensemble bootstrap

Note that while the command should display a message indicating it has finished successfully, that does not mean the bootstrapping instance is immediately ready for usage. Bootstrapping an instance can require a couple of minutes. To check on the status of the Ensemble deployment, we can use the status command:

$ bin/ensemble status

If the bootstrapping node has not yet completed bootstrapping, the status command may either mention the environment is not yet ready, or may display a connection timeout such as:

INFO Connecting to environment.
ERROR Connection refused
ProviderError: Interaction with machine provider failed:
ConnectionTimeoutException('could not connect before timeout after 2
retries',)
ERROR ProviderError: Interaction with machine
provider failed: ConnectionTimeoutException('could not connect before timeout
after 2 retries',)

This is simply an indication the environment needs more time to complete initialization. It is recommended you retry every minute. Once the environment has properly initialized, the status command should display:

machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  services: {}

Note the following, machine “0” has been started. This is the bootstrapping node and the first node to be started. The dns-name for the node is printed. Also the EC2 instance-id is printed. Since no services are yet deployed to the Ensemble system yet, the list of deployed services is empty

Starting debug-log
While not a requirement, it is beneficial for the understanding of Ensemble to start a debug-log session. Ensemble’s debug-log provides great insight into the execution of various hooks as they are triggered by various events. It is important to understand that debug-log shows events from a distributed environment (multiple-instances). This means that log lines will alternate between output from different instances. To start a debug-log session, from a secondary terminal issue:

$ bin/ensemble debug-log
INFO Connecting to environment.
INFO Enabling distributed debug log.
INFO Tailing logs - Ctrl-C to stop.

This will connect to the environment, and start tailing logs.

Deploying service units
Now that we have bootstrapped the Ensemble environment, and started the debug-log viewer, let’s proceed by deploying a mysql service:

$ bin/ensemble deploy --repository=examples mysql
INFO Connecting to environment.
INFO Formula deployed as service: 'mysql'
INFO 'deploy' command finished successfully

Checking the debug-log window, we can see the mysql service unit being downloaded and started:

Machine:1: ensemble.agents.machine DEBUG: Downloading formula
local:mysql-11...
Machine:1: ensemble.agents.machine INFO: Started service unit mysql/0

It is important to note the different debug levels. DEBUG is used for very detailed logging messages, usually you should not care about reading such messages unless you are trying to debug (hence the name) a specific problem. INFO debugging level is used for slightly more important informational messages. In this case, these messages are generated as the mysql formula’s hooks are being executed. Let’s check the current status:

$ bin/ensemble status
machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
services:
  mysql:
    formula: local:mysql-11
    relations: {}
    units:
      mysql/0:
        machine: 1
        relations: {}
        state: null

We can see a new EC2 instance has now been spun up for mysql. Information for this instance is displayed as machine number 1 and mysql is now listed under services. It is apparent the mysql service unit has no relations, since it has not been connected to wordpress yet. Since this is the first mysql service unit, it is being referred to as mysql/0, subsequent service units would be named mysql/1 and so on.

Note

An important distinction to make is the difference between a service and a service unit. A service is a high level concept relating to an end-user visible service such as mysql. The mysql service would be composed of several mysql service units referred to as mysql/0, mysql/1 and so on.

The mysql service state is listed as null since it’s not ready yet. Downloading, installing, configuring and starting mysql can take some time. However we don’t have to wait for it to configure, let’s proceed deploying wordpress:

$ bin/ensemble deploy --repository=examples wordpress

Let’s wait for a minute for all services to complete their configuration cycle and get properly started, then issue a status command:

$ bin/ensemble status
machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
  2: {dns-name: ec2-184-72-156-54.compute-1.amazonaws.com, instance-id: i-9c7e29f3}
services:
  mysql:
    formula: local:mysql-11
    relations: {}
    units:
      mysql/0:
        machine: 1
        relations: {}
        state: started
  wordpress:
    formula: local:wordpress-29
    relations: {}
    units:
      wordpress/0:
        machine: 2
        relations: {}
        state: started

mysql/0 as well as wordpress/0 are both now in the started state. Checking the debug-log would reveal wordpress has been started as well

Adding a relation
While mysql and wordpress service units have been started, they are still isolated from each other. An important concept for Ensemble is connecting various service units together to create a bigger ensemble! Adding a relation between service units causes hooks to trigger, in effect causing all service units to collaborate and work together to reach the desired end state. Adding a relation is extremely simple:

$ bin/ensemble add-relation wordpress mysql
INFO Connecting to environment.
INFO Added mysql relation to all service units.
INFO 'add_relation' command finished successfully

Checking the Ensemble status we see that the db relation now exists with state up:

$ bin/ensemble status
machines:
  0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
  1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
  2: {dns-name: ec2-184-72-156-54.compute-1.amazonaws.com, instance-id: i-9c7e29f3}
services:
  mysql:
    formula: local:mysql-11
    relations: {db: wordpress}
    units:
      mysql/0:
        machine: 1
        relations:
          db: {state: up}
        state: started
  wordpress:
    formula: local:wordpress-29
    relations: {db: mysql}
    units:
      wordpress/0:
        machine: 2
        relations:
          db: {state: up}
        state: started

You can now point your browser at the public dns-name for instance 2 (running wordpress) to view the wordpress blog

Patience is a virtue, you did read the full thing after all! I would really appreciate some feedback, what you liked, or did not like. And how we can improve the docs. Drop me a comment, shoot what's on your mind

Ensemble at UDS-O

UDS-O is now over, I had a chance to meet with the Ensemble team (a bunch of awesome engineers), also had a chance to attend or lead a few sessions concerning future directions of Ensemble. I'll try to summarize UDS outcome wrt Ensemble from a project newcomer perspective

• Ensemble is now able to do a multi-machine deployment and orchestration
• It can do Dynamic reconfiguration which means passing parameters to running formulas to adjust behavior. More work needs to land here, but the foundations are there
• Does formula upgrades, more bits landing soon
• Firewall auto-configuration (expose, unexpose services). Again, still more bits to land soonish
• It has a ppa and docs from trunk live at https://ensemble.ubuntu.com/docs/

The focus for the 11.10 Oneiric cycle is going to be stability and polish. While currently Amazon EC2 is the only supported deployment target, the 11.10 cycle should hopefully see more targets added such as Linux Containers (LXC) for local development and testing of Ensemble formulas. Having LXC support in 11.10 is a bit optimistic, so if you can lend a helping hand, please do! Also Eucalyptus cloud support is the works (isn't this just great!)

During 11.10 as well, the infrastructure for Ensemble as a project should be improved. This involves adding better and more structured content to the Ensemble website. Adding more documentation, guides and screencasts. While you can start writing and contributing Ensemble formulas today! (ping me if you're interested) More work will go into refining the process and integrating it into the Ensemble command line tools.

That's mostly it. It is an incredibly interesting time for cloud technologies! Diving into discussions with the Ensemble team about the vision and decisions they have taken, I was blown away. I firmly believe Ensemble is a game changer with respect to rapid provisioning and orchestration. Right now, is such a great time to get involved with Ensemble. If you're interested in joining the Ensemble project, or in learning more about it in any way, please do leave me a comment here, or ping me on irc (kim0)

Cloud Portal one-click launch

Announcing two nice little additions to the cloud portal AMI tool that should make everyone's cloud life a little easier

• Amazon now allows passing parameters to the AWS console to basically be able to choose the region and AMI-ID to launch. This is now integrated in the cloud portal so that you search for the ami, click the link and it's pre-selected, ready to be launched in AWS console. Here's a screenshot

• As that image shows as well, you can search now for "cluster" to get Ubuntu cloud cluster-ready images!

Interested to add any features? Have any thoughts or comments, let me know, leave me a word

Help Squash Natty Server Bugs

Mirroring a post from Ubuntu Server developer Dave Walker, If you're capable and willing to help squash some server bugs, your effort and contribution is really needed now! Join us on irc #ubuntu-server, ask around if you need help. Thanks in advance

Here is a list of current bugs that should hopefully be fixed for Natty release.  The bugs with "Not assigned to anyone :(", could really do  with someone volunteering to take.  Please feel free to assign yourself, if you are willing to take on working towards a fix. 

If you are an assignee, please make sure that the current status (and comments) accurately represent the status before tomorrows server team meeting. 

== Bugs for the server team == 

[655533] [likewise-open] [master] package likewise-open 5.4.0.42111-2ubuntu2 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1 (http://pad.lv/655533) 

 * Bug Status: Confirmed 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-18 

[726769] [eucalyptus] package eucalyptus-common 2.0.1 bzr1255-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1 (http://pad.lv/726769) 

 * Bug Status: Confirmed 

 * Assigned to: Dave Walker 

 * Last updated: 2011-04-18 

[661453] [dovecot] dovecot.conf always shows as having been locally modified on update (http://pad.lv/661453) 

 * Bug Status: Confirmed 

 * Assigned to: Clint Byrum 

 * Last updated: 2011-04-18 

[745946] [cloud-init] cloud-final job did not run in ec2-automated-tests (http://pad.lv/745946) 

 * Bug Status: Confirmed 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-18 

[759943] [mod-wsgi] mod_wsgi.so-3.2 gives errors (http://pad.lv/759943) 

 * Bug Status: Confirmed 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-15 

[764391] [cobbler] cobbler fails to manage bind9  (http://pad.lv/764391) 

 * Bug Status: Confirmed 

 * Assigned to: James Page 

 * Last updated: 2011-04-18 

[745930] [cloud-init] cloud-init timeout waiting for metadata service on EC2 (http://pad.lv/745930) 

 * Bug Status: New 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-15 

[760288] [Ubuntu] JeOS is oversized (http://pad.lv/760288) 

 * Bug Status: Confirmed 

 * Not assigned to anyone :( 

 * Last updated: 2011-04-15 

 == Bugs being worked on in other areas, more for information == 

[728088] [debian-installer] iscsi root with or without auth fails to boot (http://pad.lv/728088) 

 * Bug Status: Confirmed 

 * Assigned to: Colin Watson 

 * Last updated: 2011-04-14 

[747090] [linux] wrong return address sometimes pushed for INT in kvm (not qemu) (http://pad.lv/747090) 

 * Bug Status: Fix Committed 

 * Assigned to: Andy Whitcroft 

 * Last updated: 2011-04-18 

[566818] [plymouth] Cryptsetup passphrase prompt during boot: every character typed repeats the prompt (http://pad.lv/566818) 

 * Bug Status: Confirmed 

 * Assigned to: Surbhi Palande 

 * Last updated: 2011-04-16 

[580319] [upstart] dhcp3-server launches before upstart brings all interface, thus failing to start (http://pad.lv/580319) 

 * Bug Status: Triaged 

 * Assigned to: James Hunt 

 * Last updated: 2011-04-18 

[759545] [grub2] user prompted to update unmodified grub configuration during Ubuntu server upgrade (http://pad.lv/759545) 

 * Bug Status: New 

 * Last updated: 2011-04-18 

Ensemble Cloud Community Meeting Summary

Yesterday was the first Ubuntu Cloud community meeting to include Ensemble. It was such a blast. Personally I find the ensemble project to be incredibly cool, it's such a paradigm shift into how services (rather than servers) should be managed especially in this cloud age. Here's a quick summary of the main points mentioned during yesterday's meeting

• The ensemble-team just landed some changes to formula authorship, specifically the relation-changed-hook was broken out into three separate hooks the relation-joined, relation-changed, relation-departed
• joined denotes that a unit of the related service came online
• changed denotes the unit of a related service changed its settings
• departed denotes the unit of a related service went offline
• This hook split is helpful to avoid subtle race conditions, and make formula authorship easier
• Ensemble team is now working on formula upgrades
• For now the team is only tackling an initial step of upgrading just the formula itself and only if it's already running
• Example command would be: ensemble upgrade-formula --repository=examples mydbservice
• That will look into the repo directory for a formula matching the formula of myblog, verify its a newer revision, upload and it mark the units for upgrade
• On the unit side, they'll detect they need upgrades, download the formula, and most importantly, execute the upgrade-hook from the new formula
• A suggestion came up, that the upgrade hook would be informed which version the upgrade process is starting from
• A Question comes up, whether hooks should be implemented in a specific language. The answer is that all hooks are effectively just executables, those executables can be in any language and use any tools they'd like
• Ensemble team is also working on orchestrating around firewall addressing such that the firewall is configurable by the ensemble user (ensemble expose, ensemble unexpose), with corresponding hooks (exposed, unexposed), and hook commands  (open-port, close-port)
• The draft for that is on http://people.canonical.com/~niemeyer/ensemble/drafts/expose-services.html
• expose and unexpose are for now operating on top of ec2 security-groups, in the future however multiple backends can be implemented (cisco firewalls..etc)! Without even needing to change the formula, quite impressive!
• A question was asked whether ensemble is ready for public collaboration, the answer to which is a definite yes!
• Grab the code today: bzr branch lp:ensemble
• Join the mailing list: https://lists.ubuntu.com/mailman/listinfo/Ensemble
• Join IRC channels #ubuntu-cloud, #ubuntu-ensemble

If you liked this, be sure to join the next meeting (Every Wed, 18:00-UTC in #ubuntu-cloud). Have questions or comment? Leave me a comment right here, or ping me (kim0) on IRC

Ensemble Cloud Community Meeting

Ensemble is an exciting new cloud technology innovated by Ubuntu. If you’re curious what it does (I bet you are) it is a service orchestration framework with focus on rapid cloud deployment. That is to say, it allows one to rapidly deploy cloud instances, bootstrap them, install software on them, configure them as well as link them all together to provide you (as a sys-admin or DevOp) with an end-user facing high-level “service” (Load Balanced CMS with sharded Database, Multinode email service installation…etc). Ensemble has been compared to being APT for the cloud! Now does that sound interesting, it sure is if you ask me! Welcome to next generation Linux infrastructure automation.

If that got you interested, be sure to attend the weekly IRC meeting on Wed 18:00-UTC on #ubuntu-cloud. 

Read More at: http://cloud.ubuntu.com/2011/04/ensemble-cloud-community-meeting/

Feel free to tweet/dent and spread this around the globe. Got feedback? Leave me comments on cloud.ubuntu.com or on my blog

Help Shape Maps.Ubuntu.com

About either months ago, I had started on a fun little project to raise awareness of how widely used Ubuntu Server is around the world. The result was http://maps.ubuntu.com/ a web tool allowing anyone using Ubuntu Server to "mark" his location on a global map. This was launched with celebrations for 10.04.1, and thousands of cities were marked (woohoo). The project code was open-sourced and lives at https://launchpad.net/mapuntu/trunk

It's been a while since the project was last updated, a few days ago, an Ubuntu community member ( hey Ronnie ) stepped up to re-shape the mapuntu project and relaunch it into an inspiring map that helps anyone record and locate Ubuntu events around the globe! The first part of that effort is creating a proper back-end API, everyone is encouraged to click that link and leave comments. If you are inspired by that project, and would like to help, please get involved! Things you can do to help include

• Review and enhance the proposed API
• Help write python code to implement the backend api
• Write html/css/js for the front-end
• Play with the code, test it, report bugs, adapt to your own website!

If you want to talk more about this, Please join #ubuntu-locoteams on IRC and ping Ronnie or myself (kim0, also on #ubuntu-cloud).

Ubuntu Cloud Days, Day-2 Summary

Yesterday was the second and final day of cloud days. It was such a blast! Very interesting stuff, and very cool community to be part of! In case you missed it, you can still read up the logs, and you can reach me and others for questions and comments on #ubuntu-cloud .. Let's quickly zip through yesterday's sessions

16.00 UTC smoser gave a very good introduction to the different ways with which you can rebundle UEC or EC2 images for Ubuntu, how you can customize them and the different pros and cons. Scott also answered lots of questions
17.00 UTC tetet started by presenting some interesting new features and tricks that can be used with UEC namely persistency, which means kexec'ing the kernel to load from an EBS volume resulting in persistent UEC instances. Thorsten zipped through the session quickly, but as a pro instructor, he was very well prepared with links to graphics, links and documentation
18.00 UTC Daviey wowed the crowd with a puppet introductory session, he showed how to configure apache with puppet and what it takes and deploy a basic system to get running fast. It was a great session
19.00 UTC edulix took over to present an awesome session about using hadoop and its map/reduce programming model to crunch on large (we're talking terabytes) of data on lots of machines. Very interesting and useful session
20.00 UTC obino had the final say on cloud days. The session was about UEC/Eucalyptus. He presented a detailed architecture of the system, what each node does, how things work. He also explained how to get started contributing to the eucalyptus community. 

It was a great event. Big Thank-Yous to everyone who presented in UCD. Till next time o/

Day-2 Ubuntu Cloud Days

Yesterday was great! It was Ubuntu Cloud Days, the event was packed with very interesting sessions, hundreds of attendees, lots of questions got answered, tons of ideas and comments were made. I personally had lots of fun, and was literally stuck to my keyboard the full time of the event


Let's zip through what happened yesterday (If you missed it, Click the time to read the logs!)
16.00 UTC kim0 (me) took-over, gave a quick introduction about cloud computing, differences between public and private clouds and what the cloud fuss is all about
17.00 UTC semiosis started a great session about building scalable distributed storage on the EC2 Cloud using Glusterfs for improving redundancy, and performance
18.00 UTC SpamapS gave a cool presentation and hands-on demo of a new Ubuntu ground breaking "service" management framework called Ensemble. While in early development phase, it's already working well and is changing how we look at server management
19.00 UTC hallyn took over presenting a technically deep discussion and hands-on demo about LXC Linux containers technology. This is a light weight Linux virtualization (or container) technology
20.00 UTC soren took over to talk about OpenStack, one of the cloud stacks that Ubuntu ships with 11.04 that's been getting lots of cloud people excited lately. Soren explained how the whole thing got started, explained the architecture, answered lots of questions and discussed what we should be seeing in the future

Here's a sneak peak of today's sessions

Thu 24th Mar
16.00 UTC rebundling/re-using Ubuntu’s UEC images — smoser
17.00 UTC UEC persistency — tetet
18.00 UTC Puppet Configuration Management — Daviey
19.00 UTC Using hadoop, divide and conquer — edulix
20.00 UTC UEC/Eucalyptus Private Cloud — obino

Let's get crunching on even more cool sessions today! If you’re new to IRC, you can simply use this web page to join. For more information, check out the wiki page, also feel free to ping “kim0″ on IRC!

Today is Ubuntu Cloud Days

Today is .. dum roll ... Ubuntu Cloud Days! The event starts at 4pm UTC on irc #ubuntu-classroom. Updates to the schedule include an OpenStack session, a Eucalyptus session and a puppet session! Those attending the Ensemble session are in for a treat :) During Cloud Days, you’ll learn lots of exciting stuff, you’ll interact with tons of smart people, and you’ll just have fun! Spread the word, join in and bring your friends!

Here’s the sessions schedule

Wed 23rd Mar
16.00 UTC Cloud Computing 101, Ask your questions — kim0
17.00 UTC Scaling shared-storage web apps in the cloud with Ubuntu & GlusterFS — semiosis
18.00 UTC What is Ensemble? – Presentation and Demo — SpamapS
19.00 UTC Using Linux Containers in Natty — hallyn
20.00 UTC Open-Stack Introduction — soren

Thu 24th Mar
16.00 UTC rebundling/re-using Ubuntu’s UEC images — smoser
17.00 UTC UEC persistency — tetet
18.00 UTC Puppet Configuration Management — Daviey
19.00 UTC Using hadoop, divide and conquer — edulix
20.00 UTC UEC/Eucalyptus Private Cloud — obino

All session happen in IRC (Freenode) in #ubuntu-classroom. If you’re new to IRC, you can simply use this web page to join. For more information, check out the wiki page, also feel free to ping “kim0″ on IRC

Additional links:

• Ubuntu Cloud Facebook page
• Ubuntu Cloud Twitter page
• Ubuntu Cloud Identi.ca page

Ubuntu Cloud Days approaching

Just a reminder, Ubuntu Cloud Days is approaching, it's only two days away! Yes it's this Wed/Thu. I just posted the full details on the Ubuntu Cloud Portal. Check it out, and leave me a comment if you have any questions

Two weeks to Ubuntu Cloud Days

Two weeks from now, Ubuntu Cloud Days begin! The very first dedicated Ubuntu Cloud Days event from March 23^rd to March 24^th 2011!

Cloud computing is changing the face of IT and Ubuntu Server is arguably the most popular Cloud based Operating System with millions of instances launched a year! Are you interested to know more about using Ubuntu Server on the Cloud ? Find out from March 23^rd to March 24^th 2011

Ubuntu Cloud Days is a series of online sessions where you can learn more about:

• Answering your questions about Ubuntu and the Cloud
• Ubuntu Enterprise Cloud Images, using Ubuntu with Amazon EC2 cloud
• Building your private cloud over Ubuntu Server platform
• New virtualization and container technologies in Ubuntu Server
• Crunching Big-Data on the cloud with hadoop
• Scaling your web-apps on the cloud with Ubuntu
• much more...

Read more about UCD at https://wiki.ubuntu.com/UbuntuCloudDays

If you are interested to deliver a session, please get in contact with me! While the timetable is now "full", there's no reason why we can't extend an extra session (or day!). Ping me (kim0 AT ubuntu.com) or (kim0 on IRC)

Mark your calendar, Join in and Bring your friends ;)

Additional links:

• Ubuntu Cloud Facebook page
• Ubuntu Cloud Twitter page
• Ubuntu Cloud Identi.ca page

FAST Win7 KVM VirtIO{Disk-Net} Install

Everyone who has to use Windows, please let's at least keep it contained inside a virtual machine! In this article I'll demo how to "Install Windows7 over Ubuntu 11.04 Natty, using KVM with System Disk over VirtIO". Quoting the libvirt wiki "Virtio is a Linux standard for network and disk device drivers where just the guest's device driver "knows" it is running in a virtual environment, and cooperates with the hypervisor. This enables guests to get high performance network and disk operations, and gives most of the performance benefits of paravirtualization"

Here's a little convenience script to download virtio Windows drivers and launch virt-install to Install Windows. Copy/Paste this script into an empty folder, chmod +x it, and run it.

#!/bin/sh
WINISO=/path/to/win7.iso    #Windows ISO
INSTALLDISK=win7virtio.img  #Disk location. Can be LVM LV
VFD=http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-1.1.16.vfd
DRVRISO=http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-1.1.16.iso

[ -e $(basename $VFD) ]     || wget $VFD
[ -e $(basename $DRVRISO) ] || wget $DRVRISO
[ -e $INSTALLDISK ]         || qemu-img create $INSTALLDISK 30G

sudo virt-install -c qemu:///system --virt-type kvm --name win7virtio --ram 1024 --disk path="$INSTALLDISK",bus=virtio \
--disk $(basename $VFD),device=floppy --os-variant win7 --cdrom $(basename $DRVRISO) --cdrom "$WINISO" --vcpus 2

et voila Windows installer kicks in, we're greeted with the usual



Next the welcome screen, and agree on the license agreement (if you want!)


In this next screenshot, we see how Windows Installer does not see any "disks". This is because Windows now does not have any drivers for the virtio disk that we attached!


Let's fix that, clicking "Load Driver" and clicking OK, Windows starts reading the VFD virtual floppy image we attached, and detects the driver


Clicking Next, Windows can now see the virtio disk

Then it's business like usual

After installation completes, be sure to check the device manager and confirm the disk (and network) are both using optimized virtio drivers


et voila, Windows in its full glory running over optimized virtio drivers. The only part that sucks is desktop VNC performance, something that I'll probably write about very soon :)

First Ever Ubuntu Cloud Days

I am very excited to announce the very first Ubuntu Cloud Days! UCD is an online event that is designed for everyone interested in cloud computing, as well as someone who is already using Ubuntu server and who's thinking about "that cloud thing" everyone is talking about. If the words "virtualization", "cloud", "server", "scalability" or "automation" intrigue and amuse you, then you should definitely attend! It's a chance to learn a lot, get your questions answered as well as share your experiences and have fun with the rest of the cloud community!

In fact I'll go one bit further, not only should you attend, you should even deliver a session! Ubuntu as usual is all about community, learning from others, and sharing your knowledge to others. It's quite easy to deliver a session! My first session, I was quite surprised how I didn't have to prepare huge content to "fill" an hour, just answering questions and discussing with people, easily gets you by and you get a warm fuzzy feeling that you just helped lots of people and shared your knowledge.

So, Ladies and Gentlemen, mark your calendars! March 23rd and March 24th

Fore more information: https://wiki.ubuntu.com/UbuntuCloudDays/

To contribute a session, (or if you're thinking about it) talk to me (kim0) on irc #ubuntu-cloud or email [ kim0 =AT= ubuntu.com ]

Cloud Backup LAMP Stack like a pro

Running Ubuntu server in the cloud to run your own LAMP environment is extremely popular. Backing up that environment is definitely something that almost everyone should think about, unless it's a test env. In this screencast I'll demo using the ec2-consistent-snapshot tool to perform consistent database backups over a XFS formatted ebs volume

If you can't see the embedded player, here's a direct link
http://www.youtube.com/watch?v=SPVqJWWiLVI



Shoot me a comment if there's something specific you'd like to see, or if you'd like to contribute one such screencast! Till next time

Hack on Ubuntu Cloud Utils

Want a chance to hack on the cloud? Yes Ubuntu cloud code? What could possibly be cooler! For us cloud geeks, almost nothing! So, here's the deal. Ubuntu aims to be the best operating system for the cloud, period. Part of that is delivering code and utilities to ease common tasks for people working with Ubuntu as a guest cloud OS. For that I've suggested a utility to migrate ec2 ebs based images across different Amazon regions. Conceptually, what's needed is simple, a tool needs to be written to spin up two virtual servers in the two regions and sync the disks across them then terminate. The Ubuntu server team agrees it's a needed utility, and if done right, it should end up being shipped with Ubuntu (coolness!)

I have written step by step, what needs to be done to accomplish this task in a BluePrint. For those new to that term, BluePrints are Ubuntu's way of writing software specs in the open, where everyone can read and comment on them (don't you just love open!). So, if you're ready to start contributing, click that blueprint link, decide whether you want to hack on this tool, ping me and let's start working on it! For any questions or discussions, please drop by on Freenode IRC #ubuntu-cloud (if you're new to IRC, use this web UI instead http://cloud.ubuntu.com/community/irc-chat/). A great time to discuss this, would be today (Wed) 6pm-UTC during the weekly cloud community meeting. If you miss the meeting no problemo, just talk in the channel anytime, most people are there all the time anyway. I'm already hyper excited. Any questions or feedback? let me know in comments below

Help Ubuntu Server, Voice your Opinion

Hello Ubuntu Cloud and Server communities. Now is your chance to voice your opinion about Ubuntu server, as well as help make it better! The Ubuntu marketing team would like more insight into why your organization has chosen to deploy Ubuntu on the server/cloud sides (or why they did not) as well as your opinion as to how Ubuntu server can improve. If you're a pro-IT person deploying or supporting Ubuntu server or cloud in any capacity, please help make Ubuntu better by helping us better understand your needs. Here's a survey designed for just that

http://survey.ubuntu.com/

Thanks!