Monday, December 8, 2008

Magical apt-clone, broken system recovery

I have received some very nice feedback regarding my last blog on D-Bus, thanks guys. This week I'm going to play once more with my Nexenta installation that we completed in part1. This time around I am going to basically demo the integration the Nexenta folks have pulled mixing apt-get with ZFS file-system capabilities, resulting in transactional system upgrades, with multi-snapshots that makes system recovery a snap!

We start off booting our nexenta NCP2-alpha installation, which we'll be upgrading to beta-1 as we speak. Grub shows the default boot kernel

001-normal-grub

Once booted, and we log in, let's update apt's cache

002-apt-get-update

I have also used the command "apt-clone" to list the currently available boot environments. Basically, using ZFS snapshots the whole machine state is snapshot'ed resulting in a instantly bootable environment. Currently we only have the default environment "rootfs-nmu-000". Now I am going to use apt-clone to "dist-upgrade" the whole system. If this is new to you (who has never seen Ubuntu :) it basically means you're upgrading the whole system to the latest tasty bits

003-apt-clone-dist-upgrade

Note that I have used verbose mode "-v" in order to get more information. After a lot of downloading and dpkg grinding, I now have the latest packages installed. apt-clone has created a new system snapshot "001" and the process ends like so

004-apt-clone-done

I have simply hit 'n' to discard any automatic changes, then I used apt-clone to activate the upgraded snapshot as instructed by the on-screen output

005-apt-clone-activate-newBE-reboot

Now let's reboot and see what kind of coolness we have

006-newBE-grub

Oh cool, Grub now displays two bootable environments! One before the upgrades and one after. The dates besides each entry make that clear. Great, now let's reboot into the new envrionment and list the available clones (cloned file systems)

007-booted-into-b104

Note that the "A" and "C" in the output mean "Active" and "Current". Great, now creating instant bootable system snapshots like that is so useful that it does not relate to software installations only. Say you're running some third party closed source installation application and you're not really sure what's it going to do to your system. Or if the new clueless administrator needs to install something to the server but you think he's probably going to destroy it! In such situations, it's very helpful to manually create a system snapshot. So, let's do just that to create the environment 002 and let's activate it

008-manually-create-clone-BE

Now we have our own little safety net. Let's now boot into that new environment. Use apt-clone to make sure "002" is the active and current environment. Then's let's simulate complete system failure. I will not be too creative and will simply delete the kernel and reboot!

009-rm-kernel-reboot

Once in Grub, boot the new and destroyed system. As expected Grub errors out and we're staring at

010-grub-stops-no-kernel

So right now we have a broken system. Now do we run to tape, dump | restore, and use some black magic, nope, we simply reboot and choose our safe snapshot 001, boot into that and destroy the corrupt environment 002 and activate the safe one 001 as so

011-recovered-older-BE-rebooting

Perfect! Once rebooted, our system is back fully functional. Mind you that taking system snapshots, destroying corrupt ones all take a couple of seconds, and of course does not consume disk space except for the changed disk blocks. Now that we have our server back, let's do something useful with it. Let's "apt-clone install apache2".

012-normal-boot-install-apache2

Once done, the SMF could not start apache because I had chosen nwam network auto-configuration (think NetworkManager), which apache2 does not like. Anyway, I simply started the SysV script manually, and verified "it works" with a direct telnet connection. You can basically start installing and configuring various server software mostly just like an Ubuntu system. All in all, the Nexenta-2 system is a very promising concept. The integration of advanced Solaris kernel features with GNU userland and attempting to port Ubuntu repos, would basically be paradise. It is however a lot of work. The system is not perfect yet, for example in this beta-1, you cannot install X yet (broken deps). That will surely be resolved by release time. However, rebuilding the huge debian/Ubuntu repos, needs a lot of work. If you have free time to push this interesting FOSS project forward, please do so. Visit nexenta and join their development team. Meanwhile and separately from nexenta, I have been interacting with various open source developers working with opensolaris projects. Unfortunately it seems that Sun is not really making it easy for "outsiders" to work on opensolaris projects. I have heard such comments multiple times, from different people. One would think that after so much time, Sun would start to understand what it takes to build a community around any piece of software! Let me use the moment to shout out: Sun, please cut the community some slack! :)

9 comments:

Ahmed Kamal said...

btw guys, I know the new opensolaris has a similar functionality. I might even be demo'ing that soon. Enjoy :)

Andrea said...

Congrats Ahmel, very great review.
I found it very illuminating!
What about Opensolaris ZFS screenshots? Do they use the same "apt-clone" thingie?

Andrea said...

sorry...I meant snapshots!

Ahmed Kamal said...

Hey Andrea, opensolaris uses a very similar technique to provide what they call a "boot environment" upon major system upgrades, or created manually as well. It's not the same tool, but the same concept

alotstuff said...

nice info bro keep posting

http://www.sd4us.blogspot.com

alotstuff said...

nice post i like it

http://www.envrionment.blogspot.com

平平 said...

^^Thanks!!

婚前徵信婚姻感情大陸抓姦外遇抓姦法律諮詢家暴婚前徵信尋人感情挽回大陸抓姦離婚工商徵信婚前徵信外遇抓姦感情挽回尋人大陸抓姦離婚家暴工商徵信法律諮詢跟蹤工商徵信婚前徵信感情挽回外遇抓姦法律諮詢家暴尋人大陸抓姦離婚大陸抓姦外遇尋人家暴工商徵信法律諮詢家暴感情挽回大陸抓姦外遇婚前徵信離婚尋人工商徵信外遇抓姦法律諮詢家暴婚前徵信大陸抓姦尋人感情挽回外遇抓姦婚前徵信感情挽回尋人大陸抓姦工商徵信法律諮詢離婚家暴工商徵信外遇抓姦法律諮詢家暴婚前徵信尋人感情挽回大陸抓姦離婚婚前徵信工商徵信外遇抓姦尋人離婚家暴大陸抓姦感情挽回法律諮詢離婚感情挽回婚前徵信外遇抓姦家暴尋人工商徵信外遇抓姦法律諮詢家暴婚前徵信尋人感情挽回">徵大陸抓姦離婚婚前徵信工商徵信外遇抓姦尋人離婚家暴大陸抓姦感情挽回法律諮詢

oy said...

WoW shares many wow gold of its features with previously launched games. Essentially, you battle with wow gold cheap monsters and traverse the countryside, by yourself or as a buy cheap wow gold team, find challenging tasks, and go on to higher aoc gold levels as you gain skill and experience. In the course of your journey, you will be gaining new powers that are increased as your skill rating goes up. All the same, in terms of its features and quality, that is a ture stroy for this.WoW is far ahead of all other games of the genre the cheap warhammer gold game undoubtedly is in a league of its own and cheapest wow gold playing it is another experience altogether.

game4power.com

Even though WoW is a Cheap Wow Gold rather complicated game, the controls and interface are done in warhammer gold such a way that you don't feel the complexity. A good feature of the game is that it Cheapest Wow Gold does not put off people with lengthy manuals. The instructions cannot be simpler and the pop up tips can help you start playing the game World Of Warcraft Gold immediately. If on the other hand, you need a detailed manual, the instructions are there for you to access. Buy wow gold in this site,good for you, BUY WOW GOLD.

blog said...

Hey Andrea, opensolaris uses a very similar technique to provide what they call a "boot environment" upon major system upgrades, or created manually as well. It's not the same tool, but the same concept