OpenSolaris 2009.06 PV under Xen Centos5 dom0

OpenSolaris 2009.06 has been recently released. Since ZFS offers a lot of flexibility and advantages as a storage platform, many would be interested in running the opensolaris stack. In this article I'll be installing OpenSolaris 2009.06 para-virtualized under a centos-5-x64 dom0. This setup has the advatange of running OpenSolaris on storage controllers that may not have native solaris driver support. Not that that would be recommended, however, it might be useful for testing. So, let's begin

- Install a minimal Centos-5-x64 installation. I installed the OS on a 20G partition, while the rest of the disk was formatted as LVM partition for creating logical volumes for xen guests to live on

- Install the latest Xen stack. I will be installing Xen-3.4. This is highly recommended since the bundled boot-loader pygrub has the ability to mount ZFS block devices.

[root@xen2 yum.repos.d]# cat > /etc/yum.repos.d/gitco.repo << EOF
> [gitco]
> name = Red Hat Enterprise $releasever - gitco
> baseurl = http://www.gitco.de/repo/xen3.4.0
> enabled = 1
> protect = 0
> gpgcheck = 0
> EOF

- Now let's install the required virtualization stack

[root@xen2 yum.repos.d]# yum install xen-libs.x86_64 bridge-utils.x86_64 qemu.x86_64 gtkglext-libs.x86_64 gtk-vnc.x86_64 gtk-vnc-python.x86_64 cyrus-sasl-md5.x86_64 libvirt.x86_64 libvirt-python.x86_64 python-virtinst virt-manager.x86_64 kernel-xen.x86_64 xen.x86_64 virt-viewer.x86_64 libvirt.x86_64 gnome-applet-vm.x86_64

Check that your menu.lst file has been updated to point to the newly installed Xen

[root@xen2 yum.repos.d]# cat /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You do not have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /, eg.
#          root (hd0,0)
#          kernel /boot/vmlinuz-version ro root=/dev/sda1
#          initrd /boot/initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-128.1.10.el5xen)
        root (hd0,0)
        kernel /boot/xen.gz-3.4.0
        module /boot/vmlinuz-2.6.18-128.1.10.el5xen ro root=LABEL=/ rhgb quiet
        module /boot/initrd-2.6.18-128.1.10.el5xen.img
title CentOS (2.6.18-128.el5)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.18-128.el5 ro root=LABEL=/ rhgb quiet
        initrd /boot/initrd-2.6.18-128.el5.img

- Reboot into your new environment, and check you are booted on Xen-3.4

[root@xen2 ~]# xm dmesg | head
 __  __            _____ _  _    ___
 \ \/ /___ _ __   |___ /| || |  / _ \
  \  // _ \ '_ \    |_ \| || |_| | | |
  /  \  __/ | | |  ___) |__   _| |_| |
 /_/\_\___|_| |_| |____(_) |_|(_)___/

(XEN) Xen version 3.4.0 (root@gitco.tld) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri May 29 21:39:26 CEST 2009
(XEN) Latest ChangeSet: unavailable
(XEN) Command line:
(XEN) Video information:

- Now we're ready to begin :) Make a folder and download the ISO in it

[root@xen2 ~]# mkdir -p /xen/osol/ && cd /xen/osol && wget http://genunix.org/distributions/indiana/osol-0906-x86.iso

- Let's loop mount the iso, extract the paravirt kernel and micro-root

[root@xen2 osol]# mkdir -p /mnt/loop
[root@xen2 osol]# mount -o ro,loop osol-0906-x86.iso /mnt/loop
[root@xen2 osol]# cp /mnt/loop/boot/amd64/x86.microroot /mnt/loop/platform/i86xpv/kernel/amd64/unix .

- Construct the Xen installation-time configuration file

[root@xen2 osol]# cat > osol0906install << EOF
> name = "osol0906install"
> memory = 1024
> vfb = [ 'type=vnc,vnclisten=127.0.0.1,vncdisplay=0' ]
> disk = ['file:/xen/osol/osol-0906-x86.iso,6:cdrom,r',
>  "phy:/dev/vgxen/osol,xvda,w" ]
> vif = [ "mac=00:16:36:10:f5:69,bridge=eth1" ]
> on_reboot = "destroy" # comment out this after installation
> kernel = "/xen/osol/unix"
> ramdisk = "/xen/osol/x86.microroot"
> extra = "/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom"
> EOF

- Create the logical volume for the OpenSolaris installtion

[root@xen2 osol]# lvcreate -L 15G -n osol vgxen
  Logical volume "osol" created
[root@xen2 osol]# ls -l /dev/vgxen/osol
lrwxrwxrwx 1 root root 22 Jun  3 22:14 /dev/vgxen/osol -> /dev/mapper/vgxen-osol

- Start the Xen VM, and hit enter a few times for the defaults

[root@xen2 osol]# xm create ./osol0906install -c
Using config file "././osol0906install".
Started domain osol0906install (id=1)
                                     v3.4.0 chgset 'unavailable'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: opensolaris
Remounting root read/write
Probing for device nodes ...
NOTICE: xdf@6: failed to read feature-barrier
NOTICE: xdf@51712: failed to read feature-barrier
Preparing live image for use
Done mounting Live image
USB keyboard
 1. Albanian                      23. Lithuanian
 2. Belarusian                    24. Latvian
 3. Belgian                       25. Macedonian
 4. Brazilian                     26. Malta_UK
 5. Bulgarian                     27. Malta_US
 6. Canadian-Bilingual            28. Norwegian
 7. Croatian                      29. Polish
 8. Czech                         30. Portuguese
 9. Danish                        31. Russian
10. Dutch                         32. Serbia-And-Montenegro
11. Finnish                       33. Slovenian
12. French                        34. Slovakian
13. French-Canadian               35. Spanish
14. Hungarian                     36. Swedish
15. German                        37. Swiss-French
16. Greek                         38. Swiss-German
17. Icelandic                     39. Traditional-Chinese
18. Italian                       40. TurkishQ
19. Japanese-type6                41. TurkishF
20. Japanese                      42. UK-English
21. Korean                        43. US-English
22. Latin-American
To select the keyboard layout, enter a number [default 43]:
 1. Arabic
 2. Chinese - Simplified
 3. Chinese - Traditional
 4. Czech
 5. Dutch
 6. English
 7. French
 8. German
 9. Greek
10. Hebrew
11. Hungarian
12. Indonesian
13. Italian
14. Japanese
15. Korean
16. Polish
17. Portuguese - Brazil
18. Russian
19. Slovak
20. Spanish
21. Swedish
To select desktop language, enter a number [default is 6]:
User selected: English
Configuring devices.
Mounting cdroms
Reading ZFS config: done.

opensolaris console login: jack
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
jack@opensolaris:~$  pfexec uname -a
SunOS opensolaris 5.11 snv_111b i86pc i386 i86xpv Solaris
jack@opensolaris:~$
jack@opensolaris:~$ ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
xnf0: flags=1004843 mtu 1500 index 2
        inet 192.168.96.121 netmask fffffc00 broadcast 192.168.99.255
lo0: flags=2002000849 mtu 8252 index 1
        inet6 ::1/128
xnf0: flags=2004841 mtu 1500 index 2
        inet6 fe80::216:36ff:fe10:f569/10

- Note that once the installation system boots up, we login with username/password = jack/jack. We use "ifconfig -a" to check the system's IP address (was configured by DHCP, VM NIC is bridged to physical host's eth1). The IP in this case is 192.168.96.121

- Now here comes a bit of a tricky part, the Xen console is text only, the OpenSolaris installtion is GUI only (yuck?) In order to get GUI access we will use a built in VNC service in the OpenSolaris Live CD. To get the password for the default vnc session, perform the following command on the Centos Host machine

[root@xen2 osol]# xenstore-ls | grep pass
     passwd = "guzc/VCg"

- Connect to the live CD and launch the installer

vncviewer 192.168.96.121:0

Continue with the installer GUI





- Once done, click Reboot on the installer. The VM actually powers off, which is what we've configured and what we need.

- Create the run-time Xen config file

[root@xen2 osol]# cat > osol0906pv << EOF
> name = "osol0906pv"
> memory = 2048
> vfb = [ 'type=vnc,vnclisten=127.0.0.1,vncdisplay=0' ]
> disk = [ "phy:/dev/vgxen/osol,xvda,w" ]
> vif = [ "mac=00:16:36:10:f5:69,bridge=eth1" ]
> bootloader = '/usr/bin/pygrub'
> EOF

- Start the VM and enjoy :)

[root@xen2 osol]# xm create -c ./osol0906pv
Using config file "././osol0906pv".
Started domain osol0906pv (id=2)
                                v3.4.0 chgset 'unavailable'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
NOTICE: xdf@51712: failed to read feature-barrier
Hostname: osol0906pv
Configuring devices.
Loading smf(5) service descriptions: 150/150
svccfg import warnings. See /var/svc/log/system-manifest-import:default.log .
Reading ZFS config: done.
Mounting ZFS filesystems: (6/6)
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair

osol0906pv console login: dev
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
dev@osol0906pv:~$ pfexec uname -a
SunOS osol0906pv 5.11 snv_111b i86pc i386 i86xpv Solaris
dev@osol0906pv:~$

Hope that's useful to someone out there

Nexenta 2.0 is out

Check out the googley style toons at
http://www.nexenta.org/os

Magical apt-clone, broken system recovery

I have received some very nice feedback regarding my last blog on D-Bus, thanks guys. This week I'm going to play once more with my Nexenta installation that we completed in part1. This time around I am going to basically demo the integration the Nexenta folks have pulled mixing apt-get with ZFS file-system capabilities, resulting in transactional system upgrades, with multi-snapshots that makes system recovery a snap!

We start off booting our nexenta NCP2-alpha installation, which we'll be upgrading to beta-1 as we speak. Grub shows the default boot kernel



Once booted, and we log in, let's update apt's cache



I have also used the command "apt-clone" to list the currently available boot environments. Basically, using ZFS snapshots the whole machine state is snapshot'ed resulting in a instantly bootable environment. Currently we only have the default environment "rootfs-nmu-000". Now I am going to use apt-clone to "dist-upgrade" the whole system. If this is new to you (who has never seen Ubuntu :) it basically means you're upgrading the whole system to the latest tasty bits



Note that I have used verbose mode "-v" in order to get more information. After a lot of downloading and dpkg grinding, I now have the latest packages installed. apt-clone has created a new system snapshot "001" and the process ends like so



I have simply hit 'n' to discard any automatic changes, then I used apt-clone to activate the upgraded snapshot as instructed by the on-screen output



Now let's reboot and see what kind of coolness we have



Oh cool, Grub now displays two bootable environments! One before the upgrades and one after. The dates besides each entry make that clear. Great, now let's reboot into the new envrionment and list the available clones (cloned file systems)



Note that the "A" and "C" in the output mean "Active" and "Current". Great, now creating instant bootable system snapshots like that is so useful that it does not relate to software installations only. Say you're running some third party closed source installation application and you're not really sure what's it going to do to your system. Or if the new clueless administrator needs to install something to the server but you think he's probably going to destroy it! In such situations, it's very helpful to manually create a system snapshot. So, let's do just that to create the environment 002 and let's activate it



Now we have our own little safety net. Let's now boot into that new environment. Use apt-clone to make sure "002" is the active and current environment. Then's let's simulate complete system failure. I will not be too creative and will simply delete the kernel and reboot!



Once in Grub, boot the new and destroyed system. As expected Grub errors out and we're staring at



So right now we have a broken system. Now do we run to tape, dump | restore, and use some black magic, nope, we simply reboot and choose our safe snapshot 001, boot into that and destroy the corrupt environment 002 and activate the safe one 001 as so



Perfect! Once rebooted, our system is back fully functional. Mind you that taking system snapshots, destroying corrupt ones all take a couple of seconds, and of course does not consume disk space except for the changed disk blocks. Now that we have our server back, let's do something useful with it. Let's "apt-clone install apache2".



Once done, the SMF could not start apache because I had chosen nwam network auto-configuration (think NetworkManager), which apache2 does not like. Anyway, I simply started the SysV script manually, and verified "it works" with a direct telnet connection. You can basically start installing and configuring various server software mostly just like an Ubuntu system. All in all, the Nexenta-2 system is a very promising concept. The integration of advanced Solaris kernel features with GNU userland and attempting to port Ubuntu repos, would basically be paradise. It is however a lot of work. The system is not perfect yet, for example in this beta-1, you cannot install X yet (broken deps). That will surely be resolved by release time. However, rebuilding the huge debian/Ubuntu repos, needs a lot of work. If you have free time to push this interesting FOSS project forward, please do so. Visit nexenta and join their development team. Meanwhile and separately from nexenta, I have been interacting with various open source developers working with opensolaris projects. Unfortunately it seems that Sun is not really making it easy for "outsiders" to work on opensolaris projects. I have heard such comments multiple times, from different people. One would think that after so much time, Sun would start to understand what it takes to build a community around any piece of software! Let me use the moment to shout out: Sun, please cut the community some slack! :)

ZFS a smash hit

Pretty freaking amazing

Nexenta, Can you say SolaBuntu (part1)

This time around I will be playing around with nexenta OS. Nexenta is a marriage between OpenSolaris and Ubuntu, or what I like to call SolaBuntu :) Solaris has a pretty decent record in the data-center. It is a solid and widely trusted paltform, however, it was showing its age pretty badly. Many of the most commonly used tools were outdated. And most users begin their *nixy knowledge with Linux, and when faced with Solaris they don't find the GNU tools they have grown accustomed to, and thus don't like the platform. OpenSolaris has been started to fix some of those issues, however, nexneta takes this concept to an extreme. It basically takes the ubuntu userland and plugs underneath it an opensolaris kernel. Nexenta also integrates unique solaris features such as zfs with ubuntu tools like apt-get to provide system wide transactional safe upgrades. I really really like this feature. It basically means you can dist-upgrade your entire system, and once up on the new version, if something is badly broken, you can very easily boot back an earlier snapshot (pre-upgrade) image of your system. We will explore those features later, however, for now, let's get started installing NCP2 alpha. As usual I am choosing to go with the latest and greatest and least stable version ;)

Download the iso, burn it, boot it and you're faced with grub



As you can see, this is based on Ubuntu Hardy. Simply hit enter and the solaris kernel boots on your system. After the boot, you get a splash screen welcoming you to nexenta



Now I'm gonna throw a lot of images, just to give you a feeling of the installer, but I'm also gonna skip some pretty obvious screens that would be mostly obvious (a la are you sure ? Yes/No). Next you choose your country



Now here is an interesting tidbit, the whole installer is running inside a screen session. So, a CTRL-A followed by a 2, takes you to screen window 2, as you can see you can use the format command to view and partition your disks as needed. Window 3 is for logging the installer actions. The log is not very verbose however!



Afterwards, you get to choose your disk, then confirm it



Afterwards, the disk is zfs formatted, and package installation begins .. Go make the usual cup of nescafe



After this step completes (around 20 minutes), you get to specify the root password. "root" can be used for direct login (no need for sudo or pfexec here).



You also create a normal user, set its password. You specify the machine name "nexy" in my case, and domain name. Afterwards, boom the installation is complete. You're ready to boot your new environment



After the reboot, grub starts, load the opensolaris kernel, and a few seconds later, you get the default login screen, which is not a pretty GUI! The default installation is a minimal installation without X.



Once logged in, you can start enjoying apt which is IMO the best main stream package manager. Let update the repos



Let's see how good the repos are. apt-cache show the system has access to around 4700 packages. This is of course not comparable to debian or ubuntu's repos, however, it still is fairly impressive considering how young the project is. If you're feeling excited, one of the best ways you can help is by becoming a packager for nexenta, so go ahead and join their hackathon.



One of the things that make nexenta an attractive opensolaris based system, is that it provides a solid array of open-source software tools that are up2date! This is pretty important at least to me. Let's check a couple of common packages



Your system is now ready, you can apt-get install any software package that's available and start running your nexenta based server. I have to say that my overall feeling is that nexenta core platform is more oritented towards being a server OS more so than a desktop one. Not that you can't install desktop apps, but rather that's what the default install offers, and that's what the core dev team seems to be focused on. NCP-2 alpha is definitely promising, and I really like the idea of an ubuntu/debian/GNU system on a solaris kernel. However, the project is very young, and I did face some bugs along the way, the is to be expected however as this release is still alpha quality. Hopefully along the upcoming posts, I will take that system for a ride and blog about it. Let me know about your experiences as well.

Update: Read part2

OpenSolaris 2008.11 pre-release, The year of the laptop?!

I've always been a Unix junkie, of course mostly running Linux for a decade now, but Solaris is starting to look more and more like a real modern option. We use a lot of opensolaris at $dayjob mostly just because of its wonderful filesystem zfs. I have to say that I am excited about opensolaris much more than I ever was with Solaris proper! With older releases, the non GNU userland was just simply painful and by my "sense" pretty stupid in many ways. Most of the pain is now gone with the new opensolaris releases. Since the release of opensolaris 2008.11 is pretty close (well it's November now already), I decided to snatch the pre-release iso and take it for a spin on my HP Pavilion 6700 laptop!

Visiting http://www.genunix.org/ to download the iso file, I snatched the file which was significantly more than 700MB, so I had to burn that to DVD. The opensolaros 2008.11 iso is both a Live DVD that is also installable. That is sweet, as I can run opensolaris now directly from the live DVD and install latter if I want. Starting the DVD was easy, just pop it in, BIOS boots chooses DVD. Afterwards you stare at a GRUB screen, I simply chose the default option and let it boot the GUI environment. You get asked for keyboard layout and language, I just hit "enter" for the defaults (US/English). More than a minute later (booting is slow since it's using an optical drive), you get the GUI desktop. It's basically a Gnome envrionment.

Let's make sure we're really on opensolaris :)

jack@opensolaris:~$ cat /etc/release
OpenSolaris 2008.11 snv_100a X86
Copyright 2008 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 24 October 2008
jack@opensolaris:~$ uname -a
SunOS opensolaris 5.11 snv_100 i86pc i386 i86pc Solaris

As you can see the default desktop looks clean and is fairly attractive. After that I started the DDU (Device Driver Utility), a nice utility made by Sun to check your hardware compatibility level with opensolaris. I was of course expecting issues here as this is a recentish laptop bought only a couple of months ago and opensolaris' hardware support on generic x86 hardware not mention laptops is well, less than stellar to say the least. Or at least that was the impression I was under. I was however very positively impressed when the DDU said "Driver Problems: 0" and showed that there were drivers for every single piece of hardware in that laptop. That was actually nice!


Other nice touches as well, is the laptop's screen automatically started in the native 1280x800 mode, and was using the native (closed) nvidia driver



As you can see, life is not full of roses, the volume while works fine and audio generally is ok, however, it seems the volume bar can only control the "master" volume mixer! My Fedora10 rawhide shows a zillion volume bars for all of my hardware features (line in/out, headphones, pcm, ... etc). In general I felt the audio stack needed some work on opensolaris. I wish they'd put in an alsa compatibility layer, that should probably help porting applications. Another issue, was connecting to my WPA2 home network. At first the network applet was very stubborn and would not connect to my network. After a lot of clicking here and there and displaying dialogs which were asking for (username?!) and password for the wireless network, it somehow got connected. After that the connection seemed pretty stable.

Being on the Internet, I started Firefox 3.0.1, pretty solid and fast as usual. Gotta love open-source portable apps ;) Now is the time to put this OS to the test. Let's see if it's gonna let me on youtube.com! It becomes quickly apparent that flash player is not installed by default. A quick visit to adobe, I download a tar file, uncompress it. There were no READMEs of any sort, just a couple of binary files. However creating a "plugins" directory under ~/.mozilla and copying those 2 files under it, seemed to have done the trick. I can now play youtube videos with no problems. Actually the whole video playback experience, and generally firefox seems faster and more responsive than under fedora. All in all, I'm very happy with the browsing experience.


Now I decided to go ahead and take a shot at crashing suspending my laptop! In order to test that, you will need to enable it by adding a line in /etc/power.conf:

S3-support enable

And then enter:

pfexec /usr/sbin/pmconfig

Afterwards, I could not find any GUI elements or commands to initiate a suspend! I hit the power button, and Gnome popped up a dialog asking what I wanted to do, I chose "Suspend" and a couple of seconds later, the laptop was suspended, with the power led blinking. I thought "sweet" .. It was pretty late, so I went to sleep, and thought I'll see if it wakes up (usually much harder) tomorrow. The next day, I open the lid, and boom, it starts back where it were! A fully successful (Hurraaay!) suspend/resume cycle, well I really did not expect that from opensolaris! Here is the relevant log part from /var/adm/messages

Nov 6 19:25:42 opensolaris genunix: [ID 535284 kern.notice] System is being suspended
Nov 6 19:25:44 opensolaris nvidia: [ID 702911 kern.notice] NOTICE: NVRM: RmPowerManagement: 4
Nov 6 19:25:44 opensolaris acpica: [ID 530733 kern.notice] NOTICE: acpica_ddi_setwake: could not evaluate _PRW
Nov 7 15:13:34 opensolaris last message repeated 9 times
Nov 7 15:13:34 opensolaris pcihp: [ID 709949 kern.notice] NOTICE: pcihp (pcie_pci0): card is inserted in the slot pcie1 (pci dev 0)
Nov 7 15:13:34 opensolaris genunix: [ID 358043 kern.warning] WARNING: ieee80211_crypto_enmic failed.
Nov 7 15:13:34 opensolaris mac: [ID 486395 kern.info] NOTICE: iwk0 link down
Nov 7 15:13:34 opensolaris acpica: [ID 530733 kern.notice] NOTICE: acpica_ddi_setwake: could not evaluate _PRW
Nov 7 15:13:34 opensolaris last message repeated 1 time
Nov 7 15:13:36 opensolaris nvidia: [ID 702911 kern.notice] NOTICE: NVRM: RmPowerManagement: 5
Nov 7 15:13:36 opensolaris gnome-session[756]: [ID 702911 daemon.warning] WARNING: Could not ask power manager to suspend: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Nov 7 15:13:36 opensolaris in.routed[474]: [ID 238047 daemon.warning] interface iwk0 to 192.168.11.3 turned off
Nov 7 15:13:38 opensolaris /sbin/dhcpagent[2275]: [ID 967406 daemon.warning] refreshing state on iwk0
Nov 7 15:13:39 opensolaris genunix: [ID 583038 kern.notice] System has been resumed.

My conclusion at this point, is that opensolaris is now much more user friendly than it once was, it had no problems detecting my hardware and using it, most of the same applications I use on Linux are there for opensolaris as well. Support for laptops and suspend/resume is not that bad it seems (or I'm extremely lucky!). However, more work needs to be put into improving the audio stack, and other niceties like bluetooth. I suspect more hardware support is also necessary. All in all, using opensolaris as a desktop seems viable if you're coming from a Linux background at least. I still prefer Linux of course for the convenience and for knowing I can almost get anything done with it. However, I will keep a close eye on opensolaris. Overall, I am impressed not only with the current state of the system, but more so with the rate of progress that has been made!