LXC lies about /.. inode number making FlexLM unhappy

One of our customers wanted to run a flexlm licensed tool on the Amazon EC2 cloud. He turned to us over here at Cloud Niners and we started doing the work. The approach we chose was to start an ec2 Ubuntu precise machine, host a CentOS LXC container on it and run the license daemon inside that. The reason for the CentOS choice is that the tools and license daemons are only certified on RHEL and CentOS is the closest next thing

Trying to start the license daemon, I hit the following error

 8:59:27 (TOOLNAME) Cannot open daemon lock file

Checking the usual suspicions of permissions and the like led no where. Google immediately led me to similar problems for people wanting to run Flexlm tools under Solaris zones (which are quite similar to Linux LXC containers). I was reading about a guy facing a similar problem, and how Solaris' legend Brendan Gregg wrote dtrace scripts to run-time patch memory structures to resolve the issue.

At first I was dismissive that this was what I was actually see'ing. A quick "ls -lai /" just to confirm /. and /.. actually had the same inode number, and I was sure it was something else

# ls -lai / | grep '\.$'

  256 dr-xr-xr-x   1 root root   212 Dec 26 09:08 .

  256 dr-xr-xr-x   1 root root   212 Dec 26 09:08 ..

but I had one of those mmm moments after stracing the binaries and confirming they are actually failing immediately after calling getdents on "/". This sounds suspiciously close to what the Solaris folks were see'ing. I grabbed gcc and built the sample code in getdents kernel doc page (Thankful!) And much to my surprise, inode numbers for /. and /.. were actually different!

# ./a.out / | grep '\.$'

     256  directory    24            1  .

   42669  directory    24          2  ..

Of course this makes sense since the LXC guest was just another directory on the host, but I didn't suspect ls -i was actually lying inside guests!

At this point, I'm not exactly sure how to resolve my issue apart from reinstalling the LXC guest on a separate block device (like lvm) which I think should resolve the issue. This blog post is simply to confirm the issue, and to gather feedback and potential solutions from smart people reading this. Shoot me a comment

Celebrating Cloud9ers is MEA's leading Amazon Consulting partner

It's been quite a while since I last blogged, I think it was over a year ago! I've been taking a break from blogging and focusing on Cloud Niners my baby startup that I co-founded with a bunch of life long friends.

While many things are interesting over here at Cloud Niners, this blog post is specifically to celebrate the fact that we've just been upgraded to become an Amazon Consulting Partner. I believe this makes us the first active middle-east Amazon AWS cloud consulting partner and certainly the best :) I do hear a lot from Amazon that we are the cloud leader in the middle-east! I am quite happy that we've achieved this milestone. I'm a big believer in cloud computing, and I simply enjoy helping our clients understand how the cloud (and Ubuntu server) can help them achieve more!

My team and I do custom cloud-application development, where we write top notch html5/CSS3/JS web applications that run and make use of the Amazon AWS scalable cloud infrastructure. We also do mobile applications backends and front-end (Android, iOS). We usually use the frankly excellent Ubuntu server, unless the customer forces us otherwise! Ubuntu server is simply the best OS for the cloud IMO. Our DevOps team helps our customers migrate and run their apps on the cloud (and on Ubuntu server) at top quality!

If you've read so far, here's a Gift for you! I'm offering my (and my team's) help (up-to-24hrs) for free in the next 30-days. If you're thinking about coding a custom cloud app, or have questions about any Amazon services, I'll give you free consultation and answer any questions. Shoot me a comment over here, or contact the team. Oh yeah, feel free to share this blog post with friends, to let them have the free goodies!