How do we do installation tests after package uploads

So, yesterday we received a new version of MongoDB packages in Saucy, and voila, not correctly installable anymore. The bug report is here: mongodb 1:2.4.6 has init script in wrong package and the upload of a fix is already in saucy proposed (oh wow my first upload since ages ;))

So, the culprit was a dangling wrong mongodb.upstart script inside the debian directory.

I know the QA team does test installation runs, but what I would like to see is something like a LXC minimal installation with the previous package in question installed and than trying to upgrade this container with the new package.

I really don't know, if this is easily doable (I didn't dig my head into this swamp), but to improve the QA goals for Ubuntu we should make sure that all packages are installable all the time (this includes all the maintainer scripts which eventually will be added automagically). If they aren't we should drop the uploaded new package and send a message out that installation/upgrade wasn't successful, including the output of dpkg/apt, to have a clue where the bug is.

Comments? Ideas?

Tales of the Python Knights

I would like to share this tale with you

If you are a Python Knight, just answer the battle call by sending a message to Kieran Scully ( k.scully a.t. )

I'd like to tell you a story.

Once upon a time (that time is now) in a land far far away (Central London if you must know), a King was building his empire. Just 5 years ago he had but 14 knights helping him in establishing his kingdom in a highly competitive place. You see there were so many Kings creating Kingdoms, and many had fallen within the year when larger Kingdoms came along to pinch their best knights.

Whilst his Kingdom was small, his vision was huge. He wanted to create a kingdom that would send shivers throughout. He knew he had to focus on one specific area to recruit his knights, the basis of their skills would form the backbone of his army. After considering his options the king decided that his knights would be of the Python realm. His decision for this was not one in haste, instead it was well thought out. He loved the collaborative environment and wanted his knights to work together and share their work with the greater community. He wanted the source of their work to be of the open variety and their basis robust.

The kingdom took shape and continued to grow at a rate far greater than anyone could of expected, the King included. All of a sudden the presidents of other lands were seeking out the King to ask for help in building their digital recognition and centralising their marketing. They knew that he had the digital asset management system that no others could create. With more and more lands needing his help he grew his army to close to 700 knights of mixed skills with Python still the core.

One day a foreign land came to him to ask for his help. They had tried to replicate one of his products but had failed and needed his help. After weeks of conversations around the round table of django they reached a multi-million dollar agreement. This was great news for the King, but he was still missing his knights. Some of his knights had become stronger and progressed. He was in need of a range of knights. Some who had been battling the land with the power of Python for several years and were now Senior Python Knights, and others who were even greater in their experience - with the power to Lead the Python Knights and Architect the process and product.

The King sent out a message to the land asking for help in finding the best Senior Python Knights and Lead Python Knights that the land had to offer and this message reached me.

So - are you the knight that can help build the kingdom and continue to expand the empire?

PS: I'm not getting paid to do this, jftr!

Some ideas for relaxation

I bought a couple of weeks back an Amazon Kindle eBook-Reader and I am a huge fan of it.

Since I bought it, I was reading like hell a lot of my favorite authors.

So, to give you a glimpse of what I am reading for relaxation and what you should do, too, when you are into reading 'strange' thriller fiction, here is a list:

  • Layton Green - The Dominic Grey Series

His books are a mixture of being a Dan Brown story and Brett Battles plot of Jonathan Quinn.

Start with 'The Summmoner' via 'The Eqyptian' to his latest writing 'The Diabolist'.

The books are about, well, let me phrase it this way, a Sherlock Holmes & Dr. Watson couple, but set in todays world and more into a religious, voodoo focused storyline. I think, what you Ubuntu people will like most, is the mentioning of 'Juju' in 'The Summoner'. He explains what actually 'Juju' is, and that 'Charms' are very related to 'Juju'. The definition of 'Juju' on Wikipedia - Juju is very close, but there are also some hints where it all started.

  • Graham Masterton - Katie Maguire Series

I only had the chance to read 'White Bones', but this was really a ride. I think, Stephen King would be really jealous, reading this book. It's detailed, and I mean, very detailed. His description of how people are getting murdered, are really colourful. You can see the pictures, the flesh-less bones, the pain. I doubt, there will be a movie version, because there is too much detail, and it has to be so detailed. This book gave me the creeps, and it will give you the creeps, too.

  • Brett Battles - Jonathan Quinn Series

Oh, well, I can't force you to read these 7 fictional novels, but honestly, you should. It's about a man who is on the other side of 'James Bond'. Jonathan Quinn works in the spook business as a 'Cleaner', but despite of his job, he is a very moral driven person. Mr. Brattles is not actually writing a thriller story, but more a phsycological bio of someone who deals with dirty jobs, but actually trying to be honest and helpful. As I am also a Tom Clancy fanatic, especially his 'Jack Ryan' novels, I would just say, Brattles is trying to picture 'Jack Ryan' but from the other side of the desk. Being a 'Cleaner' is a dirty job, but 'Jonathan' is really only cleaning up the scumbags.

  • Simon Wood - No Show

Ok, this was a mistake. I trusted Amazon on this one, but it was wrong. It's actually too easy to read, and has no depth. The characters described are totally transparent, and the whole story is actually told after 15% of the book. British guys marries a girl from California, after some time he is able to become a US citizen. When reaching San Francisco his wife didn't pick him up and he is worried. Now, becoming a suspect of the Police, being hunted by the neighbor-hood watch, and with some little help from a guy who is running somekind of a Pub, he finds his wife...and the story ends without a happy end. Totally stupid, not well written, no detail, no depth. So don't buy this book, it's not worth it.

  • Dan Brown - Inferno

What should I say, it's good, it's bad, it's good, it's bad. I don't know. I like Dan Browns detailed writings, most of it you can go and see for yourself. And yes, sometimes he is overdoing it, but it's a good read, like all his other novels. This time, again, he takes his main character, 'Robert Langdon', on a ride through Europe. And again, he is using another persons idea to its limit. This time it's 'Dantes Inferno' which he position into todays world. A good read, but even as ebook it's still expensive. Btw, if you want to read 'Dantes Inferno' you can check out 'The Gutenberg Project'. They have a lot of translations of this lyric. I think I like this translation most: The Project Gutenberg EBook of Divine Comedy, Longfellow's Translation

So, this is for our 'we are still reading books' people.

But I have also an advice for our 'movie fanatics'.

Sadly, in Germany, the release dates of good movies are behind the international releases, but anyway:

  • Trance - Danny Boyle

IMDB Entry 'Trance' Awesome movie with James McAvoy (X-Men: First Class) and Rosario Dawson (Zookeeper) as main actors.

It's one of the stories, where you don't know what's real and what not. Especially Rosario is one of main characters here. She's acting so good, that you think she's only a sidekick.

  • This is the end - Evan Goldberg, Seth Rogen

IMDB Entry 'This is the end' Living in California, had a broken toe, went to a doctor and got some Dope as Prescription?

Than this is for you, a totally crazy movie about the Apocalypse. The actors are actually playing themselves (hopefully not ;)). I don't want to go into details, but this is real fun (not the german translation though). Especially Emma Watson, even when only seen for a couple of minutes, she's so awesome, that you even forget that she was Hermine in Harry Potter ;))

  • Man of Steel - Zack Snyder

IMDB Entry 'Man of Steel' Ok this is bad. Really bad.

So, let's get this right: Christopher Nolan is one of the producers, and he co-wrote the script to this movie. When you hear: 'What? Chris Nolan? That has to be good, because he did a very good job on the Dark Knight', it's wrong. Totally wrong. This movie has nothing from 'Batman'. I don't know what Nolan or Snyder thought but this is more a different plot of 'Matrix'. So, in the Prologue, the first 5 - 10 minutes, you think: Oh wow, this movie will be good, but when you see the 'Cloned Baby Eggs' you are pulled back to 'Matrix'. For me, as someone who was growing up with 'Superman' and 'Christopher Reeve' it was so 'crap, I thought Nolan wanted to give us a different view of the Superman/Clark Kent character', but in the end, it's just: 'Matrix' and 'let's destroy Metropolis'. Ok, this movie is better than 'Superman Returns' but still, not really a good watch.

  • Pacific Rim - Guillermo del Toro

IMDB Entry 'Pacific Rim' My Favorite. Really.

The CGIs are awesome, and the story is a mixture of old japanese 'Godzilla' movies and 'Power Rangers'. Why 'Power Rangers', well, in the old Bandai TV Series the 'Power Rangers' tried to fight their enemies together, in a way, where they assembled a Robot (similar to a Transformer). So, the team riding the 'Jaeger' in 'Pacific Rim' is similar.

When you remember the 'Godzilla' Remake of Roland Emmerich from 1998, there is a scene where 'Godzilla' roars, and when you really watch 'Pacific Rim' there is a 'Kaiju' Monster, which also roars, and the silhouette is totally matching the scene from Emmerichs 'Godzilla'

  • Pain and Gain - Michael Bay

IMDB Entry 'Pain and Gain' This movie is based on a true story.

Yes it is, and people are discussing this movie highly controversial. The main characters are still alive, so the authors of the script changed the names. What I can say, it's a fun movie, and you shouldn't take the true story into account. I read about the truth later, but still, this movie is fun, especially seeing Mark Wahlberg and Dwayne 'The Rock' Johnson playing the 'dumb body builders with a plan'.

mod-wsgi segfault fix...

...or how to improve your Google-Foo

this morning I was doing some coding on a simple wsgi app with and mod-wsgi under Ubuntu Saucy. So, now Saucy has Apache 2.4 on board, and somehow after starting a simple apache2.4 config with mod_wsgi configured in daemon mode, I saw a strange logmessage, which told me something like this:

[Sat Aug 10 14:06:18.945155 2013] [core:notice] [pid 19099:tid 140698944931712] AH00051: child pid 19105 exit signal Segmentation fault (11), possible coredump in /etc/apache2

First I thought, it's my simple wsgi application, but looking deeper, I was pretty much sure, it wasn't my app.

First looked on Launchpad, if there is any bug filed about this issue, but I couldn't find anything related. Second I looked on, but honestly, I didn't find anything which would match my bug.

Finally, Google:

Searching for: apache 2.4 mod_wsgi process has died

And some results were popping up.

Not only looking on the titles, but also correlating dates, I found my bug as fifth entry on the search result page: Immediate Segmentation Fault in Daemon Mode

Well, the last entry in this thread gave me a hint: mod_wsgi 3.5, but actually this is not released, so I could only guess the patch is somewhere in the tree. It looks like that the author bisected, or just guessed it right, so I checked upstream source tree on googlecode, and found the mentioned changeset.

Scoreboard handle in daemon mode should be set to NULL for Apache 2.4 to avoid crash in lingering close.

Ok, now the work starts.

  • DGet the source package from launchpad,
  • dpkg-source -x it
  • quilt push -a
  • quilt new
  • quilt add mod_wsgi.c
  • finding the correct source area
  • apply the fix
  • quilt refresh
  • debuild -S
  • build the package on my local buildserver
  • install it, restart apache and check the logfiles
  • segfault's gone


Now for the paperwork:

  1. file a bug in launchpad (this is the bugtracker for my distro of choice)
  2. create a debdiff between original package in the distro and my new one
  3. attach debdiff to bugreport
  4. write an email to with a similar bugreport, attach debdiff to the mail, send it.

Now let's wait until this is getting fixed.

But why is this bugging me, that I have to write a post for it?

Because, apache and mod_wsgi is so commonly used, and one of the tasks for developing a distro is that we should test those critical packages much better. This segfault is very easy to find, but eventually not so easily to find with automated tests. It also needs some work like cherrypicking commits from upstream, apply it to the package, and rebuild it and finally do some real world testing.

Eventually this would be one good excercise to leverage a CI like Jenkins much more. If I have the time to play around, eventually it would make sense, to use LXC on a build slave, creating saucy containers, installing apache installing modules, and having a simple script task which eyes out for segfaults in certain combinations.

But still, we need more human triggered tests, means real world testing, but this alone doesn't give us any bugreports or fixed packages, right? So we need more people, who are working as SysAdmins, and doing real world tests, and who have a clue on how to fix software.

Every SysAdmin should know his/her OS of choice, and should be able to fix bugs for their distro of choice. People Managers should think if they put that as requisite when writing job descriptions.

New blog

After a long time, I am trying to revive this blog. So, I decided again to go back to self hosting services, and changed from to Octopress.

It's integrated with my own instance of GitLab and updates automagically.

I migrated all old posts from to Octopress, thanks to this little Gist

More to come during the next days and weeks.

Good to be back, btw.

My appologies if there are some old posts coming up do to the migration of from, I hope it works smoothly and without interruption of the usual planet traffic.

Time flies...

yes, time flies.

And I didn't write anything on this blog since April, just before I started again for COL.
But that's the issue when you are busy with Life and Work, right?

So what happened during the last couple of months?

First of all, as already said, I started, again, to work for COL as an Architect in the OPS department. And directly, after my second day back in the Karlsruhe Office, I was already working on a product and company acquisition. Success. Product was migrated to our infrastructure in less then 3 weeks, and was ready to be shown during the Citrix Synergy event in San Francisco (while you, fellow Ubuntians were enjoying the UDS in May in the Bay Area :))

Beginning of June I traveled to the USA. Oh well, me and the States. Difficult topic. But, you know what, I'm am actually here:

Santa Barbara, CA (Courtesy of Wikipedia Author John Wiley)

Yes, this is California, this is Santa Barbara. And no, I'm not on holiday here, but actually it feels like.

Sadly, I had to leave my family over there in Germany. Well, next time this will change.

To be honest, I had to change my whole opinion about the american people, especially here in California. I didn't meet anybody who was unfriendly or nasty to me, this german bad boy.

I met good, friendly, open minded people, and some of them I  already call 'Friends'.

Today is actually my second month here, and I still have one month to go.

Right now I'm working on different projects. And one of these projects will be to support Ubuntu in our datacenters. Working on that is a challenge, because right now it's only RedHat. So we have to change a lot of infrastructure. Distribution agnostic deployment system, Puppet, and other cool topics.

One of the coolest topics, but, is working with one of the FreeIPA maintainers. We need to support FreeIPA on Ubuntu, which is not that easy right now, because the state of FreeIPA in Ubuntu is more than perfect.

"But FreeIPA is a RedHat/Fedora Project" you will say, but I have to admit, that FreeIPA works (on Fedora), and it's more then cool.

But, when there is a will, there will be also a way. And there we go.

(Linked from:

Collaboration to the rescue.

The FreeIPA Upstream team is very helpful. And having one of the contributors sitting 2 or 3 cubes away from me, is even better.

Anyways, I filed already some nasty bug reports, and we will fix them for the future.

On of the next steps should be to provide support for FreeIPA 2.2.0 (or even FreeIPA 2.3.x, which is in beta state).

Anyways, I don't want to bore you with technical details, this will be another blog post, I'm already writing :)

So, how is the life here in SBA, CA as a German?

Life's easy :) One day you go deep sea fishing, and on the other day you fly to the Silicon Valley, San Jose with your Boss to do some datacenter work.

My greek Boss :)

Beautiful, isn't it?

Happy Happy Smile

And if this is not enough action, actually in SBA you can do better.

Canary Hotel, you'll get German Erdinger Weissbier from tap.

When you are around this area, just visit the Bouchon Restaurant, this is awesome, not cheap, but the food is really, really, really (do I repeat myself???) [sounds like Jono] AWESOME!!

Especially when you are a lucky guy like me, not paying for it ;) And when you are even more lucky, as I was, the owner of this restaurant will speak German to you, and present you this good mexican "water":

Anyhow, if you are missing your own country, especially the German Country, go to Brummis Restaurant, and enjoy the German hospitality.

Is it just fun to be here? No, not at all. But when you are surrounded by the sun and you palmtrees infront of your house, it is fun. Promise.

Anyhow, I'm not just here for the fun. I have to work, and I'm actually doing a lot of work.

DC² is actually having  a 1.0.0 release in the next upcoming weeks.

There are some bugfixes and improvements to several Debian and Ubuntu packages waiting on my disk to be pushed out to the bugtrackers.

Integration work between Ubuntu and FreeIPA, and as well some documentation on how to communicate to the FreeIPA server with remote APIs in a secure manner (with Kerberos Ticket Delegation from one host to another host and then executing commands on this one without being logged in, sounds weird? Yes, impossible? No !! And it's fun, believe me).

And some other work on projects, you can find on Launchpad, but not in Ubuntu or Debian, like the Percona MySQL Project. They could need some help btw, with regards to a clean build system and preparing packages from upstream source for Debian/Ubuntu and RPM Based distributions.

AND!!! There is CloudStack, the Cloud Project of Citrix Inc. which was handed over to the Apache Foundation.

Anyhow, I don't want to brag more...but I love to be here in this area. So, let's see, it is my first trip since ages to the US, but this won't be my last trip to the US. I could also imagine to move here.

Well, anyways in the meantime I'm working and waiting for the next big bang on August, 15th in Mountain View . Kiss, Moetley Crue and The Treatment will play a concert in the Shoreline Amphitheatre

and this is "just around the corner" so I had to book some tickets. So if you are going to, just let me know, eventually we can meet before or after the concert in Mountain View and have a drink or two :)

On the 16th I'll be in San Francisco ( at least that's the plan ), if this is a place for you to meet up, you know how to get in contact ;)

Anyways, I heard and read today, that the next UDS will be in Copenhagen, Denmark. This is awesome, and eventually I can combine a business trip to Copenhagen and visiting UDS to have some good discussions with some key people :)

There we go. A lot of updates, good stories, awesome country.

It's time to make some noise

It's April and I didn't touch this weblog for more than 4 months now.
What happend? Why is "SAdig" so silent?

'Cause I'm busy, that's why.

So what happened during the last months? A lot I have to say...

The December 2011 Story

December 2011 was my last month of being employed at Citrix Online Germany, formerly known as Netviewer. Sadly, after all the fuzz and stress about my resignation I became sick and my doctor recommended to stay at home. Well actually he put me on sick leave for almost half of December. I don't know if I should thank him for that or if I should hate him for that. Anyhow, during this time, I had really a hard time. I had fever, my whole body refused to work properly. From 250% to 0% working powers it's just like a cold turkey for a drug addict.
Really, I wasn't ok and my Christmas holiday wasn't really nice.

But this is the past.

The January to March 2012 Story

From January 2012 on I started to work for a consultant Company named Inovex. The good thing of this story was, that I took my work with me. As a result of my resignation from Citrix Online/Netviewer my former bosses hired me as a consultant for the Netviewer system from Inovex.
Normally people will be happy, you start a new job, you have probation time, and you can work on the old daily business again, for better money and with less stress. Furthermore I was able to work on DC² more often and finally had the chance to create good installation packages.

But did this change of job changed anything for me?

You know, it's hard when you are a consultant coming back to your old company. No one will see you as external resource and most likely you don't see the others as customers. Eventually it's a personal issue, but this was how I felt. Therefore, nothing changed for me.
Furthermore, the work I had to do was not that complicated, I built this system in coorperation with my fellow OPS buddies. We created a system which just runs. No big incidents, no serious crashes on the hardware, no serious crashes of the software. The OS is running like a charm, thanks to Ubuntu 10.04 LTS, so not so much to do.
But my job wasn't to do anything exceptional, except writing documentation. Documentation which was already there or not needed anymore, because this system has only a limited lifetime and some of the things we created are not needed anymore. So, sitting there writing documentation for something which isn't needed anymore, doesn't make sense, and  worse, it didn't tickle my brain. I felt like "Crap, I don't do anything".

After some discussions with my family and with myself, I decided to quit this job, too.

Honestly, the people at Inovex are smart people. Friendly, cooperative, fast, and living on the edge of technology. I just say, it was not an issue with Inovex but me, that's why I quit.

I learned in the past, that it's more important to be happy for yourself, and not for anything or anybody else. If you are not happy, you can't make others happy.

The Future

Honestly, I really don't know what the future has for me, but I'm not without a "new" job.
Sometimes, you need some luck and good friends and a company which can learn from the past.

So, I was hired again by Citrix Online, but with a complete different working area. This deal was made on a short notice, and I never expected to do that, but hey, even my family and me are in need of some money for food, drinks and fuel.

I can't write about what 'm doing or what I will be doing, because all of this is very confidential.
I won't work anymore on our old Netviewer System, that's the past.

There are new opportunities showing up, there are really great projects coming along and I'm being involved. There is also Ubuntu work involved.

Furthermore, it is good to hear and read about Citrix Systems Inc. that they are moving their CloudStack Project to the Apache Project and that Citrix System is now a Platinum Sponsor of the Apache Foundation.

A Short Notice To The Fellow Readers

I know that some people will be surprised others will say "I did know that" or "I thought about this already".

All this wasn't planned. It just happened like many other situations are just happening in life.

I won't enable comments for this entry. So if you want to get in touch, you know my email address so write an email.

Life Changes

So, the end of the year is near and with New Years Eve my life will change in some areas.

First of all, I'll be leaving Netviewer (now: Citrix Online) after (round about) 4 years of giving my energy to the company. During these 4 years we (StephanT, SvenW, FelixR and JensG) accomplished a lot.

StephanT and I were designing and building 2 new products on the datacenter level. Long nights, lot of brainfck and (of course we are talking about the real IT people: Operations) a lot of beer. Together with SvenW we worked  out a nice, not so complicated, redundant, high avaiable and secure MySQL infrastructure, we introduced a new monitoring system (OpenNMS), added Puppet Magic to our deployment and we (or let's just say I) developed on top of FAI a new and easy system to deploy bare metal, VMs and everything which is able to run Linux (Debian/Ubuntu/RH/SuSE/...) and does PXE boot.

We changed the behavior between OPS and Development, we introduced On Duty weeks and we succeed to redeploy a complete datacenter infrastructure in less then 48 hours, with less then 15 minutes downtime.

During these 4 years my son Sean Ryan was born and I got married to the most adorable woman of the world. I changed my name, and I finally didn't touch a computer in the evenings (even when this was really hard).

From next year on, I'll be working for a company named Inovex. It's a consulting company and I'll join the Systems Engineering team. I'm really happy to work for them, because some of my former colleagues from ComBOTS (now: Kizoo) are working there too. And what's really amazing, FelixR and JensG are joining Inovex, too.

So, the next couple of months will be filled with new duties, ideas and fresh work.

Furthermore, I'm working on the DC² side of life to have an easy installation process for that, so I can keep my promise to Thomas Lange (FAI Maintainer) that he can release the Enterprise Version of FAI next year ;)

Ubuntu work is also on the list of todos, hopefully I can dedicated more time to Ubuntu again. I'm really missing it.

This is my last post for this year, and I wish to all of you around the world a good time during the upcoming holidays and wish you a good start into the new year 2012.

[20111025] Ubuntu 12.04 Merge/Sync Report

Todays merges and syncs:

  • libunwind (merge, uploaded)
  • fai (nosync, nomerge, waiting for 4.0 release)
  • abiword (sync)
  • agave (merge, uploaded)
  • gnome-mousetrap (sync)
  • hoichess (sync)
  • lasso (sync)
  • libgwenhywfar (merge, uploaded)

DC² goes Android

Somehow I wanted to learn something about Android Development.

So, DC² goes Android. Check the video:

Or click on the source video 

As always, you'll find the source of the application on launchpad.

Have fun.