(DC)²: Going forward to a 1.0 Release

  |   Source

It was a bit blog silence about me during the last 2 months, but I was really busy with some projects @work.

Today, I would like to write some bits and pieces about the "DataCenter Deployment Control" Project aka (DC)².
In my last article you could see (DC)² in action, deploying some virtual machines on a Xen Server (or on VMWare or on BareMetal or on every device which is able to do some PXE).

At this time, (DC)² was using as backend the Django Framework and as RPC module the fantastic RPC4Django. As database engine was MySQL in use. We used as tftp server the very good working tftpy and as PXE bootloader PXELinux from Syslinux. As frontend development framework I used the "Qooxdoo"  JavaScript framework.

Now, I was improving all of this.

The Backend

First of all, I replaced Django and RPC4Django with web.py and a self developed XMLRPC and JSON-RPC module. With less overhead all RPC calls are much faster now.
Furthermore, I revisited the whole RPC namespace and refactored most of it.

Another important change was to go away from the relational database (MySQL), as this was introducing more complexity to the project.
When I started to think about moving away from the relational model to a document oriented model, I was  giving first CouchDB a try. But CouchDB wasn't the best candidate for this, so I had a look at MongoDB.

And MongoDB it is.

So, with MongoDB and PyMongo you can work without special table models, but if you want you are able to implement a relational db style, which was needed from some workflows in my case.
Furthermore, the replication and sharding functionality of MongoDB was exactly what I was looking for. Easy to setup and configure.

And MongoDB gives you JSON output, or when you work with PyMongo native dictionary types, which was important to me, because one feature I wanted for (DC)² that its documents can be easily improved.

Example:

We do auto inventory for servers. That means, I needed some infos from the servers which are unique.
I defined my server document like that:


SERVER_RECORD = {
    "uuid":True,
    "serial_no":True,
    "product_name":False,
    "manufacturer":False,
    "location":False,
    "asset_tags":False 
}


Reading this, we just need a server UUID (which you can normally find under /sys/class/dmi/id/product_uuid , if this displays 00000 or something else then a uuid nowadays, you should stone your hardware manufacturer) and a serial number (/sys/class/dmi/id/product_serial).
These informations are needed to identify a server. Any other infos are not necessary (during the inventory job I try to get those infos, but actually they are just not that important).

But this record is not complete. Some server admins do need more informations like "how many CPU cores does the server has?" or "How much memory does the server has?" If you want to add this information you just add them to the inventory job (how you do it, is a topic for another article). But you just push the record with the needed fields and your added fields just to the same RPC call, and (DC)² will just save it to MongoDB.

And this is possible all over the system. I defined some informations which are needed for the standard work, which is really enough to help you deploy servers and help you with the bookkeeping, but you can add as much informations you need on top. Without changing one bit of backend code.

The Middleware

Well, (DC)² is mostly bookkeeping and configuration management and helping you to control your server fleet.
The deployment itself is done by FAI - Fully Automatic Installation. Which is an easy tool to deploy Debian Based Distros as well as RPM Based Distros like RHEL, CentOS, SLES, Fedora, Scientific Linux etc.

So, how does it interact with (DC)²?

As said before, the backend is speaking XMLRPC and JSON-RPC. The JSON-RPC part is for the frontend, the XMLRPC part is for the middleware and all tools needing data from (DC)².

The PXE Booting is also improved. Instead of using TFTP for loading the kernel and initrd I switched from the old pxelinux to the new gpxelinux (included in syslinux 4.02).
GPXElinux needs only tftp for loading the gpxelinux.0 file, all other files are being downloaded via HTTP protocol.
This gives you a wonderful possibility to cheaply scale your deployment infrastructure.


The Frontend

The frontend changed not so dramatically as the backend, but good things are still be found.
First of all, I put most of the code into separate modules. So, right now, there are modules for the models, which are used for JSON-RPC calls and pushing back the data to the widgets.
There is a module for all globally used widgets. You'll see that there is one widget which is mostly used. It's called "TableWidget" and has mostly all functionality in it.

But you put any widget you need into the tab view.

You see that the webfrontend is just looking like a desktop application. Which was indeed the purpose of using Qooxdoo and no "HTML add on framework" like  Dojo or YUI. I needed a real developers framework, and Qooxdoo is really one of the best. You can code like Nokias Qt and it's following the OOP paradigma most of the time.

And even for me, someone who had no clue about Javascript it was easy to learn and adapt.

To show you how easy it is, to add a new tab with a tablewidet, here is the javascript code of the Servers Tab.


Code Example:

_showInventoryServers:function(e) {            
if ("inventory_server" in this._tabPages) {
this._tabView.setSelection([this._tabPages["inventory_server"]]);
} else {
var server_tbl=new dc2.models.Servers(dc2.helpers.BrowserCheck.RPCUrl(false));
var server_search_dlg=new dc2.dialogs.search.Servers();
var server_edit_dialog=new dc2.dialogs.EditServer();
var server_table_options={
enableAddEntry:false,
enableEditEntry:true,
enableDeleteEntry:true,
enableReloadEntry:true,
editDialog:server_edit_dialog,
searchFunctions: {
searchDialog:server_search_dlg
},
tableModel:server_tbl,
columnVisibilityButton:false,
columnVisibility:[
{
column:0,
visible:false
}
]
};
var server_table=new dc2.widgets.TableWidget(server_table_options);
this._addTabPage("inventory_server",'Servers',server_table);
server_table.showData();
}
},

A closer look to the other code you can have on the DC² code browsing page on Launchpad. You'll find all the code on Launchpad.
The current frontend version of (DC)² is using Qooxdoo Version 1.5.


New Features

CS²

As you can see on the screenshots, there is a another menu entry with the name "CS²".




This (CS)²  means "Centralized SSL CA System" and helps you to manage your SSL host keys, CSRs, Certs and CRLs. Mostly used in the deployment system for Puppet or FreeIPA or whatever tools you are using which are in need of SSL Authorization.
(CS)² can be integrated in (DC)² but is also usable as standalone application. It has, equally to (DC)², a XMLRPC and JSON-RPC backend, has a qooxdoo frontend and is completly written in Python. Check out the screenshots.


RPM Based Distributions

Thanks to the work of the great Michael Goetze FAI is able to install RPM Based Distros like CentOS or Scientific Linux. I converted the CentOS Deployment to RHEL 5 and RHEL 6, so now, you are able to deploy mostly all world wide used  RPM based Distributions with FAI.
Thanks to Thomas Lange, who is the new maintainer of Rinse, who added my patch to it. 



What's still going to come?


I'm working on a Xen Management Center for (DC)², so you can provision Xen VMs (HVMs/PVs) in one tool without using any other tool.

This is a bit tricky, but it's coming along.
This module will also be available as integration into (DC)²  and as standalone application.
You will also have an XMLRPC and JSON-RPC Backend.
Eventually (this is not set) this RPC backend will also handle VMWare ESX server provisioning. We'll see.