CONFINE project

It's been more than half a year since my last post on CONFINE, which may give you an idea of the feverish months we've had at the project. We still are in that rush, but the Christmas days bring a little calm so I won't miss the chance of writing one last post for this year.

After upgrading the VCT container to Wheezy, the turn came to the Debian sliver template. Besides the upgrade, I decided to make templates more useful by including enough configuration (to be discarded during sliver deployment in a node) as for being directly usable as containers for testing in one's PC. Since the template comes in a read-only Squashfs I included instructions on how to unpack the template into the local file system, but I also had a good time using LXC mount hooks to place a writable directory on top of the read-only template using AuFS or overlayfs. I also tried a more sophisticated approach which has the writable directory loop-mounted from a fixed-size image file, a simple and neat way to limit the disk space used by a container. Serge Hallyn liked the idea a lot, but unfortunately on container stop the image doesn't get properly unmounted and loop devices remain undetachable.

Also, while testing on Ubuntu hosts I found that the read-only proc causes the error "lxc-start: Read-only file system - failed to change apparmor profile to lxc-container-default", so I made it writable since it is now supposed to be properly isolated from the host's proc. I also found that keeping the sys_boot capability makes possible to halt and reboot the container properly (regardless of Debian bug 706676). :)

I also added these latter fixes to the VCT container, but the really important change is how CONFINE software is now installed in it. Previously, the container only included a clone of the confine-dist repository (which includes VCT), a bare installation of the at-the-time-latest confine-controller, and their dependencies. The software wasn't configured at all. This saved some download time for some software, but VCT installation and configuration always had to be run, which resulted in software being downloaded and replaced and more data files (node images and sliver templates) being downloaded. It also implied that container preparation had to somehow replicate VCT and controller installation.

To avoid these problems and provide a container where VCT can be run out of the box with no installation or downloads whatsoever, I changed the way the container is prepared to simply include an invocation of vct_system_install. This simplified its preparation, installation and usage enormously, of course at the price of having a bigger container image that includes all downloaded files. However users will be glad to know that they only need to run vct_system_init to have a working VCT environment. :)

While writing a mail to help some colleague in CONFINE connect his research device to Community-Lab's management network, I find one bug in our software. While reporting the bug, I find two more bugs. While reporting one of the latter bugs, I find a bug in Redmine. This starts to look like a software development version of Inception.

With the recent launch of the new Debian 7.0 “Wheezy”, some users that were testing the CONFINE controller found some incompatibilities between Wheezy and the previous Debian “Squeeze” regarding task management. I decided to upgrade the VCT container to Wheezy to ease the testing of these issues and Marc managed to fix or work around them. As result, he published new versions of the controller and I packed a new VCT container based on Wheezy with one of these versions. We also found some issues with the handling of new hosts in tinc that Guus helped clarify. With all this testing, node software and controller software are quickly getting really usable and stable!

Also, Pau asked me to find a way to provide Internet access (at least NAT) to Community-Lab slivers running on community networks which use private IPv4 addressing. Since tampering with community routes is not an option, we decided to follow the VPN path. I'm working on leveraging the presence of the tinc mesh already used for the management network to also provide VPN access to testbed gateways connected to the Internet. Not an easy redesign so late in the project, but I have some proposals that make everything (management network, VPN, tinc) fit quite well.

Lately I've continued with the testing that I began for the latest Battle Mesh to check that the Community-Lab testbed and CONFINE software in general are actually usable for the participants of the first Open Call. I've sent even more bug reports, but this time Axel (who fortunately restrained himself from chasing and hitting me) has had time to fix some of them so we've been in a tight loop of test-report-fix-test.

The good news is that I find the testbed in its current state to be quite usable, at least for a trusted set of researchers with close assistance from testbed developers and administrators. I even found VCT to be working (as a container!) for running test experiments, albeit some bugs which make it not work out of the box. Another important factor in usability is having good documentation but I'm afraid we're still green on that, although Davide is working on updating the BitTorrent tutorial, and documentation and support is one of this year's objectives.

Friday was my last day in this year's Wireless Battle of the Mesh. After Pau's insistence, I finally decided to attend this event for the first time, and I must say that I liked it a lot. For those of you who don't know it, it's some kind of mesh and community network-oriented event in the likes of FOSDEM, but on a much more familiar scale: we all fitted in a single room at the NOVI building in the Aalborg University, and most people already knew each other.

Tables full of wires in a wireless event.

I finished restructuring the testbed architecture start page to make each main topic have its own page which fits into a narrative that can be read from start to end. This should help newcomers understand how CONFINE testbeds work while gently introducing the most relevant concepts. The page that got most attention was that about the management network, which was extended to include a good introduction about its fundamental role and nature, the need for the IPv6 overlay and its tinc-based implementation. The high-level introduction of the old IPv6 overlay page was merged into that page while the low-level details where moved to the software pages.

I was also working on my demo for next week's Battle Mesh using a VCT container, but some issues make me suspect that it'll be less troublesome to test real nodes from Community-Lab where at least the server part is already set up and working. I also intend to take the chance of visiting the Battle Mesh to discuss some wild ideas that Dani, Leandro and me were informally discussing today about using Linux's kexec in the initial node system… well, more on that in another post. :)

Since I didn't know where to publish my script for generating configuration files for all tinc hosts in a testbed's management network, and Marc pointed out that people used to ask for repositories for very small projects, we decided to create the confine-utils project and repository for hosting assorted utilities related with CONFINE testbeds. We reused the repo of Pau's firmware generator conFW (now in the confw subdir) and I also uploaded my script there (under fetch-tinc-hosts). I wanted the script to use no external dependencies but it turned out that urllib2 isn't well suited for REST API programming, so I used the excellent requests library. Now you can generate all the tinc host configuration files needed by e.g. a gateway by running python fetch_tinc_hosts.py REST_API_BASE_URI.

Javi was unable to run the prebuilt VCT container template on his old Core Duo box since the container is 64-bit and those CPUs (though supporting hardware virtualization) are 32-bit only. He also had a hard time trying to build his own container from scratch following the instructions in the wiki because of the old version of the lxc package included in his Ubuntu install.

Manos is working on his master thesis on software-defined networking (SDN) and he's also participating in CONFINE. Together with Leandro, we had some discussions on the potential changes to CONFINE's testbed architecture to support SDN with OpenFlow, Open vSwitch and related technologies so that slices could e.g. define virtual L2 links between nodes that have no such physical links.

During the last days I've been working on defining the node storage layout at a finer grain, including the specific paths to be used in filesystems and how to manage them during the boot process and operation of the node, which turned to be challenging because we're talking about a system which will probably perform two root pivots while booting.

I also specified the behaviour of the programs to manage both the node images stored in it and the persistent data sets to be shared across node images so that the gory details of mounts, paths, symbolic links, etc. are isolated from the user. I have the impression that these could be separated in two independent packages for reusing outside of CONFINE. I wouldn't be surprised if some people at the Battle Mesh showed some interest in these developments. ;)

Well, here's my first post in a series that will report the progress of my current work at the CONFINE project.

Axel and Marc recently managed to integrate the CONFINE controller software into VCT. However, the installation of the controller is more invasive that that of VCT itself (change of default locale, installation and activation of system services…), so using a VCT container instead of a host installation becomes even more handy. Marc and me managed to fix some hidden issues in the controller that prevented it to be installed in a clean VCT container, and I extended the VCT container creation procedure to include the CONFINE controller and dependent software so that few packages need to be downloaded later when using different versions of the controller. The new VCT container image is available in the CONFINE images repository.