As you might (or might not) know, this summer, I have taken on mentoring of a GSoC project by Simon Chopin (a.k.a. laarmen) which goal is to bring fedmsg, the Fedora Infrastructure message bus, to Debian. Most of the work I’ll be talking about here is Simon’s work, please send all the praise towards him (I can take the blame, though).

What is this about?

As the project proposal states, the idea is to provide Debian with a unified, real-time, and open mechanism of communication between its services. This communication bus would allow anyone, anywhere, to start consuming messages and reacting to events happening in Debian’s infrastructure:

• trigger a test build on a git push to a source repository
• trigger automated testing (piuparts, lintian) as soon as an upload hits the archive
• get a desktop notification when a package you care about gets changed
• …

When we told upstream about our plan of adapting fedmsg to work on Debian, they were thrilled. And they have been very supportive of the project.

How is the project going?

Are you excited? I know I’m excited.

Well, the general idea was easy enough, but the task at hand is a challenge. First of all, fedmsg has a lot of (smallish) dependencies, most of them new to Debian.

Thanks to Simon’s work during the bonding period, and thanks to paultag’s careful reviews, the first batch of packages (the first dependency level, comprising kitchen, bunch, m2ext, grapefruit, txws, txzmq and stomper) is currently sitting in the NEW queue. The four remaining packages (fabulous, moksha.common, moksha.hub and fedmsg proper) are mostly ready, waiting in the Debian Python Module Team SVN repository for a review and sponsorship.

While we’re waiting for the packages to trickle into Debian, Simon is not twiddling his thumbs. Work has taken place on a few fronts:

• Getting fedmsg integrated to mentors.debian.net (actually sending the first messages from “Debian”‘s infrastructure)
• Writing a (stop-gap) wrapper to convert some of our mail “broadcasts” (debian-devel-changes, debian-bugs-dist) into fedmsg messages, until a more proper integration can be done
• Working on reliability of the message transport, following some concerns raised by DSA.

fedmsging mentors.debian.net

Package backports

mentors.debian.net was chosen because I’m an admin and could do the integration quickly. That involved backporting the eleven aforementioned packages, plus zeromq3 and python-zmq (that only have TCP_KEEPALIVE on recent versions), to wheezy, as that’s what the mentors.d.n host is running. (Also, python-zmq needs a new-ish cython to build so I had to backport that too). Thankfully, those were no-changes backports, that were easily scripted, using a pbuilder hook to allow the packages to depend on previously built packages.

I have made a wheezy package repository available here. It’s signed with my GnuPG key, ID 0xB8E5087766475AAF, which should be fairly well connected.

Code changes

After Simon’s initial setup of debexpo (which is not an easy task), the code changes have been fairly simple (yes, this is just a proof of concept). You can see them on top of the live branch on debexpo’s sources. I finally had the time to make them live earlier this week, and mentors.debian.net has been sending messages on Debian’s fedmsg bus ever since.

Deployment

mentors.d.n sends its messages on five endpoints, tcp://mentors.debian.net:3000 through tcp://mentors.debian.net:3004. That is one endpoint per WSGI worker, plus one for the importer process(es). You can tap in directly, by following the instructions below.

debmessenger

Debmessenger is the stop-gap email-to-fedmsg bridge that Simon is developing. The goal is to create some activity on the bus without disrupting or modifying any infrastructure service. It’s written in hy, and it leverages the existing Debian-related python modules to do its work, using inotify to react when a mail gets dropped in a Maildir.

Right now, it’s supposed to understand changes mails (received from debian-devel-changes) and bugs mail (from debian-bugs-dist).

I’ll work on deploying an instance of debmessenger this weekend, to create some more traffic on the bus.

Reliability of the bus

I suggested using fedmsg as this was something that already existed, and that solved a problem identical to the one we wanted to tackle (open interconnection of a distribution’s infrastructure services). Reusing a piece of infrastructure that already works in another distro means that we can share tools, share ideas, and come up with solutions that we might not have considered when working alone. The drawback is that we have to either adapt to the tool’s idiosyncrasies, or to adapt the tool to our way of working.

One of the main points raised by DSA when the idea of using fedmsg was brought up, was that of reliability. Debian’s infrastructure is spread in datacenters (and basements :D) all over the world, and thus faces different challenges than Fedora’s infrastructure, which is more tightly integrated. Therefore, we have to ensure that a critical consumer (say, a buildd) doesn’t miss any message it would need for its operation (say, that a package got accepted).

There has been work upstream, to ensure that fedmsg doesn’t lose messages, but we need to take extra steps to make sure that a given consumer can replay the messages it has missed, should the need arise. Simon has started a discussion on the upstream mailing list, and is working on a prototype replay mechanism. Obviously, we need to test scenarios of endpoints dropping off the grid, hence the work on getting some activity on the bus.

How can I take a look?

a.k.a. “Another one rides the bus”

(Picture © Yves-Laurent Allaert, CC-By-SA v2.5 / GFDL v1.2 license)

So, the bus is pretty quiet right now, as only two kinds of events are triggering messages: a new upload to mentors.debian.net, and a new comment on a package there. Don’t expect a lot of traffic. However, generating some traffic is easy enough: just login to mentors.d.n, pick a package of mine (not much choice there), or a real package you want to review, and leave a comment. poof, a message appears.

For the lazy

Join #debian-fedmsg on OFTC, and look for messages from the debmsg bot.

Current example output:

01:30:25 <debmsg> debexpo.voms-api-java.upload (unsigned) --
02:03:16 <debmsg> debexpo.ocamlbricks.comment (unsigned) --

(definitely needs some work, but it’s a start)

Listening in by yourself

You need to setup fedmsg. I have a repository of wheezy packages and one of sid packages, signed with my GnuPG key, ID 0xB8E5087766475AAF. You can add them to a file in /etc/apt/sources.list.d like this:

deb http://perso.crans.org/dandrimont/fedmsg-<sid|wheezy>/ ./

Then, import my GnuPG key into apt (apt-key add), update your sources (apt-get update), and install fedmsg (apt-get install python-fedmsg). The versions are << to anything real, so you should get the real thing as soon as it hits the archive.

Finally, in /etc/fedmsg.d/endpoints.py, you can comment-out the Fedora entries, and add a Debian entry like this:

    "debian": [
        "tcp://fedmsg.olasd.eu:9940",
    ],

fedmsg.olasd.eu runs a fedmsg gateway connected to the mentors.d.n endpoints, and thus forwards all the mentors messages. It’ll be connected to debmessenger as soon as it’s running too.

To actually see mesages, disable validate_signatures in /etc/fedmsg.d/ssl.py, setting it to False. The Debian messages aren’t signed yet (it’s on the roadmap), and we don’t ship the Fedora certificates so we can’t authenticate their messages either.

Finally, you can run fedmsg-tail --really-pretty in a terminal. As soon as there’s some activity, you should get that kind of output (color omitted):

{
  "i": 1, 
  "msg": {
    "version": "2.0.9-1.1", 
    "uploader": "Emmanuel Bourg <ebourg@apache.org>"
  }, 
  "topic": "org.debian.dev.debexpo.voms-api-java.upload", 
  "username": "expo", 
  "timestamp": 1373758221.491809
}

Enjoy real-time updates from your favorite piece of infrastructure!

What’s next?

While Simon continues working on reliability, and gets started on message signing according to his schedule, I’ll take a look at deploying the debmessenger bridge, and making the pretty-printer outputs useful for our topics. There will likely be some changes to the messages sent by debexpo, as we got some feedback from the upstream developers about making them work in the fedmsg “tool ecosystem” (datanommer and datagrepper come to mind).

You can tune in to Simon’s weekly reports on the soc-coordination list, and look at the discussions with upstream on the fedora messaging-sig list. You can also catch us on IRC, #debian-soc on OFTC. We’re also hanging out on the upstream channel, #fedora-apps on freenode.