olasd's corner of the 'tubes

We need your help to make GSoC and Outreachy in Debian a success this summer!
Hi everyone,

A quick announcement: Debian has applied to the Google Summer of Code, and will also participate in Outreachy (formerly known as the Outreach Program for Women) for the Summer 2015 round! Those two mentoring programs are a great way for our project to bootstrap new ideas, give an new impulse to some old ones, and of course to welcome an outstanding team of motivated, curious, lively new people among us.

We need projects and mentors to sign up really soon (before February 27th, that’s next week), as our project list is what Google uses to evaluate our application to GSoC. Projects proposals should be described on our wiki page. We have three sections:
1. Coding projects with confirmed mentors are proposed to both GSoC and Outreachy applicants
2. Non-Coding projects with confirmed mentors are proposed only to Outreachy applicants
3. Project ideas without confirmed mentors will only happen if a mentor appears. They are kept on the wiki page until the application period starts, as we don’t want to give applicants false hopes of being picked for a project that won’t happen.
Once you’re done, or if you have any questions, drop us a line on our mailing-list (soc-coordination@lists.alioth.debian.org), or on #debian-soc on OFTC.

We also would LOVE to be able to welcome more Outreachy interns. So far, and thanks to our DPL, Debian has committed to fund one internship (US$6500). If we want more Outreachy interns, we need your help :).

If you, or your company, have some money to put towards an internship, please drop us a line at opw@debian.org and we’ll be in touch. Some of the successes of our Outreachy alumni include the localization of the Debian Installer to a new locale, improvements in the sources.debian.net service, documentation of the debbugs codebase, and a better integration of AppArmor profiles in Debian.

Thanks a lot for your help!
February 19, 2015
Debian proposals in GSoC 2014

The GSoC student application period is over, and the last two days were pretty interesting.

For a few years now, Olly Betts has provided us with a spreadsheet to graph the number of applicants to an organization over time.

Here’s the graph for Debian this year:

(Historical graphs: 2013, 2012. Spreadsheet available from Olly’s blog)

On Wednesday, I was thinking “hmm, 30 applicants, this is a slow year…”. Well, the number of proposals more than doubled in the last two days, to conclude on a whooping 68 applications! The last one was submitted just three seconds before the deadline…

If you want to take a look at the proposals, head over to the Debian wiki.

Time to get on reviewing! The final student acceptances will be published in just less than a month, on April 21st.

March 22, 2014
Hello from DebCamp
Small update as someone was complaining about the lack of pictures from DebCamp on planet.

Not to worry, everything is going fine, and some of the most important stuff is ready…

You can see a few pictures from the gallery. The view from the venue is quite outstanding (it was better when there was some sun on Tuesday, but my camera battery was out…).

On my TODO-list:
- A lot of mentors.d.n related stuff : prototyping things to make our codebase more bearable by actually using other people’s work, e.g. debianqueued, firehose, …, and building upon Lucas’s idea from his new contributor survey (see the quite vague idea at the bottom)
- FedMsg stuff (will probably pick up when Simon arrives)
- Some 3D-printing packaging stuff (actually getting software in NEW, I got started on Printrun, and maybe Slic3r)
See you there!
August 8, 2013
Bootstrapping fedmsg for Debian
As you might (or might not) know, this summer, I have taken on mentoring of a GSoC project by Simon Chopin (a.k.a. laarmen) which goal is to bring fedmsg, the Fedora Infrastructure message bus, to Debian. Most of the work I’ll be talking about here is Simon’s work, please send all the praise towards him (I can take the blame, though).

What is this about?

As the project proposal states, the idea is to provide Debian with a unified, real-time, and open mechanism of communication between its services. This communication bus would allow anyone, anywhere, to start consuming messages and reacting to events happening in Debian’s infrastructure:
- trigger a test build on a git push to a source repository
- trigger automated testing (piuparts, lintian) as soon as an upload hits the archive
- get a desktop notification when a package you care about gets changed
- …
When we told upstream about our plan of adapting fedmsg to work on Debian, they were thrilled. And they have been very supportive of the project.

How is the project going?

Are you excited? I know I’m excited.

Well, the general idea was easy enough, but the task at hand is a challenge. First of all, fedmsg has a lot of (smallish) dependencies, most of them new to Debian.

Thanks to Simon’s work during the bonding period, and thanks to paultag’s careful reviews, the first batch of packages (the first dependency level, comprising kitchen, bunch, m2ext, grapefruit, txws, txzmq and stomper) is currently sitting in the NEW queue. The four remaining packages (fabulous, moksha.common, moksha.hub and fedmsg proper) are mostly ready, waiting in the Debian Python Module Team SVN repository for a review and sponsorship.

While we’re waiting for the packages to trickle into Debian, Simon is not twiddling his thumbs. Work has taken place on a few fronts:
- Getting fedmsg integrated to mentors.debian.net (actually sending the first messages from “Debian”‘s infrastructure)
- Writing a (stop-gap) wrapper to convert some of our mail “broadcasts” (debian-devel-changes, debian-bugs-dist) into fedmsg messages, until a more proper integration can be done
- Working on reliability of the message transport, following some concerns raised by DSA.
fedmsging mentors.debian.net

Package backports

mentors.debian.net was chosen because I’m an admin and could do the integration quickly. That involved backporting the eleven aforementioned packages, plus zeromq3 and python-zmq (that only have TCP_KEEPALIVE on recent versions), to wheezy, as that’s what the mentors.d.n host is running. (Also, python-zmq needs a new-ish cython to build so I had to backport that too). Thankfully, those were no-changes backports, that were easily scripted, using a pbuilder hook to allow the packages to depend on previously built packages.

I have made a wheezy package repository available here. It’s signed with my GnuPG key, ID 0xB8E5087766475AAF, which should be fairly well connected.

Code changes

After Simon’s initial setup of debexpo (which is not an easy task), the code changes have been fairly simple (yes, this is just a proof of concept). You can see them on top of the live branch on debexpo’s sources. I finally had the time to make them live earlier this week, and mentors.debian.net has been sending messages on Debian’s fedmsg bus ever since.

Deployment

mentors.d.n sends its messages on five endpoints, tcp://mentors.debian.net:3000 through tcp://mentors.debian.net:3004. That is one endpoint per WSGI worker, plus one for the importer process(es). You can tap in directly, by following the instructions below.

debmessenger

Debmessenger is the stop-gap email-to-fedmsg bridge that Simon is developing. The goal is to create some activity on the bus without disrupting or modifying any infrastructure service. It’s written in hy, and it leverages the existing Debian-related python modules to do its work, using inotify to react when a mail gets dropped in a Maildir.

Right now, it’s supposed to understand changes mails (received from debian-devel-changes) and bugs mail (from debian-bugs-dist).

I’ll work on deploying an instance of debmessenger this weekend, to create some more traffic on the bus.

Reliability of the bus

I suggested using fedmsg as this was something that already existed, and that solved a problem identical to the one we wanted to tackle (open interconnection of a distribution’s infrastructure services). Reusing a piece of infrastructure that already works in another distro means that we can share tools, share ideas, and come up with solutions that we might not have considered when working alone. The drawback is that we have to either adapt to the tool’s idiosyncrasies, or to adapt the tool to our way of working.

One of the main points raised by DSA when the idea of using fedmsg was brought up, was that of reliability. Debian’s infrastructure is spread in datacenters (and basements :D) all over the world, and thus faces different challenges than Fedora’s infrastructure, which is more tightly integrated. Therefore, we have to ensure that a critical consumer (say, a buildd) doesn’t miss any message it would need for its operation (say, that a package got accepted).

There has been work upstream, to ensure that fedmsg doesn’t lose messages, but we need to take extra steps to make sure that a given consumer can replay the messages it has missed, should the need arise. Simon has started a discussion on the upstream mailing list, and is working on a prototype replay mechanism. Obviously, we need to test scenarios of endpoints dropping off the grid, hence the work on getting some activity on the bus.

How can I take a look?

a.k.a. “Another one rides the bus”

(Picture © Yves-Laurent Allaert, CC-By-SA v2.5 / GFDL v1.2 license)

So, the bus is pretty quiet right now, as only two kinds of events are triggering messages: a new upload to mentors.debian.net, and a new comment on a package there. Don’t expect a lot of traffic. However, generating some traffic is easy enough: just login to mentors.d.n, pick a package of mine (not much choice there), or a real package you want to review, and leave a comment. poof, a message appears.

For the lazy

Join #debian-fedmsg on OFTC, and look for messages from the debmsg bot.

Current example output:
```
01:30:25 <debmsg> debexpo.voms-api-java.upload (unsigned) --
02:03:16 <debmsg> debexpo.ocamlbricks.comment (unsigned) --
```
(definitely needs some work, but it’s a start)

Listening in by yourself

You need to setup fedmsg. I have a repository of wheezy packages and one of sid packages, signed with my GnuPG key, ID 0xB8E5087766475AAF. You can add them to a file in /etc/apt/sources.list.d like this:
```
deb http://perso.crans.org/dandrimont/fedmsg-<sid|wheezy>/ ./
```
Then, import my GnuPG key into apt (apt-key add), update your sources (apt-get update), and install fedmsg (apt-get install python-fedmsg). The versions are << to anything real, so you should get the real thing as soon as it hits the archive.

Finally, in /etc/fedmsg.d/endpoints.py, you can comment-out the Fedora entries, and add a Debian entry like this:
```
    "debian": [
        "tcp://fedmsg.olasd.eu:9940",
    ],
```
fedmsg.olasd.eu runs a fedmsg gateway connected to the mentors.d.n endpoints, and thus forwards all the mentors messages. It’ll be connected to debmessenger as soon as it’s running too.

To actually see mesages, disable validate_signatures in /etc/fedmsg.d/ssl.py, setting it to False. The Debian messages aren’t signed yet (it’s on the roadmap), and we don’t ship the Fedora certificates so we can’t authenticate their messages either.

Finally, you can run fedmsg-tail --really-pretty in a terminal. As soon as there’s some activity, you should get that kind of output (color omitted):
```
{
  "i": 1, 
  "msg": {
    "version": "2.0.9-1.1", 
    "uploader": "Emmanuel Bourg <ebourg@apache.org>"
  }, 
  "topic": "org.debian.dev.debexpo.voms-api-java.upload", 
  "username": "expo", 
  "timestamp": 1373758221.491809
}
```
Enjoy real-time updates from your favorite piece of infrastructure!

What’s next?

While Simon continues working on reliability, and gets started on message signing according to his schedule, I’ll take a look at deploying the debmessenger bridge, and making the pretty-printer outputs useful for our topics. There will likely be some changes to the messages sent by debexpo, as we got some feedback from the upstream developers about making them work in the fedmsg “tool ecosystem” (datanommer and datagrepper come to mind).

You can tune in to Simon’s weekly reports on the soc-coordination list, and look at the discussions with upstream on the fedora messaging-sig list. You can also catch us on IRC, #debian-soc on OFTC. We’re also hanging out on the upstream channel, #fedora-apps on freenode.
July 14, 2013
Hello world

Or rather, hello Planet!

Here’s a somewhat traditional introductory post.

I’m Nicolas Dandrimont, I’m French, I’m sysadmin in a grande école, where I’m mostly in charge of the GNU/Linux workstations and servers.

In Debian, I’m a DM, currently in the NM queue, so I might become a DD soon-ish. I am (rather inactively) co-maintaining a few packages. In my Debian “career”, I have been involved in OCaml packaging and Python packaging, although lately most of my time has been spent on Google Summer of Code (mentor for two mentors.debian.net projects in 2012, org admin for Debian in 2013), and on mentors.debian.net.

In other free-software related projects, I own a RepRap 3D printer, and I grew some interest in the related software, e.g. Slic3r and printrun. There have been a lot of action in Fedora about packaging 3D-printing-related software, and it’d be great to get a team together to work on that in Debian during the jessie release cycle. Consider this a call for interested parties 🙂

Unrelatedly, paultag has tricked me into working on hy, which is way too much fun. Blame him if you feel that I have been inactive lately, this has been eating way too much of my free time 😉

Hopefully I’ll be able to make regular updates on the work I do in Debian and free software, so stay tuned!

May 19, 2013