Fedora Activity Day: Release Engineering 2015

This past weekend a number of members of the Fedora Community who spend time focused on Release Engineering came together for a Fedora Activity Day (FAD) to work on some items of interest that will fix current issues being faced as well as work towards solutions for how to handle future challenges such as how to deliver the Fedora.Next Rings concept in a clean, well defined, standard reproduce-able manner via build systems, tooling, and processes. One of the big items that I'm personally very excited about is Fedora Atomic (which is Fedora's implementation of Project Atomic).

A big ticket item for Fedora Release Engineering right now is Pungi Version 4 which is a complete rewrite from Pungi Version 3.x and it will enable a large array of new functionality for composes that would have been quite difficult to implement in the older version. Of those items are things such as Koji integration, enabling nightly Rawhide composes to match a normal Test Candidate or Release build, and enabling all outlets of Fedora Atomic including ISO installer  and pxe-to-live nightly builds. Since this is will enable more rapid iteration on the Fedora Atomic composes and I'm very enthusiastic about that project, this is where I spent a large amount of my time over the weekend.

A quick side note, I was fortunate enough to attend this event *and* I was also fortunate enough to attend a session about Fedora Hubs the day before the Activity Day which was amazing and I can't wait for the project to come to fruition, but I won't go too far into that because this particular post is about the FAD.

Friday 2015-06-05
We kicked off the Release Engineering FAD, the first few hours are to go back through proposed deliverables to bring everyone up to speed on all the issues, their background, the motivations to resolve, and so that we can discuss and debate what items will provide the most "bang for buck" outcome in terms of what we work on while we're fortunate enough to all be sitting in a room together over the weekend. We took a scrum-style approach, lead by Ian McLeod, to scoring work tasks for how long we thought the work would take over all. Once this was done, we broke out into subgroups to divide and conquer tasks. I joined Jon Disnard in a break out session to work on Pungi4 which is something that will enable a lot of other items in the deliverables list for the FAD. As mentioned before, Pungi version 4.x is a complete rewrite from Pungi 3.x and beyond what was previously mentioned it has taken compose times from 8+ hours down to roughly an hour in most cases. The work on Pungi4 was unfortunately quite a bit more than we had originally thought, there were APIs that it depended upon that were changed out from under it, there were a number of namespace bugs, and there was a considerable amount of code that we were able to remove once we realized that the data that code was parsing and producing was already held in productmd which is a new dependency of Pungi in version 4 over version 3. This work continued late into the evening and on into the next day.

Saturday 2015-06-06
All Fedora contributors met up and did a short status check-in based on progress made and/or work completed from the previous day so that we could sort out what to work on next, re-prioritize if necessary based on if things had become blocked or not. Once this was complete, everyone broke out into groups again to continue getting work done. On this day I worked with a few more people than before including Dan Mach (the original author of the recently rewritten Pungi version 4) which as a massive help in terms of institutional knowledge. The end result from this group of collaboration was a functional pungi4 for the workflows we were testing at the time (I leave this open ended mostly because there are a *lot* of possible workflows in Pungi4 and we've not been able to test them all yet). As a result, we filed a pull request with upstream Pungi and continued to work on other items. However, one thing stuck out here is that iterating on Pungi was unnecessarily painful because you had to have specific access to sets of RPMs in order to perform compose workflows. This was very time consuming and did not bode well for rapid development, we started a discussion of how to improve this but that bled into the evening. We also ran into an interesting bug where createrepo_c will fail but report success, this is still something we're attempting to track down but reverted to createrepo in the interest of time. Interestingly enough we've run into a new bug here where the createrepo pathing is always prepended with a bindmounted path, but it didn't stop us and it's on the list of items to be resolved.

Sunday 2015-06-07
Once again, we all met and assessed the current status of ongoing work. The good news is that a lot of work had been completed at this point but the bad news is that some items ended up blocked and we had to resolve those blockers before we could move on so some of the estimates we made on Friday were off. However, we documented all of this so there was a solid plan of action to move forward with. At this point we broke out into groups again and I joined in with the group that started to discuss next-gen tooling. We diverged from writing code for a while so that we could make sure not to run out of time before discussing these next generation tools as per the agenda. Koji2 was discussed at length and some initial design proposals should be coming down the pipeline this week on the koji-devel mailing list (the core developers committed to getting that done soon). We started defining requirements for ComposeDB (project name pending, we don't feel this is really the best naming), a mailing list thread with initial discussion results and a write up of the end result will follow. A few of us also re-converged on the topic of slow iteration for Pungi4 development and the end result was the ability to rapidly run tests out of a git clone/checkout. With this pull request, developers can run tests in roughly 20 seconds (after the inital setup to create the "dummy data", which takes a minute or two) which is a massive improvement over the hour-long test runs we were having against the real Fedora rpm sets.

This has been mostly an account of things I was directly involved in, there was a lot of work that got done over the course of the FAD, I couldn't even remotely pretend to be able to keep track of it all. That being said, many folks did a wonderful job of keeping tabs on their specific areas of work as well as the results of the daily check-ins, this was captured in an Etherpad here. Different members of the FAD volunteered to summarize and sent out status updates, summaries, and plans of action for continued work to appropriate outlets for the different areas they worked on. All in all I think it was a successful FAD, though I lack some perspective as this was the first FAD I've ever been to. Others who have participated in FADs before had positive things to say so I feel good about it.

Next, I plan to follow up and make sure to get Fedora Atomic running in Pungi4 as soon as we possibly can so that the nightly builds allow the project to iterate more rapidly. From there I hope to work on both ComposeDB and Koji 2.0.

Here's to continuing to get work done in Fedora land!

Happy hacking,
-AdamM