Speeding up local debian builds with sbuild (eatmydata, apt-cacher-ng and config laziness)

Categories: English Geeky Uncategorized

As part of the team that maintains several testing tools for Ubuntu, including checkbox, I sometimes find myself needing to build .deb packages from our source tree.

'building stuff is hard'

A simple way of achieving this is of course to run dpkg-buildpackage or even bzr-buildpackage. Assuming all build-deps are correctly installed in the host system, this will result in a nicely built set of .debs.

This approach has a few caveats, in that it’s different from the build process actually employed to create the packages that ultimately get uploaded to Ubuntu (or even the ones available in Launchpad PPAs).

The two main differences are that Launchpad builds the packages in a “clean” environment, installing build-deps from scratch, whereas dpkg-buildpackage will rely on what’s installed in the system. So if you miss specifying a build-dep, your local build may work because you have it installed, but the PPA build will fail because it will not be present.

The second big difference is that with the local approach, you’re “limited” to building packages for the “host” system. Sure, you can specify a different target release in your debian/changelog file, but some aspect of your build may be tied to your system’s tools, versions and layout, and if for some reason they don’t match the actual target at installation time, things will fail in interesting ways.

Clearly, one way to test what the Launchpad build process will spit out is to build a source package and dput that to be built directly on a PPA. The problem here is that the feedback loop becomes excruciatingly slow; PPAs are a shared resource and build times can go from minutes to many hours.

Based on all this, it makes sense to try to use a local build environment that more closely replicates what PPAs do to build your packages.

Fortunately, the PPA builders use free software, so it’s relatively easy to do local builds in a similar environment, completing quickly due to use of local resources, and only upload to Launchpad once you’re pretty sure your build will succeed.

The software in question is sbuild, and I already wrote a post detailing how to install sbuild and set up a build environment for any Ubuntu release you need.

This setup worked fine for the occasional package build when you know packaging is mostly correct. For a fast build such as checkbox, setting up the build environment with all needed packages and build-deps takes about 10 minutes (depending mostly on download speed for all the packages). Of course on a more complex package, compilation time may start to be a factor.

Anyway, the 10-minute time can be too slow if you’re trying to fix a tricky problem and need a fast feedback loop. Plus the process produces a lot of transient files and downloads a set of packages many times, so there’s plenty of room for improvement here.

Speeding up local package installation and build

Eatmydata: it's so fast! (but not too safe)

A large part of the time spent doing the “local” part of the process is writing files to disk. One way to speed this up is to use a ramdisk to store the build. I’m too lazy and have too little RAM to use this approach, so the alternative was setting up eatmydata inside the chroot. Since these are mostly temporary files or throwaway packages, it’s OK to lose the safety of constant syncs in exchange for a huge boost in speed.

The setup for eatmydata inside the chroot is described here. This looks a bit hard to automate, but luckily we don’t have to, as recent versions of mk-sbuild simply support a –eatmydata parameter, if given this will install eatmydata inside the chroot and do the choot config file change to enable eatmydata.

Adding PPA

You can add a custom PPA to an image. Once the chroot image is built, enter the “golden master”:

sudo schroot -c source:saucy-amd64 -u root

You can add a deb line (get it from launchpad) to your sources:

cat >>/etc/apt/sources.list.d/something.list
 # Copy line here

Then you need to get the GPG key for the PPA and add it manually with the very basic tools provided in the chroot (sorry, no apt-add-repository):

apt-key add -
# Paste GPG armored key here

Then exit the golden image. After this, your builds from this chroot will be able to fetch packages from the PPA.

Again, that’s a bit of work to do for each VM. Instead, what I did was create a file in /etc/schroot/setup.d to do this automatically. You can of course replace the PPAs you need in the echo lines at the end. Name the file something like 81add-ppas:

#!/bin/sh
set -e
. "$SETUP_DATA_DIR/common-data"
. "$SETUP_DATA_DIR/common-functions"
. "$SETUP_DATA_DIR/common-config"
echo "$STAGE" >>/tmp/stages
 if [ $STAGE = "setup-start" ] || [ $STAGE = "setup-recover" ]; then
 echo "APT::Get { AllowUnauthenticated "1"; };" > $CHROOT_PATH/etc/apt/apt.conf.d/80unauthenticate
 info "ADDING PPAS"
 SLD_PATH="${CHROOT_PATH}/etc/apt/sources.list.d/roadmr.list"
 . $CHROOT_PATH/etc/lsb-release
 MY_RELEASE=$DISTRIB_CODENAME
 [ -n "$MY_RELEASE" ] || MY_RELEASE=trusty
 echo "# Added by the schroot setup mechanism (roadmr)" > $SLD_PATH
 echo "deb http://ppa.launchpad.net/checkbox-dev/ppa/ubuntu $MY_RELEASE main" > $SLD_PATH
 echo "deb http://ppa.launchpad.net/ubuntu-sdk-team/ppa/ubuntu $MY_RELEASE main" > $SLD_PATH
 fi

Notice that again, I was very lazy and instead of downloading the gpg keys as shown above (as for some reason trying to run gpg from the setup script didn’t work), I just configured apt to allow unauthenticated packages. Since this sbuild is mainly for testing purposes it’s not a big deal to skip this verification step. Also, there’s some logic to automatically detect the chroot release, so the same config file works equally well for any Ubuntu release.

Apt-cacher-ng

As the name suggests, this nifty utility will cache packages so the next time you need them they’ll be fetched from local storage rather than from the network. A bit of config is needed to have sbuild download packages from here.

Hello, I got these packages cached for you

First, install apt-cacher-ng on the host system. You can verify it’s listening on port 3142 by any means you like.

Then, to set it up automatically in chroots, add this to the host system’s  /etc/schroot/setup.d/80apt-cacher-ng (rather, create that file; it doesn’t exist by default):

#!/bin/sh
 set -e
 . "$SETUP_DATA_DIR/common-data"
 . "$SETUP_DATA_DIR/common-functions"
 . "$SETUP_DATA_DIR/common-config"
 if [ $STAGE = "setup-start" ] || [ $STAGE = "setup-recover" ]; then
 echo "# Added by the schroot setup mechanism (roadmr)" > "${CHROOT_PATH}/etc/apt/apt.conf.d/80proxy"
 echo "Acquire::http::Proxy \"http://127.0.0.1:3142\";" >> "${CHROOT_PATH}/etc/apt/apt.conf.d/80proxy"
fi

With these two setup.d scripts and the –eatmydata magic, it’s easy to create sbuild environments which will be much faster when building packages.

As a comparison, building msmtp (chosen because this tests mainly the speedup components, not needing any packages from a PPA) takes about 40 seconds with these suggested tweaks:

Build Architecture: amd64
 Build-Space: 5948
 Build-Time: 17
 Distribution: trusty
 Host Architecture: amd64
 Install-Time: 12
 Job: msmtp_1.4.31-1.dsc
 Machine Architecture: amd64
 Package: msmtp
 Package-Time: 40
 Source-Version: 1.4.31-1
 Space: 5948
 Status: successful
 Version: 1.4.31-1
 ─────────────────────────────────────────────────
 Finished at 20140320-1301
 Build needed 00:00:40, 5948k disc space

Whereas on a non-tweaked chroot it takes about 1:38 minutes:

Build Architecture: amd64
 Build-Space: 5568
 Build-Time: 17
 Distribution: trusty
 Host Architecture: amd64
 Install-Time: 31
 Job: msmtp_1.4.31-1.dsc
 Machine Architecture: amd64
 Package: msmtp
 Package-Time: 98
 Source-Version: 1.4.31-1
 Space: 5568
 Status: successful
 Version: 1.4.31-1
──────────────────────────────────────────────────Finished at 20140320-1310
Build needed 00:01:38, 5568k disc space

It looks like they’re about 3 times faster, but that’s misleading because I deliberately chose a small, quick-to-compile package. Still, you can at least reduce network and disk access very easily now. Note, also, that my test system has a fast SSD. Speedup on a traditional rotary magnetic hard-disk is likely to be much higher.