Automated AUR repository

07 May 2018

For a while now I’ve entertained the idea of an automated setup to build packages from the Arch User Repository (AUR), keep them updated and serve them to my machines via a regular repository for pacman (Arch Linux’ package manager). I want to build in a chroot, as this is an easy way to find missing dependencies. I also want the whole setup to live inside a container, in case I want to migrate it to another machine and for containment, as this repo will eventually be publically available.

I’ve settled on Linux Containers (LXC) for the container solution as it’s widely available, very mature and has a low overhead. The current host is my regular work machine, running Arch, but later I want to move it to the HTPC/server/homelab, which will eventually run Proxmox Virtualisation Environment (PVE).

Setup

  1. Install required software on the host 
    # pacman -S lxc arch-install-scripts
  2. Configure LXC 
    /etc/lxc/default.conf
---------------------
lxc.net.0.type = empty
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
  3. Create a privileged LXC: 
    sudo lxc-create -n repo -t download -- --dist archlinux --release current --arch amd6

    This is necessary because makechrootpkg creates a fully populated devfs in the nspawn container it creates. Once that is no longer the case I may be able to switch to an unprivileged container.

  4. Configure network
    I want the container to be accessible from the local network and potentially the internet, so a proper bridge is the easiest solution. Since I need the container’s IP to be static I set it in its LXC config and only configure DNS inside with openresolv.
    Host: 
    /etc/systemd/network/bridge.netdev
----------------------------------
[NetDev]
Name=br0
Kind=bridge

    /etc/systemd/network/bridge.network
-----------------------------------
[Match]
Name=br0

[Network]
DHCP=ipv4
DNS=192.168.1.1
Domains=mydomain.tld

[DHCP]
UseDomains=true

    /etc/systemd/network/wired.network
----------------------------------
[Match]
Name=enp3s0

[Network]
Bridge=br0

    /var/lib/lxc/repo/config
------------------------
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.ipv4.address = 192.168.1.50/24
lxc.net.0.ipv4.gateway = 192.168.1.1

    This is not the complete container configuration, but the rest is unchanged.

    Container:

    /etc/resolvconf.conf
--------------------
name_servers=192.168.1.1
search_domains=my_domain.tld

    With this setup the container reliably gets an address and can resolve names.

  5. Install required packages in the container 
    # pacman -Syu --needed base-devel devtools darkhttpd btrfs-progs sudo

    btrfs-progs is only required because the container lives in a btrfs partition and mkarchroot will create a subvolume in this case.

Configuration

With everything set up I can install AUR/aurutils and configure the repo and automation.

  1. Create a user to handle the repo and let it the necessary commands as root 
    # useradd -m repo

    /etc/sudoers.d/10-aurutils
--------------------------
repo repo=(root) NOPASSWD: SETENV: /usr/bin/makechrootpkg *
repo repo=(root) NOPASSWD: /usr/bin/arch-nspawn *
  2. Create a directory for the chroot and create it 
    # mkdir -p /var/lib/aurbuild/x86_64
# mkarchroot -c /var/cache/pacman/ -C /etc/pacman.conf /var/lib/aurbuild/x86_64/root base-devel
  3. Create a directory for the repository and create the DB 
    # mkdir /var/cache/pacman/myrepo
# chown repo:repo /var/cache/pacman/myrepo

    And as the repo user:

    $ repo-add /var/cache/pacman/myrepo/myrepo.db.tar.gz
  4. Configure pacman to use the repository At the very end of /etc/pacman.conf add: 
    /etc/pacman.conf
----------------
Include = /etc/pacman.d/myrepo

    /etc/pacman.d/myrepo
--------------------
[options]
CacheDir = /var/cache/pacman/pkg
CacheDir = /var/cache/pacman/myrepo
CleanMethod = KeepCurrent

[myrepo]
SigLevel = Optional TrustAll
Server = file:///var/cache/pacman/myrepo

    I will modify this file once I start signing my packages.

  5. Set up darkhttp to serve the repo Run 
    # systemctl edit darkhttpd

    and create a file containing these lines:

    [Service]
ExecStart=
ExecStart=/usr/bin/darkhttpd /var/cache/pacman/myrepo --uid http --gid http --chroot --no-listing --mimetypes /etc/conf.d/mimetypes

    This will be saved to /etc/systemd/system/darkhttpd.service.d/override.conf
    Update the service, start and enable it:

    # systemctl daemon-reload
# systemctl enable --now darkhttpd

    If you want to be able to download packages with the browser, remove the --no-listing option from the ExecStart line.

  6. Create automation scripts This can be done entirely as the repo user. Since I don’t want to have to watch the whole build process via ssh I want to have a file that I write package names into and a script that passes them on to aursync. 
    $ mkdir -p .local/bin

    /home/repo/.local/bin/aurbuilder
--------------------------------
#!/bin/bash

watchfile=$WATCHFILE

if [ "$#" -eq 1 ]; then
    watchfile=$1
fi

packages=$(<$watchfile)
echo '' > $watchfile
aursync -cT --no-view $packages

    $ chmod +x .local/bin/aurbuilder

    The file /home/repo/.aurbuilder is watched by systemd, which triggers a service calling this script, if the file is closed after writing.

    $ mkdir -p .config/systemd/user

    /home/repo/.config/systemd/user/aurbuilder.path
-----------------------------------------------
[Unit]
Description=Watch aurbuilder input file

[Path]
PathChanged=/home/repo/.aurbuilder

[Install]
WantedBy=default.target

    /home/repo/.config/systemd/user/aurbuilder.service
--------------------------------------------------
[Unit]
Description=Pass package names from a file to aursync

[Service]
Type=simple
ExecStart=/home/repo/.local/bin/aurbuilder /home/repo/.aurbuilder

[Install]
WantedBy=multi-user.target

    I need to prevent the AURDEST directory from becoming too large.

    /home/repo/.local/bin/aurgc
---------------------------
#!/bin/bash

readonly argv0=gc
readonly XDG_CACHE_HOME=${XDG_CACHE_HOME:-$HOME/.cache}
readonly AURDEST=${AURDEST:-$XDG_CACHE_HOME/aurutils/sync}

find "${AURDEST}" -name .git -execdir git clean -xf \;

    Thanks to AladW for this neat little script. After an update run seems a decent time to run this script.

    /home/repo/.config/systemd/user/aurupdate.service
-------------------------------------------------
[Unit]
Description=Update AUR packages

[Service]
Type=oneshot
ExecStart=/usr/bin/aursync -cuT --no-view
ExecStartPost=/home/repo/.local/bin/aurgc

[Install]
WantedBy=multi-user.target

    This should run daily.

    /home/repo/.config/systemd/user/aurupdate.timer
-----------------------------------------------
[Unit]
Description=Run aurupdate daily

[Timer]
OnActiveSec=1m
OnCalendar=daily
RandomizedDelaySec=10s

[Install]
WantedBy=timers.target

    Lastly, for all those user services to be activated without a running session, lingering needs to be enabled for the user.

    # loginctl enable-linger repo

    I’ve tested this setup for about a week now, and it seems to work as intended. I have yet to check whether it handles VCS packages, meaning packages that build the latest version from git or any other VCS, rather than a tagged or otherwise marked version, correctly.