This year I rebuilt my main home server using vanilla LXD/LXC instead of Proxmox. One issue I’d had previously was the process of updating all my containers, at times twenty ArchLinux containers which would download the same package repeatedly.
Therefore I looked into sharing the pacman cache (and subsequently AUR cache) with my LXC containers. The process below was inspired by the great wiki article about sharing your Pacman cache over a network.
Permission Setup
First thing we need is a common group mapping between host and containers to share files and permissions. For this post I will be using the following values:
- Group Name:
dcache - Group GID:
9999
On host and container(s):
groupadd --gid 9999 dcacheAllow root to use this supplementary group on the host and container(s):
usermod -aG dcache rootEnable your LXC unprivileged containers the ability to map this group. Append to /etc/subgid:
root:9999:1Allow the container to map these IDs:
echo -en "gid 9999 9999" | lxc config set [CONTAINER_NAME] raw.idmap -This group can also be used to write files back to the host from inside the container. You can achieve this by changing the users primary group to dcache or overriding a systemd service file changing the executing group.
Setup Pacman
This section will assume you have setup an AUR helper to build and cache packages. In my instance I use aurutils and save packages to /var/cache/aur/custompkgs. I compile my AUR packages on the host as this bypasses any restrictive CPU limits I have set on my containers designed to limit services during normal operation.
By default pacman cache’s group is set to root, lets change this on our host to the new group:
setfacl -R -m g:dcache:rw /var/cache/pacman/pkg
setfacl -R -m d:g:dcache:rw /var/cache/pacman/pkg
chmod -R root:dcache /var/cache/pacman/pkg
setfacl -R -m g:dcache:rw /var/cache/aur/custompkgs
setfacl -R -m d:g:dcache:rw /var/cache/aur/custompkgs
chmod -R root:dcache /var/cache/aur/custompkgsCreate mount points in our container:
lxc exec [CONTAINER_NAME] -- mkdir /mnt/pkg-cache
lxc exec [CONTAINER_NAME] -- mkdir /mnt/aur-cacheShare host folder to container mount points:
lxc config device add [CONTAINER_NAME] pkg-cache disk source=/var/cache/pacman/pkg/ path=/mnt/pkg-cache/
lxc config device add [CONTAINER_NAME] aur-cache disk source=/var/cache/aur/custompkgs/ path=/mnt/aur-cache/Instruct Pacman where to find cached packages, modify /etc/pacman.conf to include the following lines:
CacheDir = /var/cache/pacman/pkg/
CacheDir = /mnt/pkg-cache/
CacheDir = /mnt/aur-cache/In my case for aurutils I have to instruct pacman where it can find the AUR packages by adding the following lines to /etc/pacman.conf:
[custom]
SigLevel = Optional TrustAll
Server = file:///mnt/aur-cacheYou should now have Pacman sharing cache between your host and containers!
Reflector with mirrorlist syncing
Another point of duplication was Pacman’s /etc/pacman.d/mirrorlist. I use reflector to keep my mirrorlist up to date which would be identical for every container. Instead of running reflector in every container I wrote a small piece of automation that allowed the hosts mirrorlist to be pushed to every container.
Host files can be pushed into a container with the following command:
lxc file push [SOURCE_FILE] [CONTAINER_NAME][DESTINATION_FILE]Using LXC’s custom metadata functionality we tag every container that requires the updated mirrorlist with the following command:
lxc config set [CONTAINER_NAME] user.reflector=trueThe following systemd service file saved to /etc/systemd/system/lxd-reflector-update.service loops through all containers with user.reflector=true metadata and updates the /etc/pacman.d/mirrorlist file:
[Unit]
Description=Pushes /etc/pacman.d/mirrorlist to all LXD containers with user.reflector=true metadata
[Service]
ExecStart=/bin/bash -c 'lxc list "user.reflector=true" -f csv -c n | xargs -I {} lxc file push /etc/pacman.d/mirrorlist {}/etc/pacman.d/'
Type=oneshot
[Install]
WantedBy=multi-user.targetThen add a timer /etc/systemd/system/lxd-reflector-update.timer:
[Unit]
Description=Run lxd-reflector-update daily
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.targetYou can then enable and start the timer to run the update daily:
systemctl enable lxd-reflector-update.timer
systemctl start lxd-reflector-update.timerIn future I would like to override the reflector.service to execute lxd-reflector-update.service once it has updated.
Automation
There have been a lot of commands throughout this post. I don’t remember them all, as such I have a simple bash script that automates the setup of a new container. Some of this can be achieved by LXD profiles but I’m yet to investigate that.
# Read in container name and launch ArchLinux
read -p "Container name: " name
lxc launch images:archlinux $name
# Wait for network connection
sleep 3
# Setup locale
lxc exec $name -- sed -i '/^#en_US.UTF-8.*/s/^#//g' /etc/locale.gen
lxc exec $name -- sed -i 's/^LANG=.*/LANG="en_US.UTF-8"/' /etc/locale.conf
lxc exec $name -- locale-gen
# Setup timezone
lxc exec $name -- ln -sf /usr/share/zoneinfo/Country/Region /etc/localtime
# Setup pacman + aur
lxc file push /etc/pacman.d/mirrorlist $name/etc/pacman.d/
lxc exec $name -- sed -i 's/^#NoExtract.*/NoExtract = etc\/pacman.d\/mirrorlist/' /etc/pacman.conf
lxc exec $name -- sed -i 's/^#CacheDir\s*=.*/CacheDir = \/var\/cache\/pacman\/pkg\/\nCacheDir = \/mnt\/pkg-cache\/\nCacheDir = \/mnt\/aur-cache\//' /etc/pacman.conf
lxc exec $name -- sed -i -e '$a\[custom\]\nSigLevel = Optional TrustAll\nServer = file:\/\/\/mnt\/aur-cache' /etc/pacman.conf
lxc exec $name -- mkdir /mnt/pkg-cache
lxc exec $name -- mkdir /mnt/aur-cache
lxc config device add $name pkg-cache disk source=/var/cache/pacman/pkg/ path=/mnt/pkg-cache/
lxc config device add $name aur-cache disk source=/var/cache/aur/custompkgs/ path=/mnt/aur-cache/
# Install common packages
lxc exec $name -- pacman -Syu --noconfirm git htop wget vim pacman-contrib sudo tmux curl openssh ufw
# Add groups
lxc exec $name -- groupadd --gid 9999 dcache
lxc exec $name -- usermod -aG dcache root
echo -en "gid 9999 9999" | lxc config set $name raw.idmap -
# Add normal user
lxc exec $name -- useradd -G users,wheel,dcache -s /bin/bash -m [USER]
# Add user to sudoers
lxc exec $name -- sed -i '/^#\s\%wheel\sALL=(ALL:ALL)\sALL/s/^#\s//g' /etc/sudoers
# Push SSH key
lxc exec $name -- mkdir /home/[USER]/.ssh
lxc exec $name -- chmod 700 /home/[USER]/.ssh
lxc file push /home/[USER]/.ssh/authorized_keys $name/home/[USER]/.ssh/
lxc exec $name -- chown -R [USER]:[USER] /home/[USER]/.ssh
lxc exec $name -- chmod 600 /home/[USER]/.ssh/authorized_keys
# Lockdown SSH
lxc exec $name -- sed -i 's/^#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
lxc exec $name -- sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
lxc exec $name -- sed -i 's/^UsePAM yes/UsePAM no/' /etc/ssh/sshd_config
lxc exec $name -- sed -i '$aAllowUsers [USER]' /etc/ssh/sshd_config
lxc exec $name -- systemctl start sshd
lxc exec $name -- systemctl enable sshd
# Firewall
lxc exec $name -- ufw allow from 10.0.0.0/24 proto tcp to any port 22
lxc exec $name -- ufw enable
lxc exec $name -- systemctl enable ufw
# Set passwords
lxc exec $name -- passwd
lxc exec $name -- passwd [USER]
# Set metadata
lxc config set $name user.reflector=trueReach out and let me know if there is a better way!