One of my work colleagues was commenting that they like the Xen PV model - where you have a fairly lightweight hypervisor that runs cooperating kernels (or, as Xen calls them, "domains"). They've been meaning to try out NixOS but couldn't figure out how to build a debootstrap-style root FS.
NixOS is interesting in that really the only thing that needs to exist on the disk is a /nix/store directory containing a built system -- everything else somewhat springs out of nothing as long as the boot process can figure out what exactly it was you were intending to boot.
This means that, if you have a system with Nix on (which doesn't have to be NixOS -- I'm going to use Debian 12 running as a Xen dom0 to demonstrate), then it's relatively easy to build a Xen PV compatible rootfs.
Table of contents
- Getting your host system set up
- Building a NixOS system
- Making the disk image
- Booting this in Xen
- Bonus round: Making a trivial configuration change
Getting your host system set up
Installing Nix itself
You'll need Nix installed. I prefer to use the Determinate Nix Installer in most cases rather than the official one - it does a better job, and provides an uninstall tool that works, so if you decide you're not interested in Nix anymore then it's easy to get rid of again.
So...
$ curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
info: downloading installer https://install.determinate.systems/nix/tag/v0.11.0/nix-installer-x86_64-linux
`nix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
Nix install plan (v0.11.0)
Planner: linux (with default settings)
Planned actions:
* Create directory `/nix`
* Fetch `https://releases.nixos.org/nix/nix-2.17.0/nix-2.17.0-x86_64-linux.tar.xz` to `/nix/temp-install-dir`
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Create build group (GID 30000)
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Configure the shell profiles
* Create directory `/etc/tmpfiles.d`
* Configure Nix daemon related settings with systemd
* Remove directory `/nix/temp-install-dir`
Proceed? ([Y]es/[n]o/[e]xplain): y
INFO Step: Create directory `/nix`
INFO Step: Provision Nix
INFO Step: Create build group (GID 30000)
INFO Step: Configure Nix
INFO Step: Create directory `/etc/tmpfiles.d`
INFO Step: Configure Nix daemon related settings with systemd
INFO Step: Remove directory `/nix/temp-install-dir`
Nix was installed successfully!
To get started using Nix, open a new shell or run `. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh`
Getting a copy of nixpkgs
The Determinate Nix installer doesn't pull a nixpkgs channel; it's expecting you to use flakes. For the purposes of this demo though, I'm not really interested in doing that right now, so I'm going to clone nixpkgs myself.
To speed things up, I'm just cloning the latest nixpkgs NixOS release branch (23.05 "Stoat" at the time of writing).
$ git clone --depth 1 --branch nixos-23.05 https://github.com/NixOS/nixpkgs.git
Cloning into 'nixpkgs'...
remote: Enumerating objects: 58617, done.
remote: Counting objects: 100% (58617/58617), done.
remote: Compressing objects: 100% (37070/37070), done.
remote: Total 58617 (delta 2534), reused 55277 (delta 2393), pack-reused 0
Receiving objects: 100% (58617/58617), 43.47 MiB | 9.97 MiB/s, done.
Resolving deltas: 100% (2534/2534), done.
Updating files: 100% (35411/35411), done.
Building a NixOS system
Writing a system configuration
Next up on our grand tour: a NixOS system configuration. This is the file which describes NixOS system looks like and should encapsulate most of the non-runtime configuration.
I wrote this file out to system-configuration.nix:
{ config, lib, pkgs, ... }:
{
# Ensure we have the Xen block device module available during boot.
boot.initrd.availableKernelModules = [ "xen-blkfront" "xen-kbdfront" ];
# Mount /dev/xvda1 at /.
fileSystems."/" = {
device = "/dev/xvda1";
fsType = "ext4";
};
# Disable GRUB: we're not using it here.
boot.loader.grub.enable = false;
networking = {
# Set our hostname.
hostName = "nixos-inside-xen";
# Disable global DHCP but enable it on the NIC we will get.
useDHCP = false;
interfaces.enX0.useDHCP = true;
# Use systemd-networkd, rather than legacy script-based networking.
useNetworkd = true;
};
# Use systemd-as-stage-1, rather than legacy script-based stage 1/stage 2.
boot.initrd.systemd.enable = true;
# Set our timezone.
time.timeZone = "Europe/London";
# Create a 'user' user, with sudo powers.
users.users.user = {
isNormalUser = true;
extraGroups = [ "wheel" ];
password = "thisisinsecure";
};
# Enable SSH for good measure.
services.openssh.enable = true;
# Make sure we have an editor.
environment.systemPackages = [ pkgs.vim ];
# This can also be written as:
# environment.systemPackages = with pkgs; [ vim ];
system.stateVersion = "23.11";
}
Building the system
Now that we have a NixOS system configuration, we can use the NixOS machinery to turn this into a system.
# Assuming that 'nixpkgs' and 'system-configuration.nix' are in the current directory:
$ nix-build 'nixpkgs/nixos' -A system --arg configuration ./system-configuration.nix
# The ./ before system-configuration.nix is important to ensure Nix interprets
# it as a path rather than a string.
[... a lot of output later ...]
building '/nix/store/125c1x5n9lgcx4gf7wswdy4m8kawmf4i-etc.drv'...
building '/nix/store/g2mlr67i0z45n695wvc5rsnpiqcmahzk-nixos-system-nixos-inside-xen-23.05pre-git.drv'...
/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git
The final line output (which also gets written to the current directory as a
symlink named result) is the Nix store path to the system you've just built.
Now we have the system, we can start making the disk image for Xen.
Making the disk image
Formatting a disk
For my purposes, I'm going to make a 10GB disk image with a single ext4 partition. If you want to do something different, you'll need to adjust the NixOS configuration above accordingly, and rebuild it.
# Make a 10G empty file.
$ truncate --size 10G nixos.img
# Create a single partition.
$ fdisk nixos.img
Welcome to fdisk (util-linux 2.38.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS (MBR) disklabel with disk identifier 0x3980dcd5.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-20971519, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-20971519, default 20971519):
Created a new partition 1 of type 'Linux' and of size 10 GiB.
Command (m for help): w
The partition table has been altered.
Syncing disks.
# Make a partitioned loop device.
$ sudo losetup --find --partscan --show nixos.img
/dev/loop0
# Format the new partition with ext4.
$ sudo mkfs.ext4 -L NIXOS /dev/loop0p1
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 2621184 4k blocks and 655360 inodes
Filesystem UUID: 77a3e697-e6b0-4645-99f3-47528256d47b
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
Installing NixOS
We'll need the disk mounted, so I'll do that first:
$ mkdir mnt
$ sudo mount /dev/loop0p1 mnt
Now actually installing the system is as simple as using nix to copy it:
$ sudo $(which nix) copy --to $(readlink -f mnt) ./result --no-check-sigs
[... progress output ...]
# Now there's a "nix/store" inside the disk.
$ ls mnt/nix
store var
$ ls mnt/nix/store | wc -l
503
We can now set this as the current system inside the image - this mostly just creates a symlink, but this sets up the generational semantics of NixOS.
$ sudo nix-env -p mnt/nix/var/nix/profiles/system --set ./result
$ ls mnt/nix/var/nix/profiles -l
total 8
drwxr-xr-x 2 root root 4096 Aug 19 14:49 per-user
lrwxrwxrwx 1 root root 13 Aug 19 14:59 system -> system-1-link
lrwxrwxrwx 1 root root 86 Aug 19 14:59 system-1-link -> /nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git
As you can see, this has created the system link, which points at the current
generation (system-1-link), which finally points at the actual system.
For our convenience, we'll copy the configuration into the disk image. The
conventional location is /etc/nixos/configuration.nix. This isn't strictly
necessary to have a working system, but it makes upkeep much easier.
$ sudo mkdir -p mnt/etc/nixos
$ sudo cp system-configuration.nix /etc/nixos/configuration.nix
For better or for worse, we'll also set up the NixOS channel definition.
$ sudo mkdir mnt/root
$ echo "https://nixos.org/channels/nixos-23.05 nixos" | sudo tee mnt/root/.nix-channels
https://nixos.org/channels/nixos-23.05 nixos
OK, we're done with the disk image now, so we can unmount it:
$ sudo umount mnt
$ sudo losetup -d /dev/loop0
Booting this in Xen
We now have a disk image with NixOS installed. We don't need to copy the kernel or ramdisk out of it because we already have it on the host. For longer term use, though, I suggest using Xen's built in grub emulator or similar to make sure that things are kept up to date. This will boot the same system configuration every time.
I wrote this into my domain.conf - you'll need to substitute the path to
your own Nix system, and your nixos.img.
kernel = "/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git/kernel"
ramdisk = "/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git/initrd"
memory = 2048
name = "nixos"
vif = [ '' ]
dhcp = "dhcp"
cmdline = "xencons=tty init=/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git/init"
disk = ['/home/lukegb/nix-blogpost/nixos.img,,hda']
And now we can boot the system:
$ sudo xl create ./domain.conf -c
[... it boots ...]
<<< Welcome to NixOS 23.05pre-git (x86_64) - hvc0 >>>
Run 'nixos-help' for the NixOS manual.
nixos-inside-xen login:
You can then log in with the user / thisisinsecure pair. The user user
has sudo permission, and when you're done you can shut the VM down. Assuming
that your Xen system is set up similar to mine, with a xenbr0 that has internet
access with a DHCP server (and hopefully even IPv6...!), then you should get
an IP address inside the VM as well.
Updating the nixpkgs channel
The first thing you'll probably want to do is actually fetch the Nix channel that we configured while creating the disk image:
[user@nixos-inside-xen:~]$ sudo nix-channel --update
unpacking channels...
Now you can freely experiment with Nix.
Bonus round: Making a trivial configuration change
This is some bonus work, to explore making a NixOS configuration change. You already have a working system at this point so you can stop reading.
As an example, say you want to have mtr available inside your system. It's
not installed by default:
[user@nixos-inside-xen:~]$ mtr --report 8.8.8.8
The program 'mtr' is not in your PATH. It is provided by several packages.
You can make it available in an ephemeral shell by typing one of the following:
nix-shell -p mtr
nix-shell -p mtr-gui
You can follow the instructions and use nix-shell to provide it temporarily:
[user@nixos-inside-xen:~]$ nix-shell -p mtr
[... nix downloads mtr from cache ...]
[nix-shell:~]$ mtr --report 8.8.8.8
Start: 2023-08-19T15:10:53+0100
HOST: nixos-inside-xen Loss% Snt Last Avg Best Wrst StDev
1.|-- _gateway 0.0% 10 0.6 0.5 0.5 0.6 0.0
2.|-- tuvok.gnet-tuvok.mldn-rd. 20.0% 10 2.0 2.2 2.0 2.6 0.2
3.|-- blade-tuvok.public.as2054 0.0% 10 2.5 2.4 1.9 2.5 0.2
4.|-- 195.66.224.125 0.0% 10 2.7 3.1 2.5 5.3 0.9
5.|-- 74.125.242.97 0.0% 10 4.7 4.7 4.2 7.2 0.9
6.|-- 192.178.46.87 0.0% 10 3.6 3.6 3.2 4.7 0.4
7.|-- dns.google 0.0% 10 3.2 3.4 3.0 3.7 0.2
except it won't quite work properly, because its mtr-packet process isn't
privileged:
[nix-shell:~]$ mtr --udp --report 8.8.8.8
Start: 2023-08-19T15:13:03+0100
HOST: nixos-inside-xen Loss% Snt Last Avg Best Wrst StDev
[nix-shell:~]$ exit
[user@nixos-inside-xen:~]$
You can solve this by installing it in your NixOS configuration using the
programs.mtr.enable configuration option, defined
in this module.
This both installs the package (into environment.systemPackages), but
also installs a setcap wrapper for mtr-packet.
Edit /etc/nixos/configuration.nix, and add programs.mtr.enable = true;
somewhere, probably just below the environment.systemPackages line.
Once you've done that, rebuild and switch to the new system:
[user@nixos-inside-xen:~]$ sudo nixos-rebuild switch
building Nix...
building the system configuration...
these 11 derivations will be built:
/nix/store/c9nkzphpp4hfgbx4da8pfh4b6xcfwy1s-system-path.drv
/nix/store/0bm8y3d51c93dr27jkzzaaz0kv43vv62-unit-systemd-fsck-.service.drv
/nix/store/k984vii2lsh7yjz8acvbfqgph0vyssfj-dbus-1.drv
/nix/store/j9lmndz2by4ridi3vyzbd32mnay2pn4q-X-Restart-Triggers.drv
/nix/store/d0j9nmy6qylmgllaw21l6w11b0gkhxkx-unit-dbus.service.drv
/nix/store/1cjakrjkp764r26fddkpixjp6cff0kq7-user-units.drv
/nix/store/8l53jy12sbls3ppxc7pqh40ialnw53x4-unit-dbus.service.drv
/nix/store/vnky6khlsj4zasp4q8g6rwp2b3jhp99l-system-units.drv
/nix/store/d5zmdd9g1zvv7f5s9w2n3praq8d4k1p6-etc.drv
/nix/store/qixzqgmf54k8xypajycry4v42llhhwi1-ensure-all-wrappers-paths-exist.drv
/nix/store/hp0dhaw2rgcpa7y70rfhnmnr2qiw8lm2-nixos-system-nixos-inside-xen-23.05.2891.ae521bd4e460.drv
[...a little bit of output...]
building '/nix/store/hp0dhaw2rgcpa7y70rfhnmnr2qiw8lm2-nixos-system-nixos-inside-xen-23.05.2891.ae521bd4e460.drv'...
Warning: do not know how to make this configuration bootable; please enable a boot loader.
activating the configuration...
setting up /etc...
reloading user units for user...
setting up tmpfiles
reloading the following units: dbus.service
...and you now have mtr available system-wide, but now it works in UDP mode:
[user@nixos-inside-xen:~]$ mtr --udp --report 8.8.8.8
Start: 2023-08-19T15:19:03+0100
HOST: nixos-inside-xen Loss% Snt Last Avg Best Wrst StDev
1.|-- _gateway 0.0% 10 0.5 0.5 0.4 0.6 0.0
2.|-- tuvok.gnet-tuvok.mldn-rd. 90.0% 10 2.3 2.3 2.3 2.3 0.0
3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4.|-- 195.66.224.125 0.0% 10 2.8 2.8 2.5 3.3 0.3
5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0