Getting to boot nixos on Xen #1

New issue

Closed

opened 2023-09-24 14:15:31 +00:00 by nerf · 1 comment

nerf commented

2023-09-24 14:15:31 +00:00

Owner

This is a bit more of a nightmare then it should be. We want certain things:

The guest should be able to update it's own kernel
We don't want to emulate the whole hardware
We want to choose generations in some kind of boot menu (this requires honoring the guests bootloader config)

This basically leaves us with the following xen virtualization modes:
pv and the newer pvh.

Both of them load images directly without emulating some sort of BIOS/efi. We some options to do this.

Hardcode kernel path on dom0
pygrub
pvgrub in pv mode
pvhgrub in pvh mode

1 is not really an option for us, as this won't give the kernel control about its boot options and kernel. 2 is out, as pygrub
is not capable of parsing the grub.cfg nixos generates.
Leaves us with 3 and 4. These both work by booting a grub image as a kernel (from xen perspective), hardcoded from dom0 into
the guest context and this grub then parses the guest config to load the real kernel. (pvgrub and pvhgrub are both part of grub not of xen).
pvgrub has a bug that fails booting zstd kernels (dafug these are selfextracting archives grub shouldn't handle the compression anyway).
leaves us exactly with pvhgrub.

Luckily for us debian supplies us with pvhgrub images with the grub-xen-host package.

So we can get a booting nixos config in the following way:

type = 'pvh'
kernel = '/usr/lib/grub-xen/grub-i386-xen_pvh.bin'
extra = '(hd0)/boot/grub/grub.cfg'

This assumes that the hardrive with the grub config is the first one.

That leaves us with the bootstrapping problem:

One working way with several drawbacks (if someone finds something better tell me):

We can boot the nixos install iso in hvm mode with only one partition and the cdrom. Install
nixos with a basic configuration, reboot the system with more hardware in pvh mode and fix all the mount
points later.

This is not ideal as for example /var/ will already be populated when the filesystem that should go there becomes available.

It kind of works though.

This is a bit more of a nightmare then it should be. We want certain things: 1. The guest should be able to update it's own kernel 2. We don't want to emulate the whole hardware 3. We want to choose generations in some kind of boot menu (this requires honoring the guests bootloader config) This basically leaves us with the following xen virtualization modes: pv and the newer pvh. Both of them load images directly without emulating some sort of BIOS/efi. We some options to do this. 1. Hardcode kernel path on dom0 2. pygrub 3. pvgrub in pv mode 4. pvhgrub in pvh mode 1 is not really an option for us, as this won't give the kernel control about its boot options and kernel. 2 is out, as pygrub is not capable of parsing the grub.cfg nixos generates. Leaves us with 3 and 4. These both work by booting a grub image as a kernel (from xen perspective), hardcoded from dom0 into the guest context and this grub then parses the guest config to load the real kernel. (pvgrub and pvhgrub are both part of grub not of xen). pvgrub has a bug that fails booting zstd kernels (dafug these are selfextracting archives grub shouldn't handle the compression anyway). leaves us exactly with pvhgrub. Luckily for us debian supplies us with pvhgrub images with the `grub-xen-host` package. So we can get a booting nixos config in the following way: ``` type = 'pvh' kernel = '/usr/lib/grub-xen/grub-i386-xen_pvh.bin' extra = '(hd0)/boot/grub/grub.cfg' ``` This assumes that the hardrive with the grub config is the first one. That leaves us with the bootstrapping problem: One working way with several drawbacks (if someone finds something better tell me): We can boot the nixos install iso in hvm mode with only one partition and the cdrom. Install nixos with a basic configuration, reboot the system with more hardware in pvh mode and fix all the mount points later. This is not ideal as for example `/var/` will already be populated when the filesystem that should go there becomes available. It kind of works though.