nixConfig

This repository contains the configuration of all our machines running NixOS.

Build a machine

There are multiple ways to build and deploy a machine configuration. Which is the most appropriate depends on the context and scenario. So first there will be a general explanation how this works and afterwards we will talk about some scenarios.

If you run nix flake show, you should get an output similar to this

$ nix flake show
git+file:///home/nerf/git/nixConfig?ref=refs%2fheads%2fnyarlathtop&rev=9d0eb749287d1e9e793811759dfa29469ab706dc
├───apps
│   └───x86_64-linux
├───checks
│   └───x86_64-linux
├───devShells
│   └───x86_64-linux
├───formatter
├───legacyPackages
│   └───x86_64-linux omitted (use '--legacy' to show)
├───nixosConfigurations
│   └───nyarlathotep: NixOS configuration
├───nixosModules
├───overlays
└───packages
    └───x86_64-linux

we can see there is an output called nixosConfigurations.nyarlathotep which contains the configuration of the machine called nyarlathotep. nixosConfigurations is special in that sense, that nixos-rebuild will automatically look for this key and assume how it is structured. The interesting part for us is the derivation config.system.build.toplevel. Its closure contains the whole system and the resulting derivation a script that changes the current system to that derivation. (called /bin/switch-to-configuration).

So what we want to archive is populate the nix store of the target machine with the closure of the derivation .#nixosConfigurations.<name>.config.system.build.toplevel and run the the resulting script on the target machine.

Local

It has multiple benefits to build the system configuration on the local computer and push it to the target server. For example one doesn't stress the server with the load of evaluating the expression and building the closure. Also the server doesn't need to fetch the build dependencies this way. One has a local check if at least the nix syntax was correct. And so on...

Build

If you have this repository local in your current directory, you can just run

$ nix build .#nixosConfigurations.<name>.config.system.build.toplevel

to build the system configuration of the machine <name>.

But you don't need to clone this repository, for more see the nix flake --help documentation about flake urls.

Copy

After we build the derivation we need to get the closure onto the target system. Luckily nix has tools to do that via ssh. We could just run:

$ nix copy -s --to <however you setup your ssh stuff> .#nixosConfigurations.<name>.config.system.build.toplevel

This will evaluate the flake again to get the store path of the given derivation. If we want to avoid this, we might supply the corresponding store path directly.

The -s is important: it makes the target machine substitute all derivations it can (by default from chache.nixos.org). So you only upload configuration files and self build things.

To be able to copy things to a machine they need to be signed by someone trusted. Additional trusted nix keys are handled in ./nixos/roles/nix_keys.nix. So to get yourself trusted you either need to install one derivation from the machine itself, or find someone who is already trusted, to push your key.

For more information on signing and key creation see nix store sign --help and nix key --help.

Activate

Log into the remote machine and execute (with root privileges)

# /nix/store/<storepath>/bin/switch-to-configuration boot

That will setup a configuration switch at reboot. You can also switch the configuration live. For more details consider the --help output of that script. The storepath (or at least the hash of the derivation) is exactly the same it was on your machine.

If you have a nixos-rebuild available on your system, it can automatize these things with the --flake and --target-host parameters. But there are some pitfalls so look at the nixos-rebuild documentation beforehand.

On the machine

Clone this repository to /etc/nixos/ and nixos-rebuild boot or nixos-rebuild switch that will select the appropriate machine based on hostname.

If the hostname is not correct, or you don't want to clone this flake, you can also use the --flake parameter.

In any case, to switch the system configuration you will need to have root privileges on the target machine.

Installing a new machine

You have written a configuration and now want to deploy it as a new machine. You need to get the build configuration on the nodens machine. You can either use either any of the versions above, or just continue then the machine will build the configuration implicitly.

Disk layout

You will need to assemble the disk layout manually, we assume you do it below /mnt as the nixos-install tools assume this as the default location (they have an option to change that consider their --help pages).

This repository loads some default configuration that expects certain things. Your hardware configuration of that machine should reflect those.

• "/" is a tmpfs
• "/persist" is the place where we keep data that can not be regenerated at any boot, so this should be a permanent disk
• "/nix" the place the nixstore resides, needed to boot the machine should also be persistent
• "/boot" the place for bootloader configuration and kernel also persistent
• any additional data paths for your machine specific needs. Choose filesystems accordingly.

My recommendation is to put "/persist" and "/nix" on a joint btrfs labelled nixos as subvolumes and "/boot" on a separate disk labelled boot (because grub will give you a hard time if you do it as a subvolume or bind mount (even though that should be possible but is an upstream problem)). For how to configure additional persistent data to be stored in "/persist" look at the impermanence section. I do not recommend this for actual high access application data like databases mailboxes and things like it. You should think about this as data that if lost can be regenerated with only little problems and read/written only a few times during setup. (Like the server ssh keys for example). The configuration also setups some paths for "/persist" automatically, again look at the impermanence sections.

Installing

Just run

nixos-install --flake 'git+https://gitea.mathebau.de/Fachschaft/nixConfig?ref=<branchname>#<name>'

where <branchname> is the branch you install from and <name> is the name of the configuration you build. If the build system is already in the nix store, this will start the installation, else it will first attempt to build it. That should be the whole installation process, just reboot. The machine should be fully setup. No additional user or service setup, after the reboot.

How to write a new machine configuration

At best, you take a first look at already existing configurations. But here are a few guidelines. Make a new folder in /nixos/machines. The name of the folder should match the hostname of your machine. The only technically required file in there is configuration.nix. So create it.

A good skeleton is probably:

{config, pkgs, lib, flake-inputs, ... }: {

imports = [
  ./hardware-configuration.nix
  ../../roles
  ../../roles/vm.nix
  ../../vmNetwork.nix

  <your additional imports here>

  ];

  <your system config here>

  vmNetwork.ipv4 = "192.168.0.XX";
  networking.hostname = "<your hostname>"; # this will hopefully disappear if I have time to refactor this.
  system.stateVersion = "<state version at time of install>";
}

The import of ../../roles loads all the nice default setup that all these machines have in common. There the impermanence configuration is loaded as well as ssh, sops, shared user configuration and much more. The other two imports are suggestions how you should organize your configuration but not enforced by anything. In your hardware configuration you should basically only write you filesystem layout and your hostPlatform. The bootloading stuff is already taken care of by ../../roles.

The flake-inputs argument is optional, but you can use it if you need to get a hold of the flake inputs, else this is a complete normal nixos system configuration module (with a lot of settings already imorted from ../../roles).

I would recommend to split your configuration into small files you import. If this is something machine specific (like tied to your ip address hostname), put it into the machine directory. If it is not, put it into /nixos/roles/ if it is not but has options to set, put it in /nixos/modules.

How this flake is organized

This flake uses flake-parts see flake.parts for more details. It makes handling system and some other modules related things more convenient. For the general layout of nixos system configuration and modules, please see the corresponding documentation.

The toplevel flake.nix contains the flake inputs as usual and only calls a file flake-module.nix. This toplevel flake-module.nix imports further more specialized flake-modules.nix files from sub-directories. Right now the only one is nixos/flake-module.nix. But if we start to ship our own software (or software versions, with specific build flags), this might get more.

nixos

The nixos folder contains all machine configurations. It separates in two folders nixos/machines and nixos/roles. The corresponding flake-module.nix file automatically searches for machines/<name>/configuration.nix, and evalutes those as nixos configurations, and populates the flake.

machines

nixos/machines contains all machine specific configuration (in a sub-folder per machine). Like hardware configuration, specific network configuration. And service configuration that are too closely interwoven with the rest of that machine (for example mailserver configuration depends heavily on network settings). It also contains the root configuration for that machine called configuration.nix. This file usually only includes other modules. These configuration.nix files are almost usual nix configurations. The only difference is that they take as an extra argument the flake inputs. This allows them to load modules from these flakes. For example, lobon loads the mathebau-mailman module that way.

roles

nixos/roles contains configuration that is potentially shared by some machines. It is expected that nixos/roles/default.nix is imported as (../../roles) in every machine. Notable are the files nixos/roles/admins.nix which contains common admin accounts for these machines and the additional trusted keys for the nix store.

sops

We are sharing secrets using sops and sops-nix As of right now we use only age keys. The machine keys are derived from their server ssh keys, that they generate at first boot. To read out a machines public key run the following command on the corresponding machine.

$ nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'

User keys are generated by the users. New keys and machines need entries into the .sops.yaml file within the root directory of this repository.

To make a secret available on a given machine you need to configure the following:

sops.secrets.example-key = {
  sopsFile = "relative path to file in the repo containing the secrets (optional else the sops.defaultSopsFile is used)";
  path = "optinal path where the secret gets symlinked to, practical if some program expects a specific path";
  owner = user that owns the secret file: config.users.users.nerf.name (for example);
  group = same as user just with groups: config.users.users.nerf.group;
  mode = "permission in usual octet: 0400 (for example)";
};

Afterwards the secret should be available in /run/secrets/example-key. If the accessing process is not root it must be member of the group config.users.groups.keys for systemd services this can be archived by setting serviceConfig.SupplementaryGroups = [ config.users.groups.keys.name ]; it the service configuration.

impermanence

These machines are setup with "/" as a tmpfs. This is there to keep the machines clean. So no clutter in home directories, no weird ad-hoc solutions of botching something into /opt/ or something like this. All will be gone at reboot.

But there are some files that we want to survive reboots, for example logs or ssh keys. The solution to this is to have a persistent storage mounted at /persist and automatically bind mount the paths of persistent things to the right places. To set this up we are using the impermanence module. In our configuration this is loaded with some default files to bind mount (ssh keys, machine-id some nixos specific things). That we have on all machines.

If you keep your application data (like recommended) on a separate partition, the chances are you don't need to interact with this, as most configuration files will be in the nix store anyway. If the application wants these nix store files in certain directories, you should use environment.etc family of options (consult the nixos documentation for this). This is for mutable files that are not core application data. (Like ssh keys, for a mailserver one could think about the hash files (not the db files) of an alias map (if one doesn't want to manage that with the nix store), things like that).

This should not be (but could be) used for large application databases. It would be more appropriate to mount its own filesystem for things like that. For small configuration files that are not in the nix-store, that might be the appropriate solution.

By default the storage is called persist and the default path for it is /persist. These can be changed with the impermanence.name and impermanence.storagePath options. To add paths to this storage you do the following.

environment.persistence.${config.impermanence.name} = {
  directories = [
    "<your path to a directory to persist>"
  ];
  files = [
    "<your path to a file to persist>"
  ];
};

For this to work config must be binded by the function arguments of you module. So the start of your module looks something like this:

{lib, pkgs, config, ...} :
<module code >

Contributing

Like with all FS projects, you are welcome to contribute. Work is done usually by the person that is most annoyed by the circumstances or by the person that didn't run fast enough. So we are happy if we get help. That doesn't mean that we don't need to have some level of quality, people after us needs to work with it. It is live infrastructure and downtime hurts someone (and in the wrong moment even really bad (Matheball ticket sales for example)).

So here are some guidelines.

Coding style and linting.

If you run nix flake check there are automated checks in place, please make sure to pass them. There is also a code autoformatter (alejandra) incorporated into those. If you want to run it you can do so over the development shell or by running nix fmt. You can also install them into your local git repository as pre-commit hooks, and setting up a shell that has even more tooling by running nix develop. That will give you a bash in which you can run all the checks manually pre-commit run -a. This will also run the autoformatter.

Process for submitting changes

1. If it is something bigger, please open an issue first describing what and why you want to do something. If it is just something small, skip this step.
2. Fork the repo and implement your changes in a branch on your fork. Afterwards open a pull request (possibly mentioning the issue). Against the main branch.
   • Your branch should be based on an up to date version of main, if it is not consider rebasing.
3. You will need to find someone with the proper rights to approve of your changes, but most of the time there will be request for changes first.