nixConfig/README.md

# nixConfig
This repository contains the configuration of all our machines running NixOS.

## Build a machine
There are multiple ways to build and deploy a machine configuration. Which is the
most appropriate depends on the context and scenario. So first there will be a general
explanation how this works and afterwards we will talk about some scenarios.

If you run `nix flake show`, you should get an output similar to this
```
$ nix flake show
git+file:///home/nerf/git/nixConfig?ref=refs%2fheads%2fnyarlathtop&rev=9d0eb749287d1e9e793811759dfa29469ab706dc
├───apps
│   └───x86_64-linux
├───checks
│   └───x86_64-linux
├───devShells
│   └───x86_64-linux
├───formatter
├───legacyPackages
│   └───x86_64-linux omitted (use '--legacy' to show)
├───nixosConfigurations
│   └───nyarlathotep: NixOS configuration
├───nixosModules
├───overlays
└───packages
    └───x86_64-linux
```
we can see there is an output called `nixosConfigurations.nyarlathotep` which contains the configuration of the machine
called nyarlathotep. `nixosConfigurations` is special in that sense, that `nixos-rebuild` will automatically look
for this key and assume how it is structured. The interesting part for us is the derivation `config.system.build.toplevel`.
Its closure contains the whole system and the resulting derivation a script that changes the current system to
that derivation. (called `/bin/switch-to-configuration`).

So what we want to archive is populate the nix store of the target machine with the closure of the derivation
`.#nixosConfigurations.<name>.config.system.build.toplevel` and run the the resulting script on the target machine.


### Local
It has multiple benefits to build the system configuration on the local computer and push it to the target server.
For example one doesn't stress the server with the load of evaluating the expression and building the closure. Also the server
doesn't need to fetch the build dependencies this way. One has a local check if at least the nix syntax was correct.
And so on...

#### Build
If you have this repository local in your current directory, you can just run
```
$ nix build .#nixosConfigurations.<name>.config.system.build.toplevel
```
to build the system configuration of the machine `<name>`.

But you don't need to clone this repository, for more see the `nix flake --help` documentation about flake urls.

#### Copy
After we build the derivation we need to get the closure onto the target system. Luckily nix has tools to do that
via ssh. We could just run:
```
$ nix copy -s --to <however you setup your ssh stuff> .#nixosConfigurations.<name>.config.system.build.toplevel
```
This will evaluate the flake again to get the store path of the given derivation. If we want to avoid this,
we might supply the corresponding store path directly.

The `-s` is important: it makes the target machine substitute all derivations it can (by default from chache.nixos.org).
So you only upload configuration files and self build things.

To be able to copy things to a machine they need to be signed by someone trusted. Additional trusted nix keys are handled
in `./nixos/roles/nix_keys.nix`. So to get yourself trusted you either need to install one derivation from the machine itself,
or find someone who is already trusted, to push your key.

For more information on signing and key creation see `nix store sign --help` and `nix key --help`.

#### Activate
Log into the remote machine and execute (with root privileges)
```
# /nix/store/<storepath>/bin/switch-to-configuration boot
```
That will setup a configuration switch at reboot. You can also switch the configuration live. For more
details consider the `--help` output of that script. The storepath (or at least the hash of the derivation)
is exactly the same it was on your machine.


If you have a `nixos-rebuild` available on your system, it can automatize these things with the `--flake` and
`--target-host` parameters. But there are some pitfalls so look at the `nixos-rebuild` documentation beforehand.

### On the machine

Clone this repository to `/etc/nixos/` and `nixos-rebuild boot` or `nixos-rebuild switch` that will select
the appropriate machine based on hostname.

If the hostname is not correct, or you don't want to clone this flake, you can also use the `--flake` parameter.

In any case, to switch the system configuration you will need to have root privileges on the target machine.

## Installing a new machine

You have written a configuration and now want to deploy it as a new machine. You need to get the build configuration on the
`nodens` machine. You can either use either any of the
versions above, or just continue then the machine will build the configuration implicitly.

### Disk layout

You will need to assemble the disk layout manually, we assume you do it below `/mnt` as the nixos-install tools
assume this as the default location (they have an option to change that consider their `--help` pages).

This repository loads some default configuration that expects certain things. Your hardware configuration of that machine should
reflect those.

- `"/"` is a tmpfs
- `"/persist"` is the place where we keep data that can not be regenerated at any boot, so this should be a permanent disk
- `"/nix"` the place the nixstore resides, needed to boot the machine should also be persistent
- `"/boot"` the place for bootloader configuration and kernel also persistent
- any additional data paths for your machine specific needs. Choose filesystems accordingly.

My recommendation is to put `"/persist"` and `"/nix"` on a joint btrfs labelled `nixos` as subvolumes and `"/boot"` on a separate disk labelled `boot` (because grub
will give you a hard time if you do it as a subvolume or bind mount (even though that should be possible but is an upstream problem)).
For how to configure additional persistent data to be stored in `"/persist"` look at the impermanence section.
I do not recommend this for actual high access application data like databases mailboxes and things like it. You should
think about this as data that if lost can be regenerated with only little problems and read/written only a few times
during setup. (Like the server ssh keys for example). The configuration also setups some paths for `"/persist"` automatically,
again look at the impermanence sections.

### Installing

Just run
```
nixos-install --flake 'git+https://gitea.mathebau.de/Fachschaft/nixConfig?ref=<branchname>#<name>'
```
where `<branchname>` is the branch you install from and `<name>` is the name of the configuration you build.
If the build system is already in the nix store, this will start the installation, else it will first attempt to build
it. That should be the whole installation process, just reboot. The machine should be fully setup. No additional user
or service setup, after the reboot.


## How to write a new machine configuration
At best, you take a first look at already existing configurations. But here are a few guidelines.
Make a new folder in `/nixos/machines`. The name of the folder should match the hostname of your
machine. The only technically required file in there is `configuration.nix`. So create it.

A good skeleton is probably:
```
{config, pkgs, lib, flake-inputs, ... }: {

imports = [
  ./hardware-configuration.nix
  ../../roles
  ../../roles/vm.nix
  ../../vmNetwork.nix

  <your additional imports here>

  ];

  <your system config here>

  vmNetwork.ipv4 = "192.168.0.XX";
  networking.hostname = "<your hostname>"; # this will hopefully disappear if I have time to refactor this.
  system.stateVersion = "<state version at time of install>";
}
```
The import of `../../roles` loads all the nice default setup that all these machines have in common. There the
impermanence configuration is loaded as well as ssh, sops, shared user configuration and much more.
The other two imports are suggestions how you should organize your configuration but not enforced by anything.
In your hardware
configuration you should basically only write you filesystem layout and your hostPlatform. The bootloading stuff
is already taken care of by `../../roles`.

The `flake-inputs` argument is optional, but you can use it if you need to get a hold of the flake inputs,
else this is a complete normal nixos system configuration module (with a lot of settings already imorted
from `../../roles`).

I would recommend to split your configuration into small files you import. If this is something machine specific (like
tied to your ip address hostname), put it into the machine directory. If it is not, put it into `/nixos/roles/` if it
is not but has options to set, put it in `/nixos/modules`.

## How this flake is organized

This flake uses `flake-parts` see [flake.parts](https://flake.parts) for more details. It makes handling
`system` and some other modules related things more convenient.
For the general layout of nixos system configuration and modules, please see the corresponding documentation.

The toplevel `flake.nix` contains the flake inputs as usual and only calls a file `flake-module.nix`.
This toplevel `flake-module.nix` imports further more specialized `flake-modules.nix` files from sub-directories.
Right now the only one is `nixos/flake-module.nix`. But if we start to ship our own software (or software versions,
with specific build flags), this might get more.

### nixos
The `nixos` folder contains all machine configurations. It separates in two folders `nixos/machines` and `nixos/roles`.
The corresponding `flake-module.nix` file automatically searches for `machines/<name>/configuration.nix`, and evalutes
those as nixos configurations, and populates the flake.

#### machines
`nixos/machines` contains all machine specific configuration (in a sub-folder per machine). Like hardware configuration, specific
network configuration. And service configuration that are too closely interwoven with the rest of that machine (for example
mailserver configuration depends heavily on network settings). It also
contains the root configuration for that machine called `configuration.nix`. This file usually only includes other modules.
These `configuration.nix` files are almost usual nix configurations. The only difference is that they take as an extra argument
the flake inputs. This allows them to load modules from these flakes. For example, lobon loads the mathebau-mailman module that way.

#### roles
`nixos/roles` contains configuration that is potentially shared by some machines. It is expected that `nixos/roles/default.nix`
is imported as (`../../roles`) in every machine. Notable are the files `nixos/roles/admins.nix` which contains
common admin accounts for these machines and the additional trusted keys for the nix store.

## sops

We are sharing secrets using [`sops`](https://github.com/getsops/sops) and [`sops-nix`](https://github.com/Mic92/sops-nix)
As of right now we use only `age` keys.
The machine keys are derived from their server ssh keys, that they generate at first boot.
To read out a machines public key run the following command on the corresponding machine.
```
$ nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'
```
User keys are generated by the users.
New keys and machines need entries into the `.sops.yaml` file within the root directory of this repository.

To make a secret available on a given machine you need to configure the following:

```
sops.secrets.example-key = {
  sopsFile = "relative path to file in the repo containing the secrets (optional else the sops.defaultSopsFile is used)";
  path = "optinal path where the secret gets symlinked to, practical if some program expects a specific path";
  owner = user that owns the secret file: config.users.users.nerf.name (for example);
  group = same as user just with groups: config.users.users.nerf.group;
  mode = "permission in usual octet: 0400 (for example)";
};
```
Afterwards the secret should be available in `/run/secrets/example-key`.
If the accessing process is not root it must be member of the group `config.users.groups.keys`
for systemd services this can be archived by setting `serviceConfig.SupplementaryGroups = [ config.users.groups.keys.name ];`
it the service configuration.

## impermanence

These machines are setup with `"/"` as a tmpfs. This is there to keep the machines clean. So no clutter in home
directories, no weird ad-hoc solutions of botching something into `/opt/` or something like this. All will be
gone at reboot.

But there are some files that we want to survive reboots, for example logs or ssh keys. The solution to this is
to have a persistent storage mounted at `/persist` and automatically bind mount the paths of persistent things
to the right places. To set this up we are using the impermanence module. In our configuration this is loaded with
some default files to bind mount (ssh keys, machine-id some nixos specific things). That we have on all machines.

If you keep your application data (like recommended) on a separate partition, the chances are you don't need
to interact with this, as most configuration files will be in the nix store anyway. If the application wants these nix
store files in certain directories, you should use `environment.etc` family of options (consult the nixos documentation
for this). This is for mutable files that are not core application data. (Like ssh keys, for a mailserver one could
think about the hash files (not the db files) of an alias map (if one doesn't want to manage that with
the nix store), things like that).

This should not be (but could be) used for large application databases. It would be more appropriate to mount
its own filesystem for things like that. For small configuration files that are not in the nix-store,
that might be the appropriate solution.

By default the storage is called `persist` and the default path for it is `/persist`. These can be changed
with the `impermanence.name` and `impermanence.storagePath` options. To add paths to this storage you do the
following.

```
environment.persistence.${config.impermanence.name} = {
  directories = [
    "<your path to a directory to persist>"
  ];
  files = [
    "<your path to a file to persist>"
  ];
};
```
For this to work `config` must be binded by the function arguments of you module. So the start of your module looks
something like this:
```
{lib, pkgs, config, ...} :
<module code >
```

# Contributing

Like with all FS projects, you are welcome to contribute. Work is done usually by the person that is most annoyed
by the circumstances or by the person that didn't run fast enough. So we are happy if we get help. That doesn't
mean that we don't need to have some level of quality, people after us needs to work with it. It is live infrastructure
and downtime hurts someone (and in the wrong moment even really bad (Matheball ticket sales for example)).

So here are some guidelines.

## Coding style and linting.
If you run `nix flake check` there are automated checks in place, please make sure to pass them.
There is also a code autoformatter (`alejandra`) incorporated into those. If you want to run
it you can do so over the development shell or by running `nix fmt`.
You can also install
them into your local git repository as pre-commit hooks, and setting up a shell that has
even more tooling by running `nix develop`. That will give you a bash in which you can run
all the checks manually `pre-commit run -a`. This will also run the autoformatter.

## Process for submitting changes

1. If it is something bigger, please open an issue first describing what and why you want to do something.
   If it is just something small, skip this step.
2. Fork the repo and implement your changes in a branch on your fork. Afterwards open a pull request (possibly mentioning the issue).
   Against the main branch.
   - Your branch should be based on an up to date version of main, if it is not consider rebasing.
3. You will need to find someone with the proper rights to approve of your changes, but most of the time there will be request
   for changes first.