diff --git a/README.md b/README.md index 44e28aa..2c6a796 100644 --- a/README.md +++ b/README.md @@ -1,38 +1,280 @@ # nixConfig +This repository contains the configuration of all our machines running NixOS. ## Build a machine +There are multiple ways to build and deploy a machine configuration. Which is the +most appropriate depends on the context and scenario. So first there will be a general +explanation how this works and afterwards we will talk about some scenarios. + +If you run `nix flake show`, you should get an output similar to this +``` +$ nix flake show +git+file:///home/nerf/git/nixConfig?ref=refs%2fheads%2fnyarlathtop&rev=9d0eb749287d1e9e793811759dfa29469ab706dc +├───apps +│ └───x86_64-linux +├───checks +│ └───x86_64-linux +├───devShells +│ └───x86_64-linux +├───formatter +├───legacyPackages +│ └───x86_64-linux omitted (use '--legacy' to show) +├───nixosConfigurations +│ └───nyarlathotep: NixOS configuration +├───nixosModules +├───overlays +└───packages + └───x86_64-linux +``` +we can see there is an output called `nixosConfigurations.nyarlathotep` which contains the configuration of the machine +called nyarlathotep. `nixosConfigurations` is special in that sense, that `nixos-rebuild` will automatically look +for this key and assume how it is structured. The interesting part for us is the derivation `config.system.build.toplevel`. +Its closure contains the whole system and the resulting derivation a script that changes the current system to +that derivation. (called `/bin/switch-to-configuration`). + +So what we want to archive is populate the nix store of the target machine with the closure of the derivation +`.#nixosConfigurations..config.system.build.toplevel` and run the the resulting script on the target machine. + + ### Local -If you want to build the machineconfiguration for machine -run +It has multiple benefits to build the system configuration on the local computer and push it to the target server. +For example one doesn't stress the server with the load of evaluating the expression and building the closure. Also the server +doesn't need to fetch the build dependencies this way. One has a local check if at least the nix syntax was correct. +And so on... + +#### Build +If you have this repository local in your current directory, you can just run ``` -nix build .#nixosConfiguration..config.system.build.toplevel +$ nix build .#nixosConfigurations..config.system.build.toplevel ``` +to build the system configuration of the machine ``. + +But you don't need to clone this repository, for more see the `nix flake --help` documentation about flake urls. + +#### Copy +After we build the derivation we need to get the closure onto the target system. Luckily nix has tools to do that +via ssh. We could just run: +``` +$ nix copy -s --to .#nixosConfigurations..config.system.build.toplevel +``` +This will evaluate the flake again to get the store path of the given derivation. If we want to avoid this, +we might supply the corresponding store path directly. + +The `-s` is important: it makes the target machine substitute all derivations it can (by default from chache.nixos.org). +So you only upload configuration files and self build things. + +To be able to copy things to a machine they need to be signed by someone trusted. Additional trusted nix keys are handled +in `./nixos/roles/nix_keys.nix`. So to get yourself trusted you either need to install one derivation from the machine itself, +or find someone who is already trusted, to push your key. + +For more information on signing and key creation see `nix store sign --help` and `nix key --help`. + +#### Activate +Log into the remote machine and execute (with root privileges) +``` +# /nix/store//bin/switch-to-configuration boot +``` +That will setup a configuration switch at reboot. You can also switch the configuration live. For more +details consider the `--help` output of that script. The storepath (or at least the hash of the derivation) +is exactly the same it was on your machine. + + +If you have a `nixos-rebuild` available on your system, it can automatize these things with the `--flake` and +`--target-host` parameters. But there are some pitfalls so look at the `nixos-rebuild` documentation beforehand. ### On the machine -clone this repo to `/etc/nixos/` and `nixos-rebuild` that will select -the appropriate machine based on hostname + +Clone this repository to `/etc/nixos/` and `nixos-rebuild boot` or `nixos-rebuild switch` that will select +the appropriate machine based on hostname. + +If the hostname is not correct, or you don't want to clone this flake, you can also use the `--flake` parameter. + +In any case, to switch the system configuration you will need to have root privileges on the target machine. + +## Installing a new machine + +You have written a configuration and now want to deploy it as a new machine. You need to get the build configuration on the +`nixos-installer` machine (regarding this machine see issue [#10]). You can either use either any of the +versions above, or just continue then the machine will build the configuration implicitly. + +### Disk layout + +You will need to assemble the disk layout manually, we assume you do it below `/mnt` as the nixos-install tools +assume this as the default location (they have an option to change that consider their `--help` pages). + +This repository loads some default configuration that expects certain things. Your hardware configuration of that machine should +reflect those. + +- `"/"` is a tmpfs +- `"/persist"` is the place where we keep data that can not be regenerated at any boot, so this should be a permanent disk +- `"/nix"` the place the nixstore resides, needed to boot the machine should also be persistent +- `"/boot"` the place for bootloader configuration and kernel also persistent +- any additional data paths for your machine specific needs. Choose filesystems accordingly. + +My recommendation is to put `"/persist"` and `"/nix"` on a joint btrfs as subvolumes and `"/boot"` on separate disks (because grub +will give you a hard time if you do it as a subvolume or bind mount (even though that should be possible but is an upstream problem)). +For how to configure additional persistent data +to be stored in `"/persist"` look at the impermanence section as soon it is merged. Before this look at issue [#9]. +I do not recommend this for actual high access application data like databases mailboxes and things like it. You should +think about this as data that if lost can be regenerated with only little problems and read/written only a few times +during setup. (Like the server ssh keys for example). The configuration also setups some paths for `"/persist"` automatically, +again look at the impermanence sections. + +#### File system uuids + +You might end with a bit of a chicken/egg problem regarding filesystem uuids. See you need to set them in your system configuration. +There are two ways around that. Either generate the filesystems read out the uuids, and push them into the repository holding +the configuration you want to build, or generate the uuids first, have them in your configuration and set them upon filesystem creation. Most +`mkfs` utilities have an option for that. + +### Installing + +Just run +``` +nixos-install --flake 'git+https://gitea.mathebau.de/Fachschaft/nixConfig?ref=#' +``` +where `` is the branch you install from and `` is the name of the configuration you build. +If the build system is already in the nix store, this will start the installation, else it will first attempt to build +it. That should be the whole installation process, just reboot. The machine should be fully setup. No additional user +or service setup, after the reboot. -### sops +## How to write a new machine configuration +At best, you take a first look at already existing configurations. But here are a few guidelines. +Make a new folder in `/nixos/machines`. The name of the folder should match the hostname of your +machine. The only technically required file in there is `configuration.nix`. So create it. + +A good skeleton is probably: +``` +flake-inputs: +{config, pkgs, lib, ... }: { + +imports = [ + ./hardware-configuration.nix + ../../roles + ./network.nix + + + + ]; + + + networking.hostname = ""; # this will hopefully disappear if I have time to refactor this. + system.stateVersion = ""; +} +``` +The import of `../../roles` loads all the nice default setup that all these machines have in common. There the +impermanence configuration is loaded as well as ssh, sops, shared user configuration and much more. +The other two imports are suggestions how you should organize your configuration but not enforced by anything. +In your hardware +configuration you should basically only write you filesystem layout and your hostPlatform. The bootloading stuff +is already taken care of by `../../roles`. + +As of moment of writing `network.nix` should contain ip, nameserver and default gateway setup. As parts of +this is constant across all systems and will undergo refactor soon. + +I would recommend to split your configuration into small files you import. If this is something machine specific (like +tied to your ip address hostname), put it into the machine directory. If it is not, put it into `/nixos/roles/` if it +is not but has options to set, put it in `/nixos/modules`. + +## How this flake is organized + +This flake uses `flake-parts` see [flake.parts](https://flake.parts) for more details. It makes handling +`system` and some other modules related things more convenient. +For the general layout of nixos system configuration and modules, please see the corresponding documentation. + +The toplevel `flake.nix` contains the flake inputs as usual and only calls a file `flake-module.nix`. +This toplevel `flake-module.nix` imports further more specialized `flake-modules.nix` files from sub-directories. +Right now the only one is `nixos/flake-module.nix`. But if we start to ship our own software (or software versions, +with specific build flags), this might get more. + +### nixos +The `nixos` folder contains all machine configurations. It separates in two folders `nixos/machines` and `nixos/roles`. +The corresponding `flake-module.nix` file automatically searches for `machines//configuration.nix`, and evalutes +those as nixos configurations, and populates the flake. + +#### machines +`nixos/machines` contains all machine specific configuration (in a sub-folder per machine). Like hardware configuration, specific +network configuration. And service configuration that are too closely interwoven with the rest of that machine (for example +mailserver configuration depends heavily on network settings). It also +contains the root configuration for that machine called `configuration.nix`. This file usually only includes other modules. +These `configuration.nix` files are almost usual nix configurations. The only difference is that they take as an extra argument +the flake inputs. This allows them to load modules from these flakes. For example, nyarlathotep loads the simple-nixos-mailserver +module that way. + +#### roles +`nixos/roles` contains configuration that is potentially shared by some machines. It is expected that `nixos/roles/default.nix` +is imported as (`../../roles`) in every machine. Notable are the files `nixos/roles/admins.nix` which contains +common admin accounts for these machines and `nixos/roles/nix_keys.nix` which contains the additional trusted +keys for the nix store. + +## sops We are sharing secrets using [`sops`](https://github.com/getsops/sops) and [`sops-nix`](https://github.com/Mic92/sops-nix) As of right now we use only `age` keys. The machine keys are derived from their server ssh keys, that they generate at first boot. +To read out a machines public key run the following command on the corresponding machine. +``` +$ nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age' +``` User keys are generated by the users. -New keys and machines need entries into the `.sops.yaml` file within the root directory of this repo. +New keys and machines need entries into the `.sops.yaml` file within the root directory of this repository. -To make a secret available on a given machine you need to do the following. Configure the following keys +To make a secret available on a given machine you need to configure the following: ``` sops.secrets.example-key = { - sopsFile = "relative path to file in the repo containing the secrets (optional else the sops.defaultSopsFile is used) - path = "optinal path where the secret gets symlinked to, practical if some programm expects a specific path" - owner = user that owns the secret file: config.users.users.nerf.name (for example) - group = same as user just with groups: config.users.users.nerf.group - mode = "premission in usual octet: 0400 (for example)" + sopsFile = "relative path to file in the repo containing the secrets (optional else the sops.defaultSopsFile is used)"; + path = "optinal path where the secret gets symlinked to, practical if some program expects a specific path"; + owner = user that owns the secret file: config.users.users.nerf.name (for example); + group = same as user just with groups: config.users.users.nerf.group; + mode = "permission in usual octet: 0400 (for example)"; +}; ``` -afterwards the secret should be available in `/run/secrets/example-key`. +Afterwards the secret should be available in `/run/secrets/example-key`. If the accessing process is not root it must be member of the group `config.users.groups.keys` for systemd services this can be archived by setting `serviceConfig.SupplementaryGroups = [ config.users.groups.keys.name ];` -it the service config. +it the service configuration. +## impermanence + +These machines are setup with `"/"` as a tmpfs. This is there to keep the machines clean. So no clutter in home +directories, no weird ad-hoc solutions of botching something into `/opt/` or something like this. All will be +gone at reboot. + +But there are some files that we want to survive reboots, for example logs or ssh keys. The solution to this is +to have a persistent storage mounted at `/persist` and automatically bind mount the paths of persistent things +to the right places. To set this up we are using the impermanence module. In our configuration this is loaded with +some default files to bind mount (ssh keys, machine-id some nixos specific things). That we have on all machines. + +If you keep your application data (like recommended) on a separate partition, the chances are you don't need +to interact with this, as most configuration files will be in the nix store anyway. If the application wants these nix +store files in certain directories, you should use `environment.etc` family of options (consult the nixos documentation +for this). This is for mutable files that are not core application data. (Like ssh keys, for a mailserver one could +think about the hash files (not the db files) of an alias map (if one doesn't want to manage that with +the nix store), things like that). + +This should not be (but could be) used for large application databases. It would be more appropriate to mount +its own filesystem for things like that. For small configuration files that are not in the nix-store, +that might be the appropriate solution. + +By default the storage is called `persist` and the default path for it is `/persist`. These can be changed +with the `impermanence.name` and `impermanence.storagePath` options. To add paths to this storage you do the +following. + +``` +environment.persistence.${config.impermanence.name} = { + directories = [ + "" + ]; + files = [ + "" + ]; +}; +``` +For this to work `config` must be binded by the function arguments of you module. So the start of your module looks +something like this: +``` +{lib, pkgs, config, ...} : + +``` diff --git a/flake.lock b/flake.lock index e3f7e40..8cc319d 100644 --- a/flake.lock +++ b/flake.lock @@ -33,6 +33,21 @@ "type": "indirect" } }, + "impermanence": { + "locked": { + "lastModified": 1694622745, + "narHash": "sha256-z397+eDhKx9c2qNafL1xv75lC0Q4nOaFlhaU1TINqb8=", + "owner": "nix-community", + "repo": "impermanence", + "rev": "e9643d08d0d193a2e074a19d4d90c67a874d932e", + "type": "github" + }, + "original": { + "owner": "nix-community", + "repo": "impermanence", + "type": "github" + } + }, "nixos-mailserver": { "inputs": { "blobs": "blobs", @@ -123,6 +138,7 @@ "root": { "inputs": { "flake-parts": "flake-parts", + "impermanence": "impermanence", "nixos-mailserver": "nixos-mailserver", "nixpkgs": "nixpkgs", "sops-nix": "sops-nix" diff --git a/flake.nix b/flake.nix index 727dd91..0c61327 100644 --- a/flake.nix +++ b/flake.nix @@ -14,6 +14,9 @@ url = "github:Mic92/sops-nix"; inputs.nixpkgs.follows = "nixpkgs"; }; + impermanence = { + url = "github:nix-community/impermanence"; + }; }; outputs = inputs@{ flake-parts, ... }: diff --git a/nixos/flake-module.nix b/nixos/flake-module.nix index ef87e41..8c44964 100644 --- a/nixos/flake-module.nix +++ b/nixos/flake-module.nix @@ -12,6 +12,7 @@ imports = [ (import (./. + "/machines/${name}/configuration.nix") inputs) inputs.sops-nix.nixosModules.sops + inputs.impermanence.nixosModules.impermanence ]; }; in lib.genAttrs machines makeSystem); diff --git a/nixos/modules/impermanence.nix b/nixos/modules/impermanence.nix new file mode 100644 index 0000000..267c9d1 --- /dev/null +++ b/nixos/modules/impermanence.nix @@ -0,0 +1,47 @@ +{lib, config, ...} : + +let + inherit (lib) + mkEnableOption + mkIf + mkOption + types + ; + cfg = config.impermanence; +in + +{ +imports = [ ]; + +options.impermanence = { + enable = mkEnableOption "impermanence"; + storagePath = mkOption { + type = types.path; + default = "/persist"; + description = "The path where persistent data is stored"; + }; + name = mkOption { + type = types.str; + default = "persist"; + description = "the name of the persistent data store"; + }; +}; + +config = mkIf cfg.enable { + environment.persistence.${cfg.name} = { + persistentStoragePath = cfg.storagePath; + directories = [ + "/var/log" + "/var/lib/nixos" + ]; + files = [ + "/etc/ssh/ssh_host_ed25519_key" + "/etc/ssh/ssh_host_ed25519_key.pub" + "/etc/ssh/ssh_host_rsa_key" + "/etc/ssh/ssh_host_rsa_key.pub" + ]; + }; + environment.etc.machine-id.source = "${cfg.storagePath}/machine-id"; +}; + +} diff --git a/nixos/roles/admins.nix b/nixos/roles/admins.nix new file mode 100644 index 0000000..43a86aa --- /dev/null +++ b/nixos/roles/admins.nix @@ -0,0 +1,30 @@ +{lib, ...} : +with lib; + +let + admins = { + nerf = { + hashedPassword = + "$y$j9T$SJcjUIcs3JYuM5oyxfEQa/$tUBQT07FK4cb9xm.A6ZKVnFIPNOYMOKC6Dt6hadCuJ7"; + keys = [ + "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEdA4LpEGUUmN8esFyrNZXFb2GiBID9/S6zzhcnofQuP nerf@nerflap2" + ]; + }; + }; + + mkAdmin = name : + {hashedPassword, keys}: { + "${name}" = { + isNormalUser = true; + createHome = true; + extraGroups = [ "wheel" ]; + group = "users"; + home = "/home/${name}"; + openssh.authorizedKeys = { inherit keys; }; + inherit hashedPassword; + }; + }; + +in { + users.users = mkMerge (mapAttrsToList mkAdmin admins); +} diff --git a/nixos/roles/default.nix b/nixos/roles/default.nix index 3c24242..1968de3 100644 --- a/nixos/roles/default.nix +++ b/nixos/roles/default.nix @@ -1,4 +1,59 @@ -{ ... } : { +{pkgs, config, lib, modulesPath, ...} : { + +imports = [ + ./admins.nix + ./nix_keys.nix + (modulesPath + "/virtualisation/xen-domU.nix") + ../modules/impermanence.nix + ]; +nix = { + extraOptions = '' + experimental-features = nix-command flakes + builders-use-substitutes = true + ''; +}; + +networking = { + firewall = { # these shoud be default, but better make sure! + enable = true; + allowPing = true; + }; + nftables.enable = true; + useDHCP = false; # We don't speak DHCP and even if we would, we should enable it per interface + # hosts = # TODO write something to autogenerate ip adresses! +}; + +users = { + mutableUsers = false; + users.root.hashedPassword = "!"; +}; + +impermanence.enable = true; sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ]; + +environment = { + systemPackages = builtins.attrValues { + inherit (pkgs) + htop lsof tmux btop; + }; +}; + +services = { + journald.extraConfig = "SystemMaxUse=5G"; + + nginx = { + recommendedOptimisation = true; + recommendedGzipSettings = true; + recommendedTlsSettings = true; + }; + + openssh = { + enable = true; + settings = { + PermitRootLogin = "no"; + PasswordAuthentication = false; + }; + }; +}; } diff --git a/nixos/roles/nix_keys.nix b/nixos/roles/nix_keys.nix new file mode 100644 index 0000000..85c7835 --- /dev/null +++ b/nixos/roles/nix_keys.nix @@ -0,0 +1,6 @@ +{ + imports = [ ]; + nix.settings.trusted-public-keys = [ + "nerflap2-1:pDZCg0oo9PxNQxwVSQSvycw7WXTl53PGvVeZWvxuqJc=" + ]; +}