Rework of the nix (the package manager) config #45

nerf · 2024-10-13T12:09:38Z

nerf commented

2024-10-13 12:09:38 +00:00

This should fix [#44], by running the garbage collector weekly deleting all old generations, it also configures some other things like storage optimization through hard linking.
This might slow down nix store operations when building, but the performance critical machines shouldn't build anything anyway.

It sets some options that are irrelevant right now, but would be better like this than the default if they should become relevant. Like fallback and builders-use-substitutes.
If there is anything missing that we should set let me know.

We might think about getting all user config in one place, aka getting the trusted keys into the admin.nix file. Not sure though if this is a great idea, any comments?

This should fix [#44], by running the garbage collector weekly deleting all old generations, it also configures some other things like storage optimization through hard linking. This might slow down nix store operations when building, but the performance critical machines shouldn't build anything anyway. It sets some options that are irrelevant right now, but would be better like this than the default if they should become relevant. Like `fallback` and `builders-use-substitutes`. If there is anything missing that we should set let me know. We might think about getting all user config in one place, aka getting the trusted keys into the admin.nix file. Not sure though if this is a great idea, any comments?

nerf added the

Kind/Bug

Priority

High

labels 2024-10-13 12:09:38 +00:00

nerf added 1 commit 2024-10-13 12:09:39 +00:00

nix config to save some space 5f1f973f72

Gonne requested changes 2024-10-13 12:34:58 +00:00

Dismissed

Gonne left a comment

Thanks for taking this up.

nixos/roles/nix.nix Outdated

					
				@ -0,0 +1,25 @@

				{

				  nix.settings = {

				    trusted-public-keys = [

Gonne commented

2024-10-13 12:34:29 +00:00

While part of nix.settings I would prefer to move these to admin.nix and leave a reference comment here. If we start our own builder we may start to trust it here.

While part of `nix.settings` I would prefer to move these to `admin.nix` and leave a reference comment here. If we start our own builder we may start to trust it here.

nerf commented

2024-10-13 13:16:43 +00:00

Yes let me think how to do this later

Gonne marked this conversation as resolved

nixos/roles/nix.nix Outdated

					
				@ -0,0 +10,4 @@

				    ];

				    auto-optimise-store = true;

				    fallback = true;

				    builders-use-substitus = true;

Gonne commented

2024-10-13 12:21:23 +00:00

typo

nerf commented

2024-10-13 13:11:47 +00:00

And i messed up the grouping completely, dafuq

Gonne marked this conversation as resolved

nixos/roles/nix.nix Outdated

					
				@ -0,0 +13,4 @@

				    builders-use-substitus = true;

				  };

				  gc = {

				    persistent = true;

Gonne commented

2024-10-13 12:23:30 +00:00

Please comment on why this is a good idea on / as tmpfs. Feels like it adds a garbage collection right after every restart compared to missed ones on downtime. I have no idea which we prefer.

Please comment on why this is a good idea on `/` as `tmpfs`. Feels like it adds a garbage collection right after every restart compared to missed ones on downtime. I have no idea which we prefer.

nerf commented

2024-10-13 13:05:13 +00:00

I don't think this is important. On true it will catch up missed runs when the machine was down, on false it will just start the timer
on boot. So if false we could run into the problem that we never garbage collect if we restarted a machine more then once per week.

And yes I didn't thought about where systemd stored this and it would need some impermanence, sigh

I don't think this is important. On true it will catch up missed runs when the machine was down, on false it will just start the timer on boot. So if false we could run into the problem that we never garbage collect if we restarted a machine more then once per week. And yes I didn't thought about where systemd stored this and it would need some impermanence, sigh

nerf commented

2024-10-13 13:30:44 +00:00

note for myself it is /var/lib/systemd/timers

note for myself it is `/var/lib/systemd/timers`

nerf commented

2024-10-14 10:28:16 +00:00

So de we want more persistency things, or risking to never garbage collect on machines that restart often.
The upside of this behavior would be that you never boot a new generation and run into a garbage collection that is scheduled
directly after. The more I think it is more sensible to have a week from last reboot and not from last garbage collect.

So de we want more persistency things, or risking to never garbage collect on machines that restart often. The upside of this behavior would be that you never boot a new generation and run into a garbage collection that is scheduled directly after. The more I think it is more sensible to have a week from last reboot and not from last garbage collect.

👍 1

Gonne marked this conversation as resolved

nixos/roles/nix.nix Outdated

					
				@ -0,0 +19,4 @@

				    randomizedDelaySec = "5h";

				  };

				  optimise = {

				    dates = [];

Gonne commented

2024-10-13 12:27:35 +00:00

optimise = true, but never? Is that how the empty dates array works?

`optimise = true`, but never? Is that how the empty `dates` array works?

nerf commented

2024-10-13 13:07:10 +00:00

Yes, line 11 makes it optimise on store operations, but if i understood it correctly only if this option is true. Date can be used to make regular optimization runs. But this setup seemed to be more sensible to me as we don't have regular store operations. So we can
just do that when we update.

Yes, line 11 makes it optimise on store operations, but if i understood it correctly only if this option is true. Date can be used to make regular optimization runs. But this setup seemed to be more sensible to me as we don't have regular store operations. So we can just do that when we update.

Gonne commented

2024-10-13 19:33:40 +00:00

The option auto-optimise-store is applied when something is built, but for me it feels like this setting needs to be set on the building system (e.g. my private config) rather than on our VMs. Furthermore it may slow down builds as well as downloads from remote builders significantly (Github comment).
The nix.optimise however configures a systemd timer that is run on the given dates. Nothing in its definition file gives me the impression that this interacts with auto-optimise-store. I think that this is what we actually want on our VMs.
Setting the dates empty gives an erroneous systemd-timer on my system.

The option `auto-optimise-store` is applied when something is built, but for me it feels like this setting needs to be set on the building system (e.g. my private config) rather than on our VMs. Furthermore it may slow down builds as well as downloads from remote builders significantly ([Github comment]( https://github.com/NixOS/nix/issues/6033#issuecomment-1615856506)). The `nix.optimise` however configures a `systemd` timer that is run on the given dates. Nothing in its [definition file](https://github.com/NixOS/nixpkgs/blob/nixos-24.05/nixos/modules/services/misc/nix-optimise.nix) gives me the impression that this interacts with `auto-optimise-store`. I think that this is what we actually want on our VMs. Setting the `dates` empty gives an erroneous systemd-timer on my system.

nerf commented

2024-10-14 10:25:55 +00:00

Hmm, i don't know how the nix store copy exactly works, my assumption was kind of that the target machine would optimize, when I copy to it. It will also substitute everything it can from cache, so local storage optimizations of the building machine shouldn't play a role. If it would then the cache would need this option.

For the second part I missread the documentation then, will fix that.

Hmm, i don't know how the nix store copy exactly works, my assumption was kind of that the target machine would optimize, when I copy to it. It will also substitute everything it can from cache, so local storage optimizations of the building machine shouldn't play a role. If it would then the cache would need this option. For the second part I missread the documentation then, will fix that.

👍 1

Gonne marked this conversation as resolved

nerf commented

2024-10-13 12:41:51 +00:00

If we are strict about this, merging would incur a redeployment of all nix machines, but I think we don't have to do this and just let this
change creep in with the natural next builds.

If we are strict about this, merging would incur a redeployment of all nix machines, but I think we don't have to do this and just let this change creep in with the natural next builds.

👍 1

nerf force-pushed nix-config from 5f1f973f72 to b2e5d28f2b

2024-10-13 13:12:51 +00:00

Compare

nerf changed title from ~~nix config to save some space~~ to WIP: nix config to save some space

2024-10-13 13:16:53 +00:00

nerf force-pushed nix-config from b2e5d28f2b to 372a42a9ed

2024-10-13 16:59:51 +00:00

Compare

nerf changed title from ~~WIP: nix config to save some space~~ to nix config to save some space

2024-10-13 16:59:59 +00:00

nerf commented

2024-10-13 17:00:37 +00:00

I changed the admin handling, please check extra carefully. I don't want to brick all logins

nerf reviewed 2024-10-13 17:02:30 +00:00

nixos/roles/admins.nix Outdated

					
				@ -32,2 +39,4 @@

				  mkNixKeys = _: {nixKeys, ...}: nixKeys;

				in {

				  users.users = mkMerge (mapAttrsToList mkAdmin admins);

				  nix.settings.trusted-public-keys = lists.concatLists (mapAttrsToList mkNixKeys admins);

nerf commented

2024-10-13 17:02:30 +00:00

If someone knows a solution without mapAttrsToList let me know. It feels a bit like overkill. (esp. as we don't need the username here).

If someone knows a solution without `mapAttrsToList` let me know. It feels a bit like overkill. (esp. as we don't need the username here).

nerf changed title from ~~nix config to save some space~~ to Rework of the nix (the package manager) config

2024-10-13 17:06:52 +00:00

nerf force-pushed nix-config from 372a42a9ed to 83b7262090

2024-10-14 10:31:24 +00:00

Compare

Gonne requested changes 2024-10-14 16:38:26 +00:00

Dismissed

nixos/roles/nix.nix Outdated

					
				@ -0,0 +1,20 @@

				{

				  nix = {

				    settings = {

Gonne commented

2024-10-14 16:34:50 +00:00

Please add a comment here that nix.settings.trusted-public-keys is also (currently only) set in admin.nix.

Please add a comment here that `nix.settings.trusted-public-keys` is also (currently only) set in `admin.nix`.

Gonne marked this conversation as resolved

nixos/roles/nix.nix Outdated

					
				@ -0,0 +10,4 @@

				      fallback = true;

				      builders-use-substitutes = true;

				    };

				    gc = {

Gonne commented

2024-10-14 16:32:24 +00:00

Please also set nix.gc.automatic = true.

Please also set `nix.gc.automatic = true`.

Gonne marked this conversation as resolved

nerf force-pushed nix-config from 83b7262090 to ec46a28278

2024-10-15 11:21:09 +00:00

Compare

Gonne approved these changes 2024-10-15 11:30:48 +00:00

nerf merged commit ec46a28278 into main

2024-10-16 15:06:08 +00:00

nerf deleted branch nix-config

2024-10-16 15:06:09 +00:00

Gonne commented

2024-10-16 17:04:59 +00:00

Login on ghatanothoa still works. Lets see if garbage collection works. Currently, we have

[gonne@ghatanothoa:~]$ df -h 
/dev/xvda2       10G  9.2G  224M  98% /nix
/dev/xvda2       10G  9.2G  224M  98% /persist

Login on ghatanothoa still works. Lets see if garbage collection works. Currently, we have ``` [gonne@ghatanothoa:~]$ df -h /dev/xvda2 10G 9.2G 224M 98% /nix /dev/xvda2 10G 9.2G 224M 98% /persist ```

Gonne commented

2024-10-21 04:49:45 +00:00

It deleted 560MB according to logs this night.

Sign in to join this conversation.

No reviewers