MASTER-NIX

The NixOS Ansible server that is replacing the server called MASTER.

Contents.

This document is going to detail the setup process so that if needed you should be able to recreate the server, but more importantly just so you can see how its working. You should Install NixOS like usual then use the Github repo to instal the flake and configration file. This will install the applications you need to follow the rest of this.

Im not providing all the cd and other basic commands that you need, you can figure that out.

Github - GH CLI

We are moving to a github way of doing things now as apart of the move to NixOS. Im going to make an ansible repository, this server will periodically pull down those files. Im planning for the NixOS configuration files and the Ansible playbooks to work like this, so lets setup access.

Github API token.

Login to Github as Itsuport and goto this page: Personal Access token, you should create a new token for each server you are doing, in this case we will remove the old dead token to keep it clean and make a new one. Fill out the appropriate details, give it access to the repository and then for permissions; Repository permissions > Contents > read-write is the only one you need to set. Minimal permsission is more secure. Set the exipry to be a year. The API Key needs refreshing every year.
Make a folder in the home directory i.e. mkdir /home/alto/GH in this folder we are going to put the auth token. nano ghapi and save just the token that Github gave you in there. Make sure theres nothing else in there.
Limiting access to the token by recursively setting the GH folder to only be readable/writeable/executable by our user. sudo chmod -R 700 GH
In the Flake.nix file you will see userSettings > gitHubPAT-File make sure you set this to the location you saved the file.
When you next run sudo nixos-rebuild switch --flake ./#MASTER-NIX the activation script I have made will log you in automatically. You can check the logs of the script in the return from that command or at /var/log/github-authentication.log.

You can find the script in the configuration.nix under system.activationScripts.authenticateGH if you want to read it.

It is very important that if you do a nix flake update or you modify these github files locally in anyway that you commit the changes or resolve the issues so the automated schedules can work properly, git pull will fail if you have local changes that are note saved. You can use the git status command in the repo directory to see what files have been changed etc.

Manual configuration (Optional)

If for some reason you want to do it manually here is what you need to do.

Logging into the GH CLI with our token cat ghapi | gh auth login --with-token, use gh auth status to check it worked. This token needs refreshing every year
. We then setup git with the auth details. gh auth setup-git this will log git in as our user with the API, so that you can git pull, commit, push etc. It is very important that if you do a nix flake update or you modify these github files locally in anyway that you commit the changes or resolve the issues so the automated schedules can work properly, git pull will fail if you have local changes that are note saved.

Setting up the System Service and Timer

We are using Systemd services && Systemd timers to schedule the github repo cloning. This will make sure our repos are upto date. I have written a module for cloning and updating (using git pull --rebase) the repo that you set. This module allows for dynamic and simple setting of the Repos and it will handle creating the systemd.service/.timer

Here is an example, you can add as many repos in the repositories section.

  services.githubClone = {
    enable = true;
    repositories = [
      {
        name = "Ansible";
          url = "github.com/${userSettings.gitHubUser}/ansible";
        destination = "/home/alto/Ansible";
        user = "${userSettings.gitHubUser}";
        schedule = "*-*-* 16:00:00";  # Run at 4pm every day.
        token = "${userSettings.gitHubPAT-File}";
      }
    ];
  };

"name": should be what you want to call the system service
"url": is the url of the repo.
"destination": is the file directory you want to clone the repo to, make sure this does not exist, so that the repo is cloned and updated properly.
"user": this is the github username, not server username.
"schedule": use anything that support the systemd OnCalendar format, which you can find online.
"token": this should be the full file path to the github token.

We will change the timer if needed, however 1Hr 30mins should be fine.
We need to do the same for any other repos as well.

Checking service logs

So if for some reason there is issues with the services/timers, you can use these commands to find more info. Of course change the name for the service your troubleshooting.

Manually run the service and check output:

systemctl start updateFlakeRepo.service && systemctl status updateFlakeRepo.service

Check the timer:

systemctl list-timers | grep updateFlakeRepo

Check the service status:

systemctl status updateFlakeRepo.service

View the logs:

journalctl -u updateFlakeRepo.service

Changing the user password.

In order to change the user password you need to change the hashed password in the flake.nix file.
Use this command on linux to generate a has for the password you want: mkpasswd --method=sha-512. Copy the output exactly and paste it into the hasedPassword = section in flake.nix.

When you change the password by doing this make sure that you check the git command can operate properly, you may need to run gh auth setup-git again.

Rebuild the system.

SSH setup

Ansible is accessing the servers using SSH, this is great and secure. We dont have to setup every server for access again since we will be copying the .ssh folder entirely from the old MASTER server our new one. Then our new server will have access to all the hosts using SSH keys just like the old server.

Copy the ~/.ssh folder from the root users home directory on the old server to the home/alto/.ssh folder on the new server. Use SCP.
Set the permissions/owners of the SSH config files; sudo chown -R alto:users ./.ssh && sudo chmod -R 700 ./.ssh If you dont have the permissions correct then SSH will fail authentication
Check this is working by SShing into one of the servers managaged by ansible. You should be able to login without and password prompt. If this fails you need to check the SSH auth logs on both servers while trying to find the issue.

Ansible

Playbooks will be managed from Github and synced using a systemd timer like above. I would like to look into using SemaphoreUI in the future, but for now we will stick to the old way of doing things.

I have written a Nix module that will automate the first clone of the repository and then handle keeping it upto date aswell. Here is the configuration for Downloading the Ansible repo. It shows the Flake repo and the Ansible repo configuration.

  services.githubClone = {
    enable = true;
    repositories = [
      {
        name = "MASTER-NIX";
        url = "github.com/${userSettings.gitHubUser}/MASTER-NIX";
        destiantion = "home/${userSettings.userName}/Flake";
        user = "${userSettings.gitHubUser}";
        schedule = "*-*-* 13:00:00";
        toekn = "${userSettings.gitHunPAT-File}";
      }
      {
        name = "Ansible";
        url = "github.com/${userSettings.gitHubUser}/ansible";
        destination = "/home/alto/Ansible";
        user = "${userSettings.gitHubUser}";
        schedule = "*-*-* 16:00:00";  # Run at 4pm every day.
        token = "${userSettings.gitHubPAT-File}";
      }
    ];

Repositories you put in here will be cloned if they dont exist or updated if the do exist. Make sure the folder you point it at either doesn't exist or is a valid git repo. It will not work otherwise.

Manual way (Optional but usefull)

This is usefull for getting the Flake and other config files onto the system for the first time. Then once you run the rebuild commands it will do the rest.

Clone the Github repo into the home directory gh repo clone ansible ./Ansible.
The configuration.nix has a Systemd timer for pulling this repo so check that you have the files in the spot that it expects.
We are going to put the Ansible-vault key in the GH directory with our other keys, as this folder is read only.
Then everytime we run a playbook we parse the key into the ansible commands. I will setup a systemd timer for each playbok that we run.
Modify the playbooks on github, not locally. If they get out of syc the automatic tasks could fail.

Automating the playbooks

Just like above we are going to automate the playbooks using sytstemd services & timers.
Below is a snip from the systemd-services.nix that shows how they should be layed out.

You can see an example of the Nix-rebuild timer and service. And the Ansible reboot-playbook timer and service. You should create a service and timer for each playbook you want to schedule.

You can make a service that doesnt have a timer if you are going to start it manually or through another program, but I will be setting these up like this.

You can view system timers with systemctl list-timers and services with systemctl list-units.

{
  systemd.services.nixosRebuild = {
    description = "Rebuild NixOS Configuration";
    after = [ "network-online.target" ];
    wants = [ "network-online.target" ];
    serviceConfig = {
      Type = "oneshot";
      ExecStart = "${pkgs.nix}/bin/nixos-rebuild switch --flake ./#${systemSettings.hostname}";
      User = userSettings.username;
      WorkingDirectory = "/home/${userSettings.username}/Flake";
    };
  };
  systemd.timers.nixosRebuild = {
    description = "Timer for nixosRebuild.service";
    wantedBy = [ "timers.target" ];
    timerConfig = {
      OnCalendar = "16:30";
      Persistent = true;
    };
  };

  systemd.services.ansibleRebootMachines = {
    description = "Reboot all the servers, daily.";
    after = [ "network-online.target" ];
    wants = [ "network-online.target" ];
    serviceConfig = {
      Type = "oneshot";
      ExecStart = "${pkgs.ansible}/bin/ansible-playbook ./main.yml --inventory ../inventory --vault-password-file /home/${userSettings.username}/GH/vault.key";
      User = userSettings.username;
      WorkingDirectory = "/home/${userSettings.username}/Ansible/${userSettings.username}/rebootmachines";
      StandardOutput = "append:/home/${userSettings.username}/Ansible/${userSettings.username}/rebootmachines/patch.log";
      StandardError = "append:/home/${userSettings.username}/Ansible/${userSettings.username}/rebootmachines/patch.log";
      Environment = [
        "HOME=/home/${userSettings.username}"
        "PATH=${pkgs.openssh}/bin:${pkgs.ansible}/bin:${pkgs.coreutils}/bin:${pkgs.python3}/bin:/run/current-system/sw/bin"
      ];
    };
  };
  systemd.timers.ansibleRebootMachines = {
    description = "Timer for ansibleRebootMachines.service";
    wantedBy = [ "timers.target" ];
    timerConfig = {
      OnCalendar = "04:30";
      Persistent = true;
    };
  };
}

Finished.

At this point you should have a fully functioning MASTER-NIX Ansible server. Re-read this multiple times if you need to, as Nix is convoluted until you get a grasp of it.