From de861a2ee07011114f6434616ccdfdc64a14292e Mon Sep 17 00:00:00 2001 From: Daniel Barlow Date: Sun, 6 Apr 2025 18:16:53 +0100 Subject: [PATCH] convert rst to asciidoc for i in *.rst ; do pandoc -f rst -t asciidoc -o `basename $i .rst`.adoc $i ;done --- doc/admin.adoc | 352 ++++++++++++++++++++++ doc/admin.rst | 371 ----------------------- doc/configuration.adoc | 470 +++++++++++++++++++++++++++++ doc/configuration.rst | 482 ------------------------------ doc/development.adoc | 311 +++++++++++++++++++ doc/development.rst | 340 --------------------- doc/index.adoc | 10 + doc/index.rst | 24 -- doc/installation.adoc | 203 +++++++++++++ doc/installation.rst | 211 ------------- doc/intro.adoc | 14 + doc/intro.rst | 15 - doc/modules.adoc | 1 + doc/modules.rst | 4 - doc/{outputs.rst => outputs.adoc} | 15 +- doc/tutorial.adoc | 322 ++++++++++++++++++++ doc/tutorial.rst | 327 -------------------- 17 files changed, 1689 insertions(+), 1783 deletions(-) create mode 100644 doc/admin.adoc delete mode 100644 doc/admin.rst create mode 100644 doc/configuration.adoc delete mode 100644 doc/configuration.rst create mode 100644 doc/development.adoc delete mode 100644 doc/development.rst create mode 100644 doc/index.adoc delete mode 100644 doc/index.rst create mode 100644 doc/installation.adoc delete mode 100644 doc/installation.rst create mode 100644 doc/intro.adoc delete mode 100644 doc/intro.rst create mode 100644 doc/modules.adoc delete mode 100644 doc/modules.rst rename doc/{outputs.rst => outputs.adoc} (58%) create mode 100644 doc/tutorial.adoc delete mode 100644 doc/tutorial.rst diff --git a/doc/admin.adoc b/doc/admin.adoc new file mode 100644 index 0000000..97b4705 --- /dev/null +++ b/doc/admin.adoc @@ -0,0 +1,352 @@ +== System Administration + +=== Services on a running system + +Liminix services are built on s6-rc, which is itself layered on s6. +Services are defined at build time in your configuration (see +`+configuration-services+` for information) and can't be added +to/changed at runtime, but to monitor events or diagnose problems you +may need to inspect them on the running system. Here are some of the +most commonly used s6,-rc commands: + +.Service management quick reference +[width="100%",cols="55%,45%",options="header",] +|=== +|What |How +|List all running services |`+s6-rc -a list+` + +|List all services that are *not* running |`+s6-rc -da list+` + +|List services that `+wombat+` depends on +|`+s6-rc-db dependencies wombat+` + +|... transitively |`+s6-rc-db all-dependencies wombat+` + +|List services that depend on service `+wombat+` +|`+s6-rc-db -d dependencies wombat+` + +|... transitively |`+s6-rc-db -d all-dependencies wombat+` + +|Stop service `+wombat+` and everything depending on it +|`+s6-rc -d change wombat+` + +|Start service `+wombat+` (but not any services depending on it) +|`+s6-rc -u change wombat+` + +|Start service `+wombat+` and all* services depending on it +|`+s6-rc-up-tree wombat+` +|=== + +`+s6-rc-up-tree+` brings up a service and all services that depend on +it, except for any services that depend on a "controlled" service that +is not currently running. Controlled services are not started at boot +time but in response to external events (e.g. plugging in a particular +piece of hardware) so you probably don't want to be starting them by +hand if the conditions aren't there. + +A service may be *up* or *down* (there are no intermediate states like +"started" or "stopping" or "dying" or "cogitating"). Some (but not all) +services have "readiness" notifications: the dependents of a service +with a readiness notification won't be started until the service signals +(by writing to a nominated file descriptor) that it's prepared to start +work. Most services defined by Liminix also have a `+timeout-up+` +parameter, which means that if a service has readiness notifications and +doesn't become ready in the allotted time (defaults 20 seconds) it will +be terminated and its state set to *down*. + +If the process providing a service dies, it will be restarted +automatically. Liminix does not automatically set it to *down*. + +(If the process providing a service dies without ever notifying +readiness, Liminix will restart it as many times as it has to until the +timeout period elapses, and then stop it and mark it down.) + +==== Controlled services + +*Controlled* services are those which are started/stopped on demand by a +*controller* (another service) instead of being started at boot time. +For example: + +* `+svc.uevent-rule.build+` creates a controlled service which is active +when a particular hardware device (identified by uevent/sysfs directory) +is present. +* `+svc.round-robin.build+` creates a service controller that invokes +two or more services in turn, running the next one when the process +providing the previous one exits. We use this for failover from one +network connection to a backup connection, for example. +* `+svc.health-check.build+` creates a service controller that runs a +controlled service and periodically tests whether it is healthy by +running an external health check command or script. If the check command +repeatedly fails, the controlled service is restarted. ++ +The Configuration section of the manual describes controlled services in +more detail. Some operational considerations +* `+round-robin+` detects a service status by looking at its `+outputs+` +directory, so it won't work unless the service creates some outputs. +This is considered a bug and will be fixed in a future release +* `+health-check+` works for longruns but not for oneshots, as it +internally relies on `+s6-svc+` to restart the process + +==== Logs + +Logs for all services are collated into `+/run/log/current+`. The log +file is rotated when it reaches a threshold size, into another file in +the same directory whose name contains a TAI64 timestamp. + +Each log line is prefixed with a TAI64 timestamp and the name of the +service, if it is a longrun. If it is a oneshot, a timestamp and the +name of some other service. To convert the timestamp into a +human-readable format, use `+s6-tai64nlocal+`. + +[source,console] +---- +# ls -l /run/log/ +-rw-r--r-- 1 0 lock +-rw-r--r-- 1 0 state +-rwxr--r-- 1 98059 @4000000000025cb629c311ac.s +-rwxr--r-- 1 98061 @40000000000260f7309c7fb4.s +-rwxr--r-- 1 98041 @40000000000265233a6cc0b6.s +-rwxr--r-- 1 98019 @400000000002695d10c06929.s +-rwxr--r-- 1 98064 @4000000000026d84189559e0.s +-rwxr--r-- 1 98055 @40000000000271ce1e031d91.s +-rwxr--r-- 1 98054 @400000000002760229733626.s +-rwxr--r-- 1 98104 @4000000000027a2e3b6f4e12.s +-rwxr--r-- 1 98023 @4000000000027e6f0ed24a6c.s +-rw-r--r-- 1 42374 current + +# tail -2 /run/log/current +@40000000000284f130747343 wan.link.pppoe Connect: ppp0 <--> /dev/pts/0 +@40000000000284f230acc669 wan.link.pppoe sent [LCP ConfReq id=0x1 ] +# tail -2 /run/log/current | s6-tai64nlocal +1970-01-02 21:51:45.828598156 wan.link.pppoe sent [LCP ConfReq id=0x1 ] +1970-01-02 21:51:48.832588765 wan.link.pppoe sent [LCP ConfReq id=0x1 ] +---- + +===== Log persistence + +Logs written to `+/run/log/+` will not survive a reboot or crash, as it +is an ephemeral filesystem. + +On supported hardware you can enable logging to +https://www.kernel.org/doc/Documentation/ABI/testing/pstore[pstore] +which means the most recent log messages will be preserved on reboot. +Set the config option `+logging.persistent.enable = true+` to log +messages to `+/dev/pmsg0+` as well as to the regular log. This is a +circular buffer, so when it fills up newer messages will overwrite the +oldest messages. + +Logs found in pstore after a reboot will be moved at startup to +`+/run/log/previous-boot+` + +=== Updating an installed system + +If your system has a writable root filesystem (JFFS2, btrfs etc +-anything but squashfs), we have mechanisms for in-places updates +analogous to `+nixos-rebuild+`, but the operation is a bit different +because it expects to run on a build machine and then copy to the host +device using `+ssh+`. + +To use this, build the `+outputs.updater+` target and then run the +`+update.sh+` script it generates. + +[source,console] +---- +nix-build -I liminix-config=./my-configuration.nix \ + --arg device "import ./devices/mydevice" \ + -A outputs.updater +./result/bin/update.sh root@the-device +---- + +The update script uses min-copy-closure to copy new or changed packages +to the device, then (perhaps) reboots it. The reboot behaviour can be +affected by flags: + +* [.title-ref]#--no-reboot# will cause it not to reboot at all, if you +would rather do that yourself. Note that none of the newly-installed or +updated services will be running until you do. +* [.title-ref]#--fast# causes it tn not do a full reboot, but instead to +restart only the services that have been changed. This will restart all +of the services that have updated store paths (and anything that depends +on them), but will not affect services that haven't changed. + +It doesn't delete old packages automatically: to do that run +`+min-collect-garbage+`, which will delete any packages not in the +current system closure. Note that Liminix does not have the NixOS +concept of environments or generations, and there is no way back from +this except for building the previous configuration again. + +==== Caveats + +* it needs there to be enough free space on the device for all the new +packages in addition to all the packages already on it - which may be a +problem if there is little flash storage or if a lot of things have +changed (e.g. a new version of nixpkgs). +* it may not be able to upgrade the kernel: this is device-dependent. If +your device boots from a kernel image on a raw MTD partition or or UBI +volume, update.sh is unable to alter the kernel partition. If your +device boots from a kernel inside the filesystem (e.g. using +bootloader.extlinux or bootloder.fit) then the kernel will be upgraded +along with the userland + +==== Recovery/downgrades + +The `+update.sh+` script also creates a timestamped symlink on the +device which points to the system configuration it installs. If you +install a configuration that doesn't work, you can revert to any other +installed configuration by + +[arabic] +. booting to some kind of rescue or recovery system (which may be some +vendor-provided rescue option, or your own recovery system perhaps based +on `+examples/recovery.nix+`) and mounting your Liminix filesystem on +/mnt +. picking another previously-installed configuration that _link:[did] +work, and switching back to it: + +[source,console] +---- +# ls -ld /mnt/*configuration +lrwxrwxrwx 1 90 /mnt/20252102T182104.configuration -> nix/store/v1w0h4zw65ah4c2r0k7nyy125qrxhq78-system-configuration-aarch64-unknown-linux-musl +lrwxrwxrwx 1 90 /mnt/20251802T181822.configuration -> nix/store/wqjl9s9xljl2wg8257292zghws9ssidk-system-configuration-aarch64-unknown-linux-musl +# : 20251802T181822 is the working system, so reinstall it +# /mnt/20251802T181822.configuration/bin/install /mnt +# umount /mnt +# reboot +---- + +This will install the previous configuration's activation binary into +/bin, and copy its kernel and initramfs into /boot. Note that it depends +on the previous system not having been garbage-collected. + +[[levitate]] +==== Adding packages + +If you simply wish to add a package without any change to services, you +can call `+min-copy-closure+` directly to install any package in Nixpkgs +or in the Liminix overlay + +[source,console] +---- +nix-build -I liminix-config=./my-configuration.nix \ + --arg device "import ./devices/mydevice" -A pkgs.tcpdump + +nix-shell -p min-copy-closure root@the-device result/ +---- + +Note that this only copies the package and its dependencies to the +device: it doesn't update any profile to add it to `+$PATH+` + +[[rebuilding the system]] +=== Reinstalling on a running system + +Liminix is initially installed from a monolithic `+firmware.bin+` - and +unless you're running a writable filesystem, the only way to update it +is to build and install a whole new `+firmware.bin+`. However, you +probably would prefer not to have to remove it from its installation +site, unplug it from the network and stick serial cables in it all over +again. + +It is not (generally) safe to install a new firmware onto the flash +partitions that the active system is running on. To address this we have +`+levitate+`, which a way for a running Liminix system to "soft restart" +into a ramdisk running only a limited set of services, so that the main +partitions can then be safely flashed. + +==== Configuration + +Levitate _needs to be configured when you create the initial system_ to +specify which services/packages/etc to run in maintenance mode. Most +likely you want to configure a network interface and an ssh for example +so that you can login to reflash it. + +[source,nix] +---- +defaultProfile.packages = with pkgs; [ + ... + (levitate.override { + config = { + services = { + inherit (config.services) dhcpc sshd watchdog; + }; + defaultProfile.packages = [ mtdutils ]; + users.root = config.users.root; + }; + }) +]; +---- + +==== Use + +Connect (with ssh, probably) to the running Liminix system that you wish +to upgrade. + +[source,console] +---- +bash$ ssh root@the-device +---- + +Run `+levitate+`. This takes a little while (perhaps a few tens of +seconds) to execute, and copies all config required for maintenance mode +to `+/run/maintenance+`. + +[source,console] +---- +# levitate +---- + +Reboot into maintenance mode. You will be logged out + +[source,console] +---- +# reboot +---- + +Connect to the device again - note that the ssh host key will have +changed. + +[source,console] +---- +# ssh -o UserKnownHostsFile=/dev/null root@the-device +---- + +Check we're in maintenance mode + +[source,console] +---- +# cat /etc/banner + +LADIES AND GENTLEMEN WE ARE FLOATING IN SPACE + +Most services are disabled. The system is operating +with a ram-based root filesystem, making it safe to +overwrite the flash devices in order to perform +upgrades and maintenance. + +Don't forget to reboot when you have finished. +---- + +Perform the upgrade, using flashcp. This is an example, your device will +differ + +[source,console] +---- +# cat /proc/mtd +dev: size erasesize name +mtd0: 00030000 00010000 "u-boot" +mtd1: 00010000 00010000 "u-boot-env" +mtd2: 00010000 00010000 "factory" +mtd3: 00f80000 00010000 "firmware" +mtd4: 00220000 00010000 "kernel" +mtd5: 00d60000 00010000 "rootfs" +mtd6: 00010000 00010000 "art" +# flashcp -v firmware.bin mtd:firmware +---- + +All done + +[source,console] +---- +# reboot +---- diff --git a/doc/admin.rst b/doc/admin.rst deleted file mode 100644 index cddf70d..0000000 --- a/doc/admin.rst +++ /dev/null @@ -1,371 +0,0 @@ -System Administration -##################### - -Services on a running system -**************************** - -Liminix services are built on s6-rc, which is itself layered on s6. -Services are defined at build time in your configuration (see -:ref:`configuration-services` for information) and can't be added -to/changed at runtime, but to monitor -events or diagnose problems you may need to inspect them on the -running system. Here are some of the most commonly used s6,-rc -commands: - -.. list-table:: Service management quick reference - :widths: 55 45 - :header-rows: 1 - - * - What - - How - * - List all running services - - ``s6-rc -a list`` - * - List all services that are **not** running - - ``s6-rc -da list`` - * - List services that ``wombat`` depends on - - ``s6-rc-db dependencies wombat`` - * - ... transitively - - ``s6-rc-db all-dependencies wombat`` - * - List services that depend on service ``wombat`` - - ``s6-rc-db -d dependencies wombat`` - * - ... transitively - - ``s6-rc-db -d all-dependencies wombat`` - * - Stop service ``wombat`` and everything depending on it - - ``s6-rc -d change wombat`` - * - Start service ``wombat`` (but not any services depending on it) - - ``s6-rc -u change wombat`` - * - Start service ``wombat`` and all* services depending on it - - ``s6-rc-up-tree wombat`` - -:command:`s6-rc-up-tree` brings up a service and all services that -depend on it, except for any services that depend on a "controlled" -service that is not currently running. Controlled services are not -started at boot time but in response to external events (e.g. plugging -in a particular piece of hardware) so you probably don't want to be -starting them by hand if the conditions aren't there. - -A service may be **up** or **down** (there are no intermediate states -like "started" or "stopping" or "dying" or "cogitating"). Some (but -not all) services have "readiness" notifications: the dependents of a -service with a readiness notification won't be started until the -service signals (by writing to a nominated file descriptor) that it's -prepared to start work. Most services defined by Liminix also have a -``timeout-up`` parameter, which means that if a service has readiness -notifications and doesn't become ready in the allotted time (defaults -20 seconds) it will be terminated and its state set to **down**. - -If the process providing a service dies, it will be restarted -automatically. Liminix does not automatically set it to **down**. - -(If the process providing a service dies without ever notifying -readiness, Liminix will restart it as many times as it has to until the -timeout period elapses, and then stop it and mark it down.) - -Controlled services -=================== - -**Controlled** services are those which are started/stopped on demand -by a **controller** (another service) instead of being started at boot -time. For example: - -* ``svc.uevent-rule.build`` creates a controlled service which is - active when a particular hardware device (identified by uevent/sysfs - directory) is present. - -* ``svc.round-robin.build`` creates a service controller that - invokes two or more services in turn, running the next one when the - process providing the previous one exits. We use this for failover - from one network connection to a backup connection, for example. - -* ``svc.health-check.build`` creates a service controller that - runs a controlled service and periodically tests whether it is - healthy by running an external health check command or script. If the - check command repeatedly fails, the controlled service is - restarted. - - The Configuration section of the manual describes controlled - services in more detail. Some operational considerations - -* ``round-robin`` detects a service status by looking at its - :file:`outputs` directory, so it won't work unless the service - creates some outputs. This is considered a bug and will be - fixed in a future release - -* ``health-check`` works for longruns but not for oneshots, as it - internally relies on ``s6-svc`` to restart the process - -Logs -==== - -Logs for all services are collated into :file:`/run/log/current`. -The log file is rotated when it reaches a threshold size, into another -file in the same directory whose name contains a TAI64 timestamp. - -Each log line is prefixed with a TAI64 timestamp and the name of the -service, if it is a longrun. If it is a oneshot, a timestamp and the -name of some other service. To convert the timestamp into a -human-readable format, use :command:`s6-tai64nlocal`. - -.. code-block:: console - - # ls -l /run/log/ - -rw-r--r-- 1 0 lock - -rw-r--r-- 1 0 state - -rwxr--r-- 1 98059 @4000000000025cb629c311ac.s - -rwxr--r-- 1 98061 @40000000000260f7309c7fb4.s - -rwxr--r-- 1 98041 @40000000000265233a6cc0b6.s - -rwxr--r-- 1 98019 @400000000002695d10c06929.s - -rwxr--r-- 1 98064 @4000000000026d84189559e0.s - -rwxr--r-- 1 98055 @40000000000271ce1e031d91.s - -rwxr--r-- 1 98054 @400000000002760229733626.s - -rwxr--r-- 1 98104 @4000000000027a2e3b6f4e12.s - -rwxr--r-- 1 98023 @4000000000027e6f0ed24a6c.s - -rw-r--r-- 1 42374 current - - # tail -2 /run/log/current - @40000000000284f130747343 wan.link.pppoe Connect: ppp0 <--> /dev/pts/0 - @40000000000284f230acc669 wan.link.pppoe sent [LCP ConfReq id=0x1 ] - # tail -2 /run/log/current | s6-tai64nlocal - 1970-01-02 21:51:45.828598156 wan.link.pppoe sent [LCP ConfReq id=0x1 ] - 1970-01-02 21:51:48.832588765 wan.link.pppoe sent [LCP ConfReq id=0x1 ] - -Log persistence ---------------- - -Logs written to :file:`/run/log/` will not survive a reboot or crash, -as it is an ephemeral filesystem. - -On supported hardware you can enable logging to `pstore -`_ which -means the most recent log messages will be preserved on reboot. Set -the config option ``logging.persistent.enable = true`` to log messages -to :file:`/dev/pmsg0` as well as to the regular log. This is a -circular buffer, so when it fills up newer messages will overwrite the -oldest messages. - -Logs found in pstore after a reboot will be moved at startup to -:file:`/run/log/previous-boot` - - - -Updating an installed system -**************************** - -If your system has a writable root filesystem (JFFS2, btrfs etc - -anything but squashfs), we have mechanisms for in-places updates -analogous to :command:`nixos-rebuild`, but the operation is a bit -different because it expects to run on a build machine and then copy -to the host device using :command:`ssh`. - -To use this, build the ``outputs.updater`` -target and then run the :command:`update.sh` script it generates. - -.. code-block:: console - - nix-build -I liminix-config=./my-configuration.nix \ - --arg device "import ./devices/mydevice" \ - -A outputs.updater - ./result/bin/update.sh root@the-device - -The update script uses min-copy-closure to copy new or changed -packages to the device, then (perhaps) reboots it. The reboot -behaviour can be affected by flags: - -* `--no-reboot` will cause it not to reboot at all, if you would - rather do that yourself. Note that none of the newly-installed or - updated services will be running until you do. - -* `--fast` causes it tn not do a full reboot, but instead to restart - only the services that have been changed. This will restart all of - the services that have updated store paths (and anything that - depends on them), but will not affect services that haven't changed. - -It doesn't delete old packages automatically: to do that run -:command:`min-collect-garbage`, which will delete any packages not in -the current system closure. Note that Liminix does not have the NixOS -concept of environments or generations, and there is no way back from -this except for building the previous configuration again. - -Caveats -------- - -* it needs there to be enough free space on the device for all the new - packages in addition to all the packages already on it - which may - be a problem if there is little flash storage or if a lot of things - have changed (e.g. a new version of nixpkgs). - -* it may not be able to upgrade the kernel: this is device-dependent. - If your device boots from a kernel image on a raw MTD partition or - or UBI volume, update.sh is unable to alter the kernel partition. - If your device boots from a kernel inside the filesystem (e.g. using - bootloader.extlinux or bootloder.fit) then the kernel will be - upgraded along with the userland - - -Recovery/downgrades -------------------- - -The :command:`update.sh` script also creates a timestamped symlink on -the device which points to the system configuration it installs. If -you install a configuration that doesn't work, you can revert to any -other installed configuration by - -1) booting to some kind of rescue or recovery system (which may be - some vendor-provided rescue option, or your own recovery system - perhaps based on :file:`examples/recovery.nix`) and mounting - your Liminix filesystem on /mnt - -2) picking another previously-installed configuration that _did_ work, - and switching back to it: - -.. code-block:: console - - # ls -ld /mnt/*configuration - lrwxrwxrwx 1 90 /mnt/20252102T182104.configuration -> nix/store/v1w0h4zw65ah4c2r0k7nyy125qrxhq78-system-configuration-aarch64-unknown-linux-musl - lrwxrwxrwx 1 90 /mnt/20251802T181822.configuration -> nix/store/wqjl9s9xljl2wg8257292zghws9ssidk-system-configuration-aarch64-unknown-linux-musl - # : 20251802T181822 is the working system, so reinstall it - # /mnt/20251802T181822.configuration/bin/install /mnt - # umount /mnt - # reboot - -This will install the previous configuration's activation binary into -/bin, and copy its kernel and initramfs into /boot. Note that it -depends on the previous system not having been garbage-collected. - - - - -.. _levitate: - -Adding packages -=============== - -If you simply wish to add a package without any change to services, -you can call :command:`min-copy-closure` directly to install -any package in Nixpkgs or in the Liminix overlay - -.. code-block:: console - - nix-build -I liminix-config=./my-configuration.nix \ - --arg device "import ./devices/mydevice" -A pkgs.tcpdump - - nix-shell -p min-copy-closure root@the-device result/ - -Note that this only copies the package and its dependencies to the -device: it doesn't update any profile to add it to ``$PATH`` - - -.. _rebuilding the system: - -Reinstalling on a running system -******************************** - -Liminix is initially installed from a monolithic -:file:`firmware.bin` - and unless you're running a writable -filesystem, the only way to update it is to build and install a whole -new :file:`firmware.bin`. However, you probably would prefer not to -have to remove it from its installation site, unplug it from the -network and stick serial cables in it all over again. - -It is not (generally) safe to install a new firmware onto the flash -partitions that the active system is running on. To address this we -have :command:`levitate`, which a way for a running Liminix system to -"soft restart" into a ramdisk running only a limited set of services, -so that the main partitions can then be safely flashed. - - - -Configuration -============= - -Levitate *needs to be configured when you create the initial system* -to specify which services/packages/etc to run in maintenance -mode. Most likely you want to configure a network interface and an ssh -for example so that you can login to reflash it. - -.. code-block:: nix - - defaultProfile.packages = with pkgs; [ - ... - (levitate.override { - config = { - services = { - inherit (config.services) dhcpc sshd watchdog; - }; - defaultProfile.packages = [ mtdutils ]; - users.root = config.users.root; - }; - }) - ]; - - - -Use -=== - -Connect (with ssh, probably) to the running Liminix system that you -wish to upgrade. - -.. code-block:: console - - bash$ ssh root@the-device - -Run :command:`levitate`. This takes a little while (perhaps a few -tens of seconds) to execute, and copies all config required for -maintenance mode to :file:`/run/maintenance`. - -.. code-block:: console - - # levitate - -Reboot into maintenance mode. You will be logged out - -.. code-block:: console - - # reboot - -Connect to the device again - note that the ssh host key will have changed. - -.. code-block:: console - - # ssh -o UserKnownHostsFile=/dev/null root@the-device - -Check we're in maintenance mode - -.. code-block:: console - - # cat /etc/banner - - LADIES AND GENTLEMEN WE ARE FLOATING IN SPACE - - Most services are disabled. The system is operating - with a ram-based root filesystem, making it safe to - overwrite the flash devices in order to perform - upgrades and maintenance. - - Don't forget to reboot when you have finished. - -Perform the upgrade, using flashcp. This is an example, -your device will differ - -.. code-block:: console - - # cat /proc/mtd - dev: size erasesize name - mtd0: 00030000 00010000 "u-boot" - mtd1: 00010000 00010000 "u-boot-env" - mtd2: 00010000 00010000 "factory" - mtd3: 00f80000 00010000 "firmware" - mtd4: 00220000 00010000 "kernel" - mtd5: 00d60000 00010000 "rootfs" - mtd6: 00010000 00010000 "art" - # flashcp -v firmware.bin mtd:firmware - -All done - -.. code-block:: console - - # reboot - diff --git a/doc/configuration.adoc b/doc/configuration.adoc new file mode 100644 index 0000000..f088960 --- /dev/null +++ b/doc/configuration.adoc @@ -0,0 +1,470 @@ +== Configuration + +There are many things you can specify in a configuration, but these are +the ones you most commonly need to change: + +* which services (processes) to run +* what packages to install +* permitted users and groups +* Linux kernel configuration options +* Busybox applets +* filesystem layout + +=== Modules + +*Modules* are a means of abstraction which allow "bundling" of +configuration options related to a common purpose or theme. For example, +the `+dnsmasq+` module defines a template for a dnsmasq service, ensures +that the dnsmasq package is installed, and provides a dnsmasq user and +group for the service to run as. The `+ppp+` module defines a service +template and also enables various PPP-related kernel configuration. + +Not all modules are included in the configuration by default, because +that would mean that the kernel (and the Busybox binary providing common +CLI tools) was compiled with many unnecessary bells and whistles and +therefore be bigger than needed. (This is not purely an academic concern +if your device has little flash storage). Therefore, specifying a +service is usually a two-step process. For example, to add an NTP +service you first add `+modules/ntp+` to your `+imports+` list, then you +create a service by calling `+config.system.service.ntp.build { .... }+` +with the appropriate service-dependent configuration parameters. + +[source,nix] +---- +let svc = config.system.service; +in { + # ... + imports = [ + ./modules/ntp + # .... + ]; + config.services.ntp = svc.ntp.build { + pools = { "pool.ntp.org" = ["iburst"]; }; + makestep = { threshold = 1.0; limit = 3; }; + }; +---- + +Merely including the module won't define the service on its own: it only +creates the template in `+config.system.service.foo+` and you have to +create an actual service using the template. This is an intentional +choice to allow the creation of multiple differently-configured services +based on the same template - perhaps e.g. when you have multiple +networks (VPNs etc) in different trust domains, or you want to run two +SSH daemons on different ports. (For the background to this, please +refer to the `+architecture decision record +`) + +[TIP] +==== +Liminix modules should be quite familiar (but also different) if you +already know how to use NixOS modules. We use the NixOS module +infrastructure code, meaning that you should recognise the syntax, the +type system, the rules for combining configuration values from different +sources. We don't use the NixOS modules themselves, because the +underlying system is not similar enough for them to work. +==== + +[[configuration-services]] +=== Services + +In Liminix a service is any kind of long-running task or process on the +system, that is managed (started, stopped, and monitored) by a service +supervisor. A typical SOHO router might have services to + +* answer DHCP and DNS requests from the LAN +* provide a wireless access point +* connect using PPPoE or L2TP to an upstream network +* start/stop the firewall +* enable/disable IP packet forwarding +* mount filesystems + +(Some of these might not be considered services using other definitions +of the term: for example, this L2TP process would be a "client" in the +client/server classification; and enabling packet forwarding doesn't +require any long-lived process - just a setting to be toggled. However, +there is value in being able to use the same abstractions for all the +things to manage them and specify their dependency relationships - so in +Liminix "everything is a service") + +The service supervision system enables service health monitoring, +restart of unhealthy services, and failover to "backup" services when a +primary service fails or its dependencies are unavailable. The intention +is that you have a framework in which you can specify policy +requirements like "ethernet wan dhcp-client should be restarted if it +crashes, but if it can't start because the hardware link is down, then +4G ppp service should be started instead". + +Any attribute in [.title-ref]#config.services# will become part of the +default set of services that s6-rc will try to bring up. Services are +usually started at boot time, but *controlled services* are those that +are required only in particular contexts. For example, a service to +mount a USB backup drive should run only when the drive is attached to +the system. Liminix currently implements three kinds of controlled +service: + +* "uevent-rule" service controllers use sysfs/uevent to identify when +particular hardware devices are present, and start/stop a controlled +service appropriately. +* the "round-robin" service controller is used for service failover: it +allows you to specify a list of services and runs each of them in turn +until it exits, then runs the next. +* the "health-check" service wraps another service, and runs a "health +check" command at regular intervals. When the health check fails, +indicating that the wrapped service is not working, it is terminated and +allowed to restart. + +==== Runtime secrets (external vault) + +Secrets (such as wifi passphrases, PPP username/password, SSH keys, etc) +that you provide as literal values in `+configuration.nix+` are +processed into into config files and scripts at build time, and +eventually end up in various files in the (world-readable) +`+/nix/store+` before being baked into a flashable image. To change a +secret - whether due to a compromise, or just as part of to a routine +key rotation - you need to rebuild the configuration and potentially +reflash the affected devices. + +To avoid this, you may instead use a "secrets service", which is a +mechanism for your device to fetch secrets from a source external to the +Nix store, and create at runtime the configuration files and scripts +that start the services which require them. + +Not every possible parameter to every possible service is configurable +using a secrets service. Parameters which can be configured this way are +those with the type `+liminix.lib.types.replacable+`. At the time this +document was written, these include: + +* ppp (pppoe and l2tp): `+username+`, `+password+` +* ssh: `+authorizedKeys+` +* hostapd: all parameters (most likely to be useful for +`+wpa_passphrase+`) + +To use a runtime secret for any of these parameters: + +* create a secrets service to specify the source of truth for secrets +* use the `+outputRef+` function in the service parameter to specify the +secrets service and path + +For example, given you had an HTTPS server hosting a JSON file with the +structure + +[source,json] +---- +"ssh": { + "authorizedKeys": { +"root": [ "ssh-rsa ....", "ssh-rsa ....", ... ] +"guest": [ "ssh-rsa ....", "ssh-rsa ....", ... ] + } +} +---- + +you could use a `+configuration.nix+` fragment something like this to +make those keys visible to ssh: + +[source,nix] +---- +services.secrets = svc.secrets.outboard.build { + name = "secret-service"; + url = "http://10.0.0.1/secrets.json"; + username = "secrets"; + password = "liminix"; + interval = 30; # minutes + dependencies = [ config.services.lan ]; +}; +services.sshd = svc.ssh.build { + authorizedKeys = outputRef config.services.secrets "ssh/authorizedKeys"; +}; +---- + +There are presently two implementations of a secrets service: + +===== Outboard secrets (HTTPS) + +This service expects a URL to a JSON file containing all the secrets. + +You may specify a username and password along with the URL, which are +used if the file is password-protected (HTTP Basic authentication). Note +that this is not a protection against a malicious local user: the +username and password are normal build-time parameters so will be +readable in the Nix store. This is a mitigation against the URL being +accidentally discovered due to e.g. a log file or error message on the +server leaking. + +===== Tang secrets (encrypted local file) + +Aternatively, secrets may be stored locally on the device, in a file +that has been encrypted using https://github.com/latchset/tang[Tang]. + +____ +Tang is a server for binding data to network presence. + +This sounds fancy, but the concept is simple. You have some data, but +you only want it to be available when the system containing the data is +on a certain, usually secure, network. +____ + +[source,nix] +---- +services.secrets = svc.secrets.tang.build { + name = "secret-service"; + path = "/run/mnt/usbstick/secrets.json.jwe"; + interval = 30; # minutes + dependencies = [ config.services.mount-usbstick ]; +}; +---- + +The encryption uses the same scheme/algorithm as +https://github.com/latchset/clevis[Clevis] : you may use the +https://github.com/latchset/clevis?tab=readme-ov-file#pin-tang[Clevis +instructions] to encrypt the file on another host and then copy it to +your Liminix device, or you can use `+tangc encrypt+` to encrypt +directly on the device. (That latter approach may pose a chicken/egg +problem if the device needs secrets to boot up and run the services you +are relying on in order to login). + +==== Writing services + +For the most part, for common use cases, hopefully the services you need +will be defined by modules and you will only have to pass the right +parameters to `+build+`. + +Should you need to create a custom service of your own devising, use the +[.title-ref]#oneshot# or [.title-ref]#longrun# functions: + +* a "longrun" service is the "normal" service concept: it has a `+run+` +action which describes the process to start, and it watches that process +to restart it if it exits. The process should not attempt to daemonize +or "background" itself, otherwise s6-rc will think it died. Whatever it +prints to standard output/standard error will be logged. + +[source,nix] +---- +config.services.cowsayd = pkgs.liminix.services.longrun { + name = "cowsayd"; + run = "${pkgs.cowsayd}/bin/cowsayd --port 3001 --breed hereford"; + # don't start this until the lan interface is ready + dependencies = [ config.services.lan ]; +} +---- + +* a "oneshot" service doesn't have a process attached. It consists of +`+up+` and `+down+` actions which are bits of shell script that are run +at the appropriate points in the service lifecycle + +[source,nix] +---- +config.services.greenled = pkgs.liminix.services.oneshot { + name = "greenled"; + up = '' +echo 17 > /sys/class/gpio/export +echo out > /sys/class/gpio/gpio17/direction +echo 0 > /sys/class/gpio/gpio17/value + ''; + down = '' +echo 0 > /sys/class/gpio/gpio17/value + ''; +} +---- + +Services may have dependencies: as you see above in the `+cowsayd+` +example, it depends on some service called `+config.services.lan+`, +meaning that it won't be started until that other service is up. + +==== Service outputs + +Outputs are a mechanism by which a service can provide data which may be +required by other services. For example: + +* the DHCP client service can expect to receive nameserver address +information as one of the fields in the response from the DHCP server: +we provide that as an output which a dependent service for a stub name +resolver can use to configure its upstream servers. +* a service that creates a new network interface (e.g. ppp) will provide +the name of the interface (`+ppp0+`, or `+ppp1+` or `+ppp7+`) as an +output so that a dependent service can reference it to set up a route, +or to configure firewall rules. + +A service `+myservice+` should write its outputs as files in +`+/run/services/outputs/myservice+`: you can look around this directory +on a running Liminix system to see how it's used currently. Usually we +use the `+in_outputs+` shell function in the `+up+` or `+run+` +attributes of the service: + +[source,shell] +---- +(in_outputs ${name} + for i in lease mask ip router siaddr dns serverid subnet opt53 interface ; do + (printenv $i || true) > $i + done) +---- + +The outputs are just files, so technically you can read them using +anything that can read a file. Liminix has two "preferred" mechanisms, +though: + +===== One-off lookups + +In any context that ends up being evaluated by the shell, use `+output+` +to print the value of an output + +[source,nix] +---- +services.defaultroute4 = svc.network.route.build { + via = "$(output ${services.wan} address)"; + target = "default"; + dependencies = [ services.wan ]; +}; +---- + +===== Continuous updates + +The downside of using shell functions in downstream service startup +scripts is that they only run when the service starts up: if a service +output _changes_, the downstream service would have to be restarted to +notice the change. Sometimes this is OK but other times the downstream +has no other need to restart, if it can only get its new data. + +For this case, there is the `+anoia.svc+` Fennel library, which allows +you to write a simple loop which is iterated over whenever a service's +outputs change. This code is from +`+modules/dhcp6c/acquire-wan-address.fnl+` + +[source,fennel] +---- +(fn update-addresses [wan-device addresses new-addresses exec] + ;; run some appropriate "ip address [add|remove]" commands + ) + +(fn run [] + (let [[state-directory wan-device] arg + dir (svc.open state-directory)] +(accumulate [addresses [] + v (dir:events)] + (update-addresses wan-device addresses + (or (v:output "address") []) system)))) +---- + +The `+output+` method seen here accepts a filename (relative to the +service's output directory), or a directory name. It returns the first +line of that file, or for directories it returns a table (Lua's +key/value datastructure, similar to a hash/dictionary) of the outputs in +that directory. + +===== Output design considerations + +For preference, outputs should be short and simple, and not require +downstream services to do complicated parsing in order to use them. +Shell commands in Liminix are run using the Busybox shell which doesn't +have the niceties of an advanced shell like Bash let alone those of a +real programming language. + +Note also that the Lua `+svc+` library only reads the first line of each +output. + +=== Module implementation + +Modules in Liminix conventionally live in +`+modules/somename/default.nix+`. If you want or need to write your own, +you may wish to refer to the examples there in conjunction with reading +this section. + +A module is a function that accepts `+{lib, pkgs, config, ... }+` and +returns an attrset with keys `+imports, options config+`. + +* `+imports+` is a list of paths to the other modules required by this +one +* `+options+` is a nested set of option declarations +* `+config+` is a nested set of option definitions + +The NixOS manual section +https://nixos.org/manual/nixos/stable/#sec-writing-modules[Writing NixOS +Modules] is a quite comprehensive reference to writing NixOS modules, +which is also mostly applicable to Liminix except that it doesn't cover +service templates. + +==== Service templates + +To expose a service template in a module, it needs the following: + +* an option declaration for `+system.service.myservicename+` with the +type of `+liminix.lib.types.serviceDefn+` + +[source,nix] +---- +options = { + system.service.cowsay = mkOption { +type = liminix.lib.types.serviceDefn; + }; +}; +---- + +* an option definition for the same key, which specifies where to import +the service template from (often `+./service.nix+`) and the types of its +parameters. + +[source,nix] +---- +config.system.service.cowsay = config.system.callService ./service.nix { + address = mkOption { +type = types.str; +default = "0.0.0.0"; +description = "Listen on specified address"; +example = "127.0.0.1"; + }; + port = mkOption { +type = types.port; +default = 22; +description = "Listen on specified TCP port"; + }; + breed = mkOption { +type = types.str; +default = "British Friesian" +description = "Breed of the cow"; + }; +}; +---- + +Then you need to provide the service template itself, probably in +`+./service.nix+`: + +[source,nix] +---- +{ + # any nixpkgs package can be named here + liminix +, cowsayd +, serviceFns +, lib +}: +# these are the parameters declared in the callService invocation +{ address, port, breed} : +let + inherit (liminix.services) longrun; + inherit (lib.strings) escapeShellArg; +in longrun { + name = "cowsayd"; + run = "${cowsayd}/bin/cowsayd --address ${address} --port ${builtins.toString port} --breed ${escapeShellArg breed}"; +} +---- + +[TIP] +==== +Not relevant to module-based services specifically, but a common gotcha +when specifiying services is forgetting to transform "rich" parameter +values into text when composing a command for the shell to execute. Note +here that the port number, an integer, is stringified with `+toString+`, +and the name of the breed, which may contain spaces, is escaped with +`+escapeShellArg+` +==== + +==== Types + +All of the NixOS module types are available in Liminix. These +Liminix-specific types also exist in `+pkgs.liminix.lib.types+`: + +* `+service+`: an s6-rc service +* `+interface+`: an s6-rc service which specifies a network interface +* `+serviceDefn+`: a service "template" definition + +In the future it is likely that we will extend this to include other +useful types in the networking domain: for example; IP address, network +prefix or netmask, protocol family and others as we find them. diff --git a/doc/configuration.rst b/doc/configuration.rst deleted file mode 100644 index c485369..0000000 --- a/doc/configuration.rst +++ /dev/null @@ -1,482 +0,0 @@ -.. _configuration: - -Configuration -############# - -There are many things you can specify in a configuration, but these -are the ones you most commonly need to change: - -* which services (processes) to run -* what packages to install -* permitted users and groups -* Linux kernel configuration options -* Busybox applets -* filesystem layout - - -Modules -******* - -**Modules** are a means of abstraction which allow "bundling" -of configuration options related to a common purpose or theme. For -example, the ``dnsmasq`` module defines a template for a dnsmasq -service, ensures that the dnsmasq package is installed, and provides a -dnsmasq user and group for the service to run as. The ``ppp`` module -defines a service template and also enables various PPP-related kernel -configuration. - -Not all modules are included in the configuration by default, because -that would mean that the kernel (and the Busybox binary providing -common CLI tools) was compiled with many unnecessary bells and whistles -and therefore be bigger than needed. (This is not purely an academic concern -if your device has little flash storage). Therefore, specifying a -service is usually a two-step process. For example, to add an NTP -service you first add :file:`modules/ntp` to your ``imports`` list, -then you create a service by calling -:code:`config.system.service.ntp.build { .... }` with the appropriate -service-dependent configuration parameters. - -.. code-block:: nix - - let svc = config.system.service; - in { - # ... - imports = [ - ./modules/ntp - # .... - ]; - config.services.ntp = svc.ntp.build { - pools = { "pool.ntp.org" = ["iburst"]; }; - makestep = { threshold = 1.0; limit = 3; }; - }; - -Merely including the module won't define the service on its own: it -only creates the template in ``config.system.service.foo`` and you -have to create an actual service using the template. This is an -intentional choice to allow the creation of multiple -differently-configured services based on the same template - perhaps -e.g. when you have multiple networks (VPNs etc) in different trust -domains, or you want to run two SSH daemons on different ports. -(For the background to this, please refer to the :doc:`architecture decision record `) - -.. tip:: Liminix modules should be quite familiar (but also different) - if you already know how to use NixOS modules. We use the - NixOS module infrastructure code, meaning that you should - recognise the syntax, the type system, the rules for - combining configuration values from different sources. We - don't use the NixOS modules themselves, because the - underlying system is not similar enough for them to work. - -.. _configuration-services: - -Services -******** - -In Liminix a service is any kind of long-running task or process on -the system, that is managed (started, stopped, and monitored) by a -service supervisor. A typical SOHO router might have services to - -* answer DHCP and DNS requests from the LAN -* provide a wireless access point -* connect using PPPoE or L2TP to an upstream network -* start/stop the firewall -* enable/disable IP packet forwarding -* mount filesystems - -(Some of these might not be considered services using other -definitions of the term: for example, this L2TP process would be a -"client" in the client/server classification; and enabling packet -forwarding doesn't require any long-lived process - just a setting to -be toggled. However, there is value in being able to use the same -abstractions for all the things to manage them and specify their -dependency relationships - so in Liminix "everything is a service") - -The service supervision system enables service health monitoring, -restart of unhealthy services, and failover to "backup" services when -a primary service fails or its dependencies are unavailable. The -intention is that you have a framework in which you can specify policy -requirements like "ethernet wan dhcp-client should be restarted if it -crashes, but if it can't start because the hardware link is down, then -4G ppp service should be started instead". - -Any attribute in `config.services` will become part of the default set -of services that s6-rc will try to bring up. Services are usually -started at boot time, but **controlled services** are those that are -required only in particular contexts. For example, a service to mount -a USB backup drive should run only when the drive is attached to the -system. Liminix currently implements three kinds of controlled service: - -* "uevent-rule" service controllers use sysfs/uevent to identify when - particular hardware devices are present, and start/stop a controlled - service appropriately. - -* the "round-robin" service controller is used for service failover: - it allows you to specify a list of services and runs each of them - in turn until it exits, then runs the next. - -* the "health-check" service wraps another service, and runs a "health - check" command at regular intervals. When the health check fails, - indicating that the wrapped service is not working, it is terminated - and allowed to restart. - -Runtime secrets (external vault) -================================ - -Secrets (such as wifi passphrases, PPP username/password, SSH keys, -etc) that you provide as literal values in :file:`configuration.nix` -are processed into into config files and scripts at build time, and -eventually end up in various files in the (world-readable) -:file:`/nix/store` before being baked into a flashable image. To -change a secret - whether due to a compromise, or just as part of to a -routine key rotation - you need to rebuild the configuration and -potentially reflash the affected devices. - -To avoid this, you may instead use a "secrets service", which is a -mechanism for your device to fetch secrets from a source external to -the Nix store, and create at runtime the configuration files and -scripts that start the services which require them. - -Not every possible parameter to every possible service is configurable -using a secrets service. Parameters which can be configured this way -are those with the type ``liminix.lib.types.replacable``. At the time -this document was written, these include: - -* ppp (pppoe and l2tp): ``username``, ``password`` -* ssh: ``authorizedKeys`` -* hostapd: all parameters (most likely to be useful for ``wpa_passphrase``) - -To use a runtime secret for any of these parameters: - -* create a secrets service to specify the source of truth for secrets -* use the :code:`outputRef` function in the service parameter to specify the secrets service and path - -For example, given you had an HTTPS server hosting a JSON file with the structure - -.. code-block:: json - - "ssh": { - "authorizedKeys": { - "root": [ "ssh-rsa ....", "ssh-rsa ....", ... ] - "guest": [ "ssh-rsa ....", "ssh-rsa ....", ... ] - } - } - -you could use a :file:`configuration.nix` fragment something like this -to make those keys visible to ssh: - -.. code-block:: nix - - services.secrets = svc.secrets.outboard.build { - name = "secret-service"; - url = "http://10.0.0.1/secrets.json"; - username = "secrets"; - password = "liminix"; - interval = 30; # minutes - dependencies = [ config.services.lan ]; - }; - services.sshd = svc.ssh.build { - authorizedKeys = outputRef config.services.secrets "ssh/authorizedKeys"; - }; - - - -There are presently two implementations of a secrets service: - -Outboard secrets (HTTPS) ------------------------- - -This service expects a URL to a JSON file containing all the secrets. - -You may specify a username and password along with the URL, which are -used if the file is password-protected (HTTP Basic -authentication). Note that this is not a protection against a -malicious local user: the username and password are normal build-time -parameters so will be readable in the Nix store. This is a mitigation -against the URL being accidentally discovered due to e.g. a log file -or error message on the server leaking. - - -Tang secrets (encrypted local file) ------------------------------------ - -Aternatively, secrets may be stored locally on the device, in a file -that has been encrypted using `Tang `_. - - Tang is a server for binding data to network presence. - - This sounds fancy, but the concept is simple. You have some data, but you only want it to be available when the system containing the data is on a certain, usually secure, network. - - -.. code-block:: nix - - services.secrets = svc.secrets.tang.build { - name = "secret-service"; - path = "/run/mnt/usbstick/secrets.json.jwe"; - interval = 30; # minutes - dependencies = [ config.services.mount-usbstick ]; - }; - -The encryption uses the -same scheme/algorithm as `Clevis `_ : you may use the `Clevis instructions `_ to -encrypt the file on another host and then copy it to your Liminix -device, or you can use :command:`tangc encrypt` to encrypt directly on -the device. (That latter approach may pose a chicken/egg problem if -the device needs secrets to boot up and run the services you are -relying on in order to login). - - -Writing services -================ - -For the most part, for common use cases, hopefully the services you -need will be defined by modules and you will only have to pass the -right parameters to ``build``. - -Should you need to create a custom service of your own devising, use -the `oneshot` or `longrun` functions: - -* a "longrun" service is the "normal" service concept: it has a - ``run`` action which describes the process to start, and it watches - that process to restart it if it exits. The process should not - attempt to daemonize or "background" itself, otherwise s6-rc will think - it died. Whatever it prints to standard output/standard error - will be logged. - -.. code-block:: nix - - config.services.cowsayd = pkgs.liminix.services.longrun { - name = "cowsayd"; - run = "${pkgs.cowsayd}/bin/cowsayd --port 3001 --breed hereford"; - # don't start this until the lan interface is ready - dependencies = [ config.services.lan ]; - } - - -* a "oneshot" service doesn't have a process attached. It consists of - ``up`` and ``down`` actions which are bits of shell script that - are run at the appropriate points in the service lifecycle - -.. code-block:: nix - - config.services.greenled = pkgs.liminix.services.oneshot { - name = "greenled"; - up = '' - echo 17 > /sys/class/gpio/export - echo out > /sys/class/gpio/gpio17/direction - echo 0 > /sys/class/gpio/gpio17/value - ''; - down = '' - echo 0 > /sys/class/gpio/gpio17/value - ''; - } - -Services may have dependencies: as you see above in the ``cowsayd`` -example, it depends on some service called ``config.services.lan``, -meaning that it won't be started until that other service is up. - -Service outputs -=============== - -Outputs are a mechanism by which a service can provide data which may -be required by other services. For example: - -* the DHCP client service can expect to receive nameserver address - information as one of the fields in the response from the DHCP - server: we provide that as an output which a dependent service for a - stub name resolver can use to configure its upstream servers. - -* a service that creates a new network interface (e.g. ppp) will - provide the name of the interface (:code:`ppp0`, or :code:`ppp1` or - :code:`ppp7`) as an output so that a dependent service can reference - it to set up a route, or to configure firewall rules. - -A service :code:`myservice` should write its outputs as files in -:file:`/run/services/outputs/myservice`: you can look around this -directory on a running Liminix system to see how it's used currently. -Usually we use the :code:`in_outputs` shell function in the -:command:`up` or :command:`run` attributes of the service: - -.. code-block:: shell - - (in_outputs ${name} - for i in lease mask ip router siaddr dns serverid subnet opt53 interface ; do - (printenv $i || true) > $i - done) - -The outputs are just files, so technically you can read them using -anything that can read a file. Liminix has two "preferred" -mechanisms, though: - -One-off lookups ---------------- - -In any context that ends up being evaluated by the shell, use -:code:`output` to print the value of an output - -.. code-block:: nix - - services.defaultroute4 = svc.network.route.build { - via = "$(output ${services.wan} address)"; - target = "default"; - dependencies = [ services.wan ]; - }; - - -Continuous updates ------------------- - -The downside of using shell functions in downstream service startup -scripts is that they only run when the service starts up: if a service -output *changes*, the downstream service would have to be restarted to -notice the change. Sometimes this is OK but other times the downstream -has no other need to restart, if it can only get its new data. - -For this case, there is the :code:`anoia.svc` Fennel library, which -allows you to write a simple loop which is iterated over whenever a -service's outputs change. This code is from -:file:`modules/dhcp6c/acquire-wan-address.fnl` - -.. code-block:: fennel - - (fn update-addresses [wan-device addresses new-addresses exec] - ;; run some appropriate "ip address [add|remove]" commands - ) - - (fn run [] - (let [[state-directory wan-device] arg - dir (svc.open state-directory)] - (accumulate [addresses [] - v (dir:events)] - (update-addresses wan-device addresses - (or (v:output "address") []) system)))) - -The :code:`output` method seen here accepts a filename (relative -to the service's output directory), or a directory name. It -returns the first line of that file, or for directories it -returns a table (Lua's key/value datastructure, similar to -a hash/dictionary) of the outputs in that directory. - - -Output design considerations ----------------------------- - -For preference, outputs should be short and simple, and not require -downstream services to do complicated parsing in order to use them. -Shell commands in Liminix are run using the Busybox shell which -doesn't have the niceties of an advanced shell like Bash let alone -those of a real programming language. - -Note also that the Lua :code:`svc` library only reads the first line -of each output. - - -Module implementation -********************* - -Modules in Liminix conventionally live in -:file:`modules/somename/default.nix`. If you want or need to -write your own, you may wish to refer to the -examples there in conjunction with reading this section. - -A module is a function that accepts ``{lib, pkgs, config, ... }`` and -returns an attrset with keys ``imports, options config``. - -* ``imports`` is a list of paths to the other modules required by this one - -* ``options`` is a nested set of option declarations - -* ``config`` is a nested set of option definitions - -The NixOS manual section `Writing NixOS Modules -`_ is a -quite comprehensive reference to writing NixOS modules, which is also -mostly applicable to Liminix except that it doesn't cover -service templates. - -Service templates -================= - -To expose a service template in a module, it needs the following: - -* an option declaration for ``system.service.myservicename`` with the - type of ``liminix.lib.types.serviceDefn`` - -.. code-block:: nix - - options = { - system.service.cowsay = mkOption { - type = liminix.lib.types.serviceDefn; - }; - }; - -* an option definition for the same key, which specifies where to - import the service template from (often :file:`./service.nix`) - and the types of its parameters. - -.. code-block:: nix - - config.system.service.cowsay = config.system.callService ./service.nix { - address = mkOption { - type = types.str; - default = "0.0.0.0"; - description = "Listen on specified address"; - example = "127.0.0.1"; - }; - port = mkOption { - type = types.port; - default = 22; - description = "Listen on specified TCP port"; - }; - breed = mkOption { - type = types.str; - default = "British Friesian" - description = "Breed of the cow"; - }; - }; - -Then you need to provide the service template itself, probably in -:file:`./service.nix`: - -.. code-block:: nix - - { - # any nixpkgs package can be named here - liminix - , cowsayd - , serviceFns - , lib - }: - # these are the parameters declared in the callService invocation - { address, port, breed} : - let - inherit (liminix.services) longrun; - inherit (lib.strings) escapeShellArg; - in longrun { - name = "cowsayd"; - run = "${cowsayd}/bin/cowsayd --address ${address} --port ${builtins.toString port} --breed ${escapeShellArg breed}"; - } - -.. tip:: - - Not relevant to module-based services specifically, but a common - gotcha when specifiying services is forgetting to transform "rich" - parameter values into text when composing a command for the shell - to execute. Note here that the port number, an integer, is - stringified with ``toString``, and the name of the breed, - which may contain spaces, is - escaped with ``escapeShellArg`` - -Types -===== - -All of the NixOS module types are available in Liminix. These -Liminix-specific types also exist in ``pkgs.liminix.lib.types``: - -* ``service``: an s6-rc service -* ``interface``: an s6-rc service which specifies a network - interface -* ``serviceDefn``: a service "template" definition - -In the future it is likely that we will extend this to include other -useful types in the networking domain: for example; IP address, -network prefix or netmask, protocol family and others as we find them. diff --git a/doc/development.adoc b/doc/development.adoc new file mode 100644 index 0000000..64026f4 --- /dev/null +++ b/doc/development.adoc @@ -0,0 +1,311 @@ +== Development + +As a developer working on Liminix, or implementing a service or module, +you probably want to test your changes more conveniently than by +building and flashing a new image every time. This section documents +various affordances for iteration and experiments. + +In general, packages and tools that run on the "build" machine are +available in the `+buildEnv+` derivation and can most easily be added to +your environment by running `+nix-shell+`. + +=== Emulated devices + +Liminix has a `+qemu+` device, which generates images suitable for +running on your build machine using the free http://www.qemu.org[QEMU +machine emulator]. This is useful for developing userland without +needing to keep flashing or messing with U-Boot: it also enables testing +against emulated network peers using +https://wiki.qemu.org/Documentation/Networking#Socket[QEMU socket +networking], which may be preferable to letting Liminix loose on your +actual LAN. To build it, + +[source,console] +---- +nix-build -I liminix-config=path/to/your/configuration.nix --arg device "import ./devices/qemu" -A outputs.default +---- + +This creates a `+result/+` directory containing a `+vmlinux+` and a +`+rootfs+`, and also a shell script `+run.sh+` which invokes QEMU to run +that kernel with that filesystem. It connects the Liminix serial console +and the https://www.qemu.org/docs/master/system/monitor.html[QEMU +monitor] to stdin/stdout. Use ^P (not ^A) to switch to the monitor. + +If you run with `+--background /path/to/some/directory+` as the first +parameter, it will fork into the background and open Unix sockets in +that directory for console and monitor. Use `+nix-shell --run +connect-vm+` to connect to either of these sockets, and ^O to +disconnect. + +[[qemu-networking]] +==== Networking + +VMs can network with each other using QEMU socket networking. We observe +these conventions, so that we can run multiple emulated instances and +have them wired up to each other in the right way: + +* multicast 230.0.0.1:1234 : access (interconnect between router and +"isp") +* multicast 230.0.0.1:1235 : lan +* multicast 230.0.0.1:1236 : world (the internet) + +Any VM started by a `+run.sh+` script is connected to "lan" and +"access", and the emulated border network gateway (see below) runs PPPoE +and is connected to "access" and "world". + +===== Border Network Gateway + +In pkgs/routeros there is a derivation to install and configure +https://mikrotik.com/software[Mikrotik RouterOS] as a PPPoE access +concentrator connected to the `+access+` and `+world+` networks, so that +Liminix PPPoE client support can be tested without actual hardware. + +This is made available as the `+routeros+` command in `+buildEnv+`, so +you can do something like: + +.... +mkdir ros-sockets +nix-shell +nix-shell$ routeros ros-sockets +nix-shell$ connect-vm ./ros-sockets/console +.... + +to start it and connect to it. Note that by default it runs in the +background. It is connected to "access" and "world" virtual networks and +runs a PPPoE service on "access" - so a Liminix VM with a PPPOE client +can connect to it and thus reach the virtual internet. [ check, but +pretty sure this is not the actual internet ] + +[.title-ref]#Liminix does not provide RouterOS licences and it is your +own responsibility if you use this to ensure you're compliant with the +terms of Mikrotik's licencing. It may be supplemented or replaced in +time with configurations for RP-PPPoE and/or Accel PPP.# + +=== Hardware devices + +==== TFTP + +[[tftp server]] +How you get your image onto hardware will vary according to the device, +but is likely to involve taking it apart to add wires to serial console +pads/headers, then using U-Boot to fetch images over TFTP. The OpenWrt +documentation has a +https://openwrt.org/docs/techref/hardware/port.serial[good explanation] +of what you may expect to find on the device. + +There is a rudimentary TFTP server bundled with the system which runs +from the command line, has an allowlist for client connections, and +follows symlinks, so you can have your device download images direct +from the `+./result+` directory without exposing `+/nix/store/+` to the +internet or mucking about copying files to `+/tftproot+`. If the +permitted device is to be given the IP address 192.168.8.251 you might +do something like this: + +[source,console] +---- +nix-shell --run "tufted -a 192.168.8.251 result" +---- + +Now add the device and server IP addresses to your configuration: + +[source,nix] +---- +boot.tftp = { + serverip = "192.168.8.111"; + ipaddr = "192.168.8.251"; +}; +---- + +and then build the derivation for `+outputs.default+` or +`+outputs.mtdimage+` (for which it will be an alias on any device where +this is applicable). You should find it has created + +* `+result/firmware.bin+` which is the file you are going to flash +* `+result/flash.scr+` which is a set of instructions to U-Boot to +download the image and write it to flash after erasing the appropriate +flash partition. + +[NOTE] +==== +TTL serial connections typically have no form of flow control and so +don't always like having massive chunks of text pasted into them - and +U-Boot may drop characters while it's busy. So don't necessarily expect +to copy-paste the whole of `+boot.scr+` into a terminal emulator and +have it work just like that. You may need to paste each line one at a +time, or even retype it. +==== + +For a faster edit-compile-test cycle, you can build a TFTP-bootable +image instead of flashing. In your device configuration add + +[source,nix] +---- +imports = [ + ./modules/tftpboot.nix +]; +---- + +and then build `+outputs.tftpboot+`. This creates a file in `+result/+` +called `+boot.scr+`, which you can copy and paste into U-Boot to +transfer the kernel and filesystem over TFTP and boot the kernel from +RAM. + +[[bng]] +==== Networking + +You probably don't want to be testing a device that might serve DHCP, +DNS and routing protocols on the same LAN as you (or your colleagues, +employees, or family) are using for anything else, because it will +interfere. You also might want to test the device against an "upstream" +connection without having to unplug your regular home router from the +internet so you can borrow the cable/fibre/DSL. + +`+bordervm+` is included for this purpose. You will need + +* a Linux machine with a spare (PCI or USB) ethernet device which you +can dedicate to Liminix +* an L2TP service such as https://www.aa.net.uk/broadband/l2tp-service/ + +You need to "hide" the Ethernet device from the host - for PCI this +means configuring it for VFIO passthru; for USB you need to unload the +module(s) it uses. I have this segment in configuration.nix which you +may be able to adapt: + +[source,nix] +---- +boot = { + kernelParams = [ "intel_iommu=on" ]; + kernelModules = [ + "kvm-intel" "vfio_virqfd" "vfio_pci" "vfio_iommu_type1" "vfio" + ]; + + postBootCommands = '' + # modprobe -i vfio-pci + # echo vfio-pci > /sys/bus/pci/devices/0000:01:00.0/driver_override + ''; + blacklistedKernelModules = [ + "r8153_ecm" "cdc_ether" + ]; +}; +services.udev.extraRules = '' + SUBSYSTEM=="usb", ATTRS{idVendor}=="0bda", ATTRS{idProduct}=="8153", OWNER="dan" +''; +---- + +Then you can execute `+run-border-vm+` in a `+buildEnv+` shell, which +starts up QEMU using the NixOS configuration in +`+bordervm-configuration.nix+`. + +In this VM + +* your Liminix checkout is mounted under `+/home/liminix/liminix+` +* TFTP is listening on the ethernet device and serving +`+/home/liminix/liminix+`. The server IP address is 10.0.0.1 +* a PPPOE-L2TP relay is running on the same ethernet card. When the +connected Liminix device makes PPPoE requests, the relay spawns L2TPv2 +Access Concentrator sessions to your specified L2TP LNS. Note that +authentication is expected at the PPP layer not the L2TP layer, so the +PAP/CHAP credentials provided by your L2TP service can be configured +into your test device - bordervm doesn't need to know about them. + +To configure bordervm, you need a file called `+bordervm.conf.nix+` +which you can create by copying and appropriately editing +`+bordervm.conf-example.nix+` + +[NOTE] +==== +If you make changes to the bordervm configuration after executing +`+run-border-vm+`, you need to remove the `+border.qcow2+` disk image +file otherwise the changes won't get picked up. +==== + +=== Running tests + +You can run all of the tests by evaluating `+ci.nix+`, which is the +input I use in Hydra. + +[source,console] +---- +nix-build -I liminix=`pwd` ci.nix -A pppoe # run one job +nix-build -I liminix=`pwd` ci.nix -A all # run all jobs +---- + +=== Troubleshooting + +==== Diagnosing unexpectedly large images + +Sometimes you can add a package and it causes the image size to balloon +because it has dependencies on other things you didn't know about. Build +the `+outputs.manifest+` attribute, which is a JSON representation of +the filesystem, and you can run `+nix-store --query+` on it. + +[source,console] +---- +nix-build -I liminix-config=path/to/your/configuration.nix \ + --arg device "import ./devices/qemu" -A outputs.manifest \ + -o manifest +nix-store -q --tree manifest +---- + +=== Contributing + +Contributions are welcome, though in these early days there may be a bit +of back and forth involved before patches are merged: Please get in +touch somehow [.title-ref]#before# you invest a lot of time into a code +contribution I haven't asked for. Just so I know it's expected and +you're not wasting time doing something I won't accept or have already +started on. + +==== Nix language style + +This section describes some Nix language style points that we attempt to +adhere to in this repo. + +* favour `+callPackage+` over raw `+import+` for calling derivations or +any function that may generate one - any code that might need `+pkgs+` +or parts of it. +* prefer `+let inherit (quark) up down strange charm+` over +`+with quark+`, in any context where the scope is more than a single +expression or there is more than one reference to `+up+`, `+down+` etc. +`+with pkgs; [ foo bar baz]+` is OK, +`+with lib; stdenv.mkDerivation { ... }+` is usually not. +* `++` is defined only when running tests, so don't refer to it +in "application" code +* the parameters to a derivation are sorted alphabetically, except for +`+lib+`, `+stdenv+` and maybe other non-package "special cases" +* indentation is whatever emacs nix-mode says it is. +* where a `+let+` form defines multiple names, put a newline after the +token `+let+`, and indent each name two characters +* to decide whether some code should be a package or a module? Packages +are self-contained - they live in `+/nix/store/eeeeeee-name+` and don't +directly change system behaviour by their presence or absense. modules +can add to `+/etc+` or `+/bin+` or other global state, create services, +all that side-effecty stuff. Generally it should be a package unless it +can't be. + +==== Copyright + +The Nix code in Liminix is MIT-licenced (same as Nixpkgs), but the code +it combines from other places (e.g. Linux, OpenWrt) may have a variety +of licences. I have no intention of asking for copyright assignment: +just like when submitting to the Linux kernel you retain the copyright +on the code you contribute. + +==== Code of Conduct + +Please govern yourself in Liminix project venues according to the +https://gti.telent.net/dan/liminix/src/commit/7bcf6b15c3fdddafeda13f65b3cd4a422dc52cd3/CODE-OF-CONDUCT.md[Code +of Conduct] + +==== Where to send patches + +Liminix' primary repo is https://gti.telent.net/dan/liminix but you +can't send code there directly because it doesn't have open +registrations. + +* There's a https://github.com/telent/liminix[mirror on Github] for +convenience and visibility: you can open PRs against that +* or, you can send me your patch by email using +https://git-send-email.io/[git send-email] +* or in the future, some day, we will have federated Gitea using +ActivityPub. diff --git a/doc/development.rst b/doc/development.rst deleted file mode 100644 index ee73414..0000000 --- a/doc/development.rst +++ /dev/null @@ -1,340 +0,0 @@ -Development -########### - -As a developer working on Liminix, or implementing a service or -module, you probably want to test your changes more conveniently -than by building and flashing a new image every time. This section -documents various affordances for iteration and experiments. - -In general, packages and tools that run on the "build" machine are -available in the ``buildEnv`` derivation and can most easily -be added to your environment by running :command:`nix-shell`. - - - -Emulated devices -**************** - -Liminix has a ``qemu`` device, which generates images suitable for -running on your build machine using the free `QEMU machine emulator `_. -This is useful for developing userland without needing to keep -flashing or messing with U-Boot: it also enables testing against -emulated network peers using `QEMU socket networking `_, -which may be preferable to letting Liminix loose on your actual LAN. -To build it, - -.. code-block:: console - - nix-build -I liminix-config=path/to/your/configuration.nix --arg device "import ./devices/qemu" -A outputs.default - -This creates a :file:`result/` directory containing a :file:`vmlinux` -and a :file:`rootfs`, and also a shell script :file:`run.sh` which -invokes QEMU to run that kernel with that filesystem. It connects the Liminix -serial console and the `QEMU monitor `_ to stdin/stdout. Use ^P (not ^A) to switch to the monitor. - -If you run with ``--background /path/to/some/directory`` as the first -parameter, it will fork into the background and open Unix sockets in -that directory for console and monitor. Use :command:`nix-shell --run -connect-vm` to connect to either of these sockets, and ^O to -disconnect. - -.. _qemu-networking: - -Networking -========== - -VMs can network with each other using QEMU -socket networking. We observe these conventions, so that we can run -multiple emulated instances and have them wired up to each other in -the right way: - -* multicast 230.0.0.1:1234 : access (interconnect between router and "isp") -* multicast 230.0.0.1:1235 : lan -* multicast 230.0.0.1:1236 : world (the internet) - -Any VM started by a :command:`run.sh` script is connected to "lan" and -"access", and the emulated border network gateway (see below) runs -PPPoE and is connected to "access" and "world". - -.. _border-network-gateway: - -Border Network Gateway ----------------------- - -In pkgs/routeros there is a derivation to install and configure -`Mikrotik RouterOS `_ as a PPPoE access -concentrator connected to the ``access`` and ``world`` networks, so that -Liminix PPPoE client support can be tested without actual hardware. - -This is made available as the :command:`routeros` command in -``buildEnv``, so you can do something like:: - - mkdir ros-sockets - nix-shell - nix-shell$ routeros ros-sockets - nix-shell$ connect-vm ./ros-sockets/console - -to start it and connect to it. Note that by default it runs in the -background. It is connected to "access" and "world" virtual networks -and runs a PPPoE service on "access" - so a Liminix VM with a -PPPOE client can connect to it and thus reach the virtual internet. -[ check, but pretty sure this is not the actual internet ] - -`Liminix does not provide RouterOS licences and it is your own -responsibility if you use this to ensure you're compliant with the -terms of Mikrotik's licencing. It may be supplemented or replaced in -time with configurations for RP-PPPoE and/or Accel PPP.` - -Hardware devices -**************** - - -TFTP -==== - -.. _tftp server: - -How you get your image onto hardware will vary according to the -device, but is likely to involve taking it apart to add wires to -serial console pads/headers, then using U-Boot to fetch images over -TFTP. The OpenWrt documentation has a `good explanation `_ of what you may expect to find on -the device. - -There is a rudimentary TFTP server bundled with the system which runs -from the command line, has an allowlist for client connections, and -follows symlinks, so you can have your device download images direct -from the :file:`./result` directory without exposing :file:`/nix/store/` to the -internet or mucking about copying files to :file:`/tftproot`. If the -permitted device is to be given the IP address 192.168.8.251 you might -do something like this: - -.. code-block:: console - - nix-shell --run "tufted -a 192.168.8.251 result" - -Now add the device and server IP addresses to your configuration: - -.. code-block:: nix - - boot.tftp = { - serverip = "192.168.8.111"; - ipaddr = "192.168.8.251"; - }; - -and then build the derivation for ``outputs.default`` or -``outputs.mtdimage`` (for which it will be an alias on any device -where this is applicable). You should find it has created - -* :file:`result/firmware.bin` which is the file you are going to flash -* :file:`result/flash.scr` which is a set of instructions to U-Boot to - download the image and write it to flash after erasing the appropriate - flash partition. - -.. NOTE:: - - TTL serial connections typically have no form of flow control and - so don't always like having massive chunks of text pasted into - them - and U-Boot may drop characters while it's busy. So don't - necessarily expect to copy-paste the whole of :file:`boot.scr` into - a terminal emulator and have it work just like that. You may need - to paste each line one at a time, or even retype it. - - -For a faster edit-compile-test cycle, you can build a TFTP-bootable -image instead of flashing. In your device configuration add - -.. code-block:: nix - - imports = [ - ./modules/tftpboot.nix - ]; - -and then build ``outputs.tftpboot``. This creates a file in -``result/`` called ``boot.scr``, which you can copy and paste into -U-Boot to transfer the kernel and filesystem over TFTP and boot the -kernel from RAM. - - -.. _bng: - -Networking -========== - -You probably don't want to be testing a device that might serve DHCP, -DNS and routing protocols on the same LAN as you (or your colleagues, -employees, or family) are using for anything else, because it will -interfere. You also might want to test the device against an -"upstream" connection without having to unplug your regular home -router from the internet so you can borrow the cable/fibre/DSL. - -``bordervm`` is included for this purpose. You will need - -* a Linux machine with a spare (PCI or USB) ethernet device which you can dedicate to Liminix - -* an L2TP service such as https://www.aa.net.uk/broadband/l2tp-service/ - -You need to "hide" the Ethernet device from the host - for PCI this -means configuring it for VFIO passthru; for USB you need to unload the -module(s) it uses. I have this segment in configuration.nix which you -may be able to adapt: - -.. code-block:: nix - - boot = { - kernelParams = [ "intel_iommu=on" ]; - kernelModules = [ - "kvm-intel" "vfio_virqfd" "vfio_pci" "vfio_iommu_type1" "vfio" - ]; - - postBootCommands = '' - # modprobe -i vfio-pci - # echo vfio-pci > /sys/bus/pci/devices/0000:01:00.0/driver_override - ''; - blacklistedKernelModules = [ - "r8153_ecm" "cdc_ether" - ]; - }; - services.udev.extraRules = '' - SUBSYSTEM=="usb", ATTRS{idVendor}=="0bda", ATTRS{idProduct}=="8153", OWNER="dan" - ''; - -Then -you can execute :command:`run-border-vm` in a ``buildEnv`` shell, -which starts up QEMU using the NixOS configuration in -:file:`bordervm-configuration.nix`. - -In this VM - -* your Liminix checkout is mounted under :file:`/home/liminix/liminix` - -* TFTP is listening on the ethernet device and serving - :file:`/home/liminix/liminix`. The server IP address is 10.0.0.1 - -* a PPPOE-L2TP relay is running on the same ethernet card. When the - connected Liminix device makes PPPoE requests, the relay spawns - L2TPv2 Access Concentrator sessions to your specified L2TP LNS. - Note that authentication is expected at the PPP layer not the L2TP - layer, so the PAP/CHAP credentials provided by your L2TP service can - be configured into your test device - bordervm doesn't need to know - about them. - -To configure bordervm, you need a file called :file:`bordervm.conf.nix` -which you can create by copying and appropriately editing :file:`bordervm.conf-example.nix` - -.. note:: - - If you make changes to the bordervm configuration after executing - :command:`run-border-vm`, you need to remove the :file:`border.qcow2` disk - image file otherwise the changes won't get picked up. - - -Running tests -************* - -You can run all of the tests by evaluating :file:`ci.nix`, which is the -input I use in Hydra. - -.. code-block:: console - - nix-build -I liminix=`pwd` ci.nix -A pppoe # run one job - nix-build -I liminix=`pwd` ci.nix -A all # run all jobs - - -Troubleshooting -*************** - -Diagnosing unexpectedly large images -==================================== - -Sometimes you can add a package and it causes the image size to balloon -because it has dependencies on other things you didn't know about. Build the -``outputs.manifest`` attribute, which is a JSON representation of the -filesystem, and you can run :command:`nix-store --query` on it. - -.. code-block:: console - - nix-build -I liminix-config=path/to/your/configuration.nix \ - --arg device "import ./devices/qemu" -A outputs.manifest \ - -o manifest - nix-store -q --tree manifest - - -Contributing -************ - -Contributions are welcome, though in these early days there may be a -bit of back and forth involved before patches are merged: -Please get in touch somehow `before` you invest a lot of time into a -code contribution I haven't asked for. Just so I know it's expected -and you're not wasting time doing something I won't accept or have -already started on. - - -Nix language style -================== - -This section describes some Nix language style points that we -attempt to adhere to in this repo. - -* favour ``callPackage`` over raw ``import`` for calling derivations - or any function that may generate one - any code that might need - ``pkgs`` or parts of it. - -* prefer ``let inherit (quark) up down strange charm`` over - ``with quark``, in any context where the scope is more than a single - expression or there is more than one reference to ``up``, ``down`` - etc. ``with pkgs; [ foo bar baz]`` is OK, - ``with lib; stdenv.mkDerivation { ... }`` is usually not. - -* ```` is defined only when running tests, so don't refer to it - in "application" code - -* the parameters to a derivation are sorted alphabetically, except for - ``lib``, ``stdenv`` and maybe other non-package "special cases" - -* indentation is whatever emacs nix-mode says it is. - -* where a ``let`` form defines multiple names, put a newline after the - token ``let``, and indent each name two characters - -* to decide whether some code should be a package or a module? - Packages are self-contained - they live in ``/nix/store/eeeeeee-name`` - and don't directly change system behaviour by their presence or - absense. modules can add to - ``/etc`` or ``/bin`` or other global state, create services, all that - side-effecty stuff. Generally it should be a package unless it - can't be. - - - -Copyright -========= - -The Nix code in Liminix is MIT-licenced (same as Nixpkgs), but the -code it combines from other places (e.g. Linux, OpenWrt) may have a -variety of licences. I have no intention of asking for copyright -assignment: just like when submitting to the Linux kernel you retain -the copyright on the code you contribute. - - -Code of Conduct -=============== - -Please govern yourself in Liminix project venues according to the -`Code of Conduct `_ - - -Where to send patches -===================== - - -Liminix' primary repo is https://gti.telent.net/dan/liminix but you -can't send code there directly because it doesn't have open registrations. - -* There's a `mirror on Github `_ for - convenience and visibility: you can open PRs against that - -* or, you can send me your patch by email using `git send-email `_ - -* or in the future, some day, we will have federated Gitea using - ActivityPub. diff --git a/doc/index.adoc b/doc/index.adoc new file mode 100644 index 0000000..202323f --- /dev/null +++ b/doc/index.adoc @@ -0,0 +1,10 @@ +== Liminix + +intro tutorial installation configuration admin development modules +hardware outputs + +=== Indices and tables + +* `+genindex+` +* `+modindex+` +* `+search+` diff --git a/doc/index.rst b/doc/index.rst deleted file mode 100644 index 83ad8ec..0000000 --- a/doc/index.rst +++ /dev/null @@ -1,24 +0,0 @@ -Liminix -####### - -.. toctree:: - :maxdepth: 3 - :caption: Contents: - - intro - tutorial - installation - configuration - admin - development - modules - hardware - outputs - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` diff --git a/doc/installation.adoc b/doc/installation.adoc new file mode 100644 index 0000000..8a194e5 --- /dev/null +++ b/doc/installation.adoc @@ -0,0 +1,203 @@ +== Installation + +Hardware devices vary wildly in their affordances for installing new +operating systems, so it should be no surprise that the Liminix +installation procedure is hardware-dependent. This section contains +generic instructions, but please refer to the documentation for your +device to find whether and how well they apply. + +=== Building a firmware image + +Liminix uses the Nix language to provide congruent configuration +management. This means that to change anything about the way in which a +Liminix system works, you make that change in your `+configuration.nix+` +(or one of the other files it references), and rerun `+nix-build+` to +action the change. It is not possible (at least, without shenanigans) to +make changes by logging into the device and running imperative commands +whose effects may later be overridden: `+configuration.nix+` always +describes the entire system and can be used to recreate that system at +any time. You can usefully keep it under version control. + +If you are familiar with NixOS, you will notice some similarities +between NixOS and Liminix configuration, and also some differences. +Sometimes the differences are due to the resource-constrained devices we +deploy onto, sometimes due to differences in the uses these devices are +put to. + +For a more full description of how to configure Liminix, see +`+configuration+`. Assuming for the moment that you want a typical home +wireless gateway/router, the best way to get started is to copy +`+examples/rotuer.nix+` and edit it for your requirements. + +[source,console] +---- +$ cp examples/rotuer.nix configuration.nix +$ vi configuration.nix # other editors are available +$ # adjust this next command for your hardware device +$ nix-build -I liminix-config=./configuration.nix \ + --arg device "import ./devices/gl-mt300a" -A outputs.default +---- + +Usually (not always, _please check the documentation for your device_) +this will leave you with a file `+result/firmware.bin+` which you now +need to flash to the device. + +=== Flashing from the boot monitor (TFTP install) + +If you are prepared to open the device and have a TTL serial adaptor of +some kind to connect it to, you can probably use U-Boot and a TFTP +server to download and flash the image. + +This is quite hardware-specific and may even involve soldering - see the +documention for your device. However, it is in some ways the most +"reliable" option: if you can see what's happening (or not happening) in +early boot, the risk of "bricking" is substantially reduced and you have +options for recovering if you misstep or flash a bad image. + +[[serial]] +==== U-Boot and serial shenanigans + +Every device that we have so far encountered in Liminix uses +https://docs.u-boot.org/en/latest/[U-Boot, the "Universal Boot +Loader"] so it's worth knowing a bit about it. "Universal" is in this +context a bit of a misnomer, though: encountering _mainline_ U-Boot is +very rare and often you'll find it is a fork from some version last +updated in 2008. Upgrading U-Boot is more or less complicated depending +on the device and is outside scope for Liminix. + +To speak to U-Boot on your device you'll usually need a serial +connection to it. This typically involves opening the box, locating the +serial header pins (TX, RX and GND) and connecting a USB TTL converter +to them. + +The Rolls Royce of USB/UART cables is the +https://cpc.farnell.com/ftdi/ttl-232r-rpi/cable-debug-ttl-232-usb-rpi/dp/SC12825?st=usb%20to%20uart%20cable[FTDI +cable], but there are cheaper alternatives based on the PL2303 and +CP2102 chipsets - or you could even get creative and use the +https://pinout.xyz/[UART GPIO pins] on a Raspberry Pi. Whatever you do, +make sure that the voltages are compatible: if your device is 3.3V (this +is typical but not universal), you don't want to be sending it 5v or +(even worse) 12v. + +Run a terminal emulator such as Minicom on the computer at other end of +the link. 115200 8N1 is the typical speed. + +[NOTE] +==== +TTL serial connections often have no flow control and so don't always +like having massive chunks of text pasted into them - and U-Boot may +drop characters while it's busy. So don't do that. + +If using Minicom, you may find it helps to bring up the "Termimal +settings" dialog (C^A T), then configure "Newline tx delay" to some +small but non-zero value. +==== + +When you turn the router on you should be greeted with some messages +from U-Boot, followed by the instruction to hit some key to stop +autoboot. Do this and you will get to the prompt. If you didn't see +anything, the strong likelihood is that TX and RX are the wrong way +around. If you see garbage, try a different speed. + +Interesting commands to try first in U-Boot are `+help+` and +`+printenv+`. + +You will also need to configure a TFTP server on a network that's +accessible to the device: how you do that will vary according to which +TFTP server you're using and so is out of scope for this document. + +==== Building and installing the image + +Follow the device-specific instructions for "TFTP install": usually, the +steps are + +* build the [.title-ref]#outputs.mtdimage# output +* copy `+result/firmware.bin+` to your TFTP server +* copy/paste the commands in `+result/flash.scr+` one at a time into the +U-Boot command line +* reset the device + +You should now see messages from U-Boot, then from the Linux kernel and +eventually a shell prompt. + +[NOTE] +==== +Before you reboot, check which networks the device is plugged into, and +disconnect as necessary. If you've just installed a DHCP server or +anything similar that responds to broadcasts, you may not want it to do +that on the network that you temporarily connected it to for installing +it. +==== + +=== Flashing from OpenWrt + +[CAUTION] +==== +Untested! A previous version of these instructions (without the -e flag) +led to bricking the device when flashing a jffs2 image. If you are +reading this message, nobody has yet reported on whether the new +instructions are any better. +==== + +If your device is running OpenWrt then it probably has the `+mtd+` +command installed. Build the [.title-ref]#outputs.mtdimage# output (as +you would for a TFTP install) and then transfer `+result/firmware.bin+` +onto the device using e.g. `+scp+`. Now flash as follows: + +[source,console] +---- +mtd -e -r write /tmp/firmware.bin firmware +---- + +The options to this command are for "erase before writing" and "reboot +after writing". + +For more information, please see the +https://openwrt.org/docs/guide-user/installation/sysupgrade.cli[OpenWrt +manual] which may also contain (hardware-dependent) instructions on how +to flash an image using the vendor firmware - perhaps even from a web +interface. + +=== Flashing from Liminix + +If the device is already running Liminix and has been configured with +`+levitate+`, you can use that to safely flash your new image. Refer to +`+levitate+` for an explanation. + +If the device is running Liminix but doesn't have `+levitate+` your +options are more limited. You may attempt to use `+flashcp+` but it +doesn't always work: as it copies the new image over the top of the +active root filesystem, surprise may ensue. Consider instead using a +serial connection: you may need one anyway after trying flashcp if it +corrupts the image. + +==== flashcp (not generally recommended) + +Connect to the device and locate the "firmware" partition, which you can +do with a combination of `+dmesg+` output and the contents of +`+/proc/mtd+` + +[source,console] +---- +<5>[ 0.469841] Creating 4 MTD partitions on "spi0.0": +<5>[ 0.474837] 0x000000000000-0x000000040000 : "u-boot" +<5>[ 0.480796] 0x000000040000-0x000000050000 : "u-boot-env" +<5>[ 0.487056] 0x000000050000-0x000000060000 : "art" +<5>[ 0.492753] 0x000000060000-0x000001000000 : "firmware" + +# cat /proc/mtd +dev: size erasesize name +mtd0: 00040000 00001000 "u-boot" +mtd1: 00010000 00001000 "u-boot-env" +mtd2: 00010000 00001000 "art" +mtd3: 00fa0000 00001000 "firmware" +mtd4: 002a0000 00001000 "kernel" +mtd5: 00d00000 00001000 "rootfs" +---- + +Copy `+result/firmware.bin+` to the device and now run (in this example) + +[source,console] +---- +flashcp -v firmware.bin /dev/mtd3 +---- diff --git a/doc/installation.rst b/doc/installation.rst deleted file mode 100644 index b72a31d..0000000 --- a/doc/installation.rst +++ /dev/null @@ -1,211 +0,0 @@ -Installation -############ - -Hardware devices vary wildly in their affordances for installing new -operating systems, so it should be no surprise that the Liminix -installation procedure is hardware-dependent. This section contains -generic instructions, but please refer to the documentation for your -device to find whether and how well they apply. - - -Building a firmware image -************************* - -Liminix uses the Nix language to provide congruent configuration -management. This means that to change anything about the way in -which a Liminix system works, you make that change in -your :file:`configuration.nix` (or one of the other files it references), -and rerun :command:`nix-build` to action -the change. It is not possible (at least, without shenanigans) to make -changes by logging into the device and running imperative commands -whose effects may later be overridden: :file:`configuration.nix` -always describes the entire system and can be used to recreate that -system at any time. You can usefully keep it under version control. - -If you are familiar with NixOS, you will notice some similarities -between NixOS and Liminix configuration, and also some -differences. Sometimes the differences are due to the -resource-constrained devices we deploy onto, sometimes due to -differences in the uses these devices are put to. - -For a more full description of how to configure Liminix, see -:ref:`configuration`. Assuming for the moment that you want a typical -home wireless gateway/router, the best way to get started is to copy -:file:`examples/rotuer.nix` and edit it for your requirements. - - -.. code-block:: console - - $ cp examples/rotuer.nix configuration.nix - $ vi configuration.nix # other editors are available - $ # adjust this next command for your hardware device - $ nix-build -I liminix-config=./configuration.nix \ - --arg device "import ./devices/gl-mt300a" -A outputs.default - -Usually (not always, *please check the documentation for your device*) -this will leave you with a file :file:`result/firmware.bin` -which you now need to flash to the device. - - -Flashing from the boot monitor (TFTP install) -********************************************* - -If you are prepared to open the device and have a TTL serial adaptor -of some kind to connect it to, you can probably use U-Boot and a TFTP -server to download and flash the image. - -This is quite hardware-specific and may even involve soldering - see -the documention for your device. However, it is in some ways the most -"reliable" option: if you can see what's happening (or not happening) -in early boot, the risk of "bricking" is substantially reduced and you -have options for recovering if you misstep or flash a bad image. - - -.. _serial: - -U-Boot and serial shenanigans -============================= - -Every device that we have so far encountered in Liminix uses `U-Boot, -the "Universal Boot Loader" `_ so -it's worth knowing a bit about it. "Universal" is in this context a -bit of a misnomer, though: encountering *mainline* U-Boot is very rare -and often you'll find it is a fork from some version last updated -in 2008. Upgrading U-Boot is more or less complicated depending on the -device and is outside scope for Liminix. - -To speak to U-Boot on your device you'll usually need a serial -connection to it. This typically involves opening the box, locating -the serial header pins (TX, RX and GND) and connecting a USB TTL -converter to them. - -The Rolls Royce of USB/UART cables is the `FTDI cable -`_, -but there are cheaper alternatives based on the PL2303 and CP2102 chipsets - or you could even -get creative and use the `UART GPIO pins `_ on a Raspberry Pi. Whatever you do, make sure -that the voltages are compatible: if your device is 3.3V (this is -typical but not universal), you don't want to be sending it 5v or -(even worse) 12v. - -Run a terminal emulator such as Minicom on the computer at other end -of the link. 115200 8N1 is the typical speed. - -.. NOTE:: - - TTL serial connections often have no flow control and - so don't always like having massive chunks of text pasted into - them - and U-Boot may drop characters while it's busy. So don't - do that. - - If using Minicom, you may find it helps to bring up the "Termimal - settings" dialog (C^A T), then configure "Newline tx delay" to - some small but non-zero value. - -When you turn the router on you should be greeted with some messages -from U-Boot, followed by the instruction to hit some key to stop -autoboot. Do this and you will get to the prompt. If you didn't see -anything, the strong likelihood is that TX and RX are the wrong way -around. If you see garbage, try a different speed. - -Interesting commands to try first in U-Boot are :command:`help` and -:command:`printenv`. - -You will also need to configure a TFTP server on a network that's -accessible to the device: how you do that will vary according to which -TFTP server you're using and so is out of scope for this document. - - - -Building and installing the image -================================= - -Follow the device-specific instructions for "TFTP install": usually, -the steps are - -* build the `outputs.mtdimage` output -* copy :file:`result/firmware.bin` to your TFTP server -* copy/paste the commands in :file:`result/flash.scr` one at a time into the U-Boot command line -* reset the device - -You should now see messages from U-Boot, then from the Linux kernel -and eventually a shell prompt. - -.. NOTE:: Before you reboot, check which networks the device is - plugged into, and disconnect as necessary. If you've just - installed a DHCP server or anything similar that responds to - broadcasts, you may not want it to do that on the network - that you temporarily connected it to for installing it. - - - -Flashing from OpenWrt -********************* - -.. CAUTION:: Untested! A previous version of these instructions - (without the -e flag) led to bricking the device - when flashing a jffs2 image. If you are reading - this message, nobody has yet reported on whether the - new instructions are any better. - -If your device is running OpenWrt then it probably has the -:command:`mtd` command installed. Build the `outputs.mtdimage` output -(as you would for a TFTP install) and then transfer -:file:`result/firmware.bin` onto the device using e.g. -:command:`scp`. Now flash as follows: - -.. code-block:: console - - mtd -e -r write /tmp/firmware.bin firmware - -The options to this command are for "erase before writing" and "reboot -after writing". - -For more information, please see the `OpenWrt manual `_ which may also contain (hardware-dependent) instructions on how to flash an image using the vendor firmware - perhaps even from a web interface. - - -Flashing from Liminix -********************* - -If the device is already running Liminix and has been configured with -:command:`levitate`, you can use that to safely flash your new image. -Refer to :ref:`levitate` for an explanation. - -If the device is running Liminix but doesn't have :command:`levitate` -your options are more limited. You may attempt to use -:command:`flashcp` but it doesn't always work: as it copies the new -image over the top of the active root filesystem, surprise may ensue. -Consider instead using a serial connection: you may need one anyway -after trying flashcp if it corrupts the image. - -flashcp (not generally recommended) -=================================== - -Connect to the device and locate the "firmware" partition, which you -can do with a combination of :command:`dmesg` output and the contents -of :file:`/proc/mtd` - -.. code-block:: console - - <5>[ 0.469841] Creating 4 MTD partitions on "spi0.0": - <5>[ 0.474837] 0x000000000000-0x000000040000 : "u-boot" - <5>[ 0.480796] 0x000000040000-0x000000050000 : "u-boot-env" - <5>[ 0.487056] 0x000000050000-0x000000060000 : "art" - <5>[ 0.492753] 0x000000060000-0x000001000000 : "firmware" - - # cat /proc/mtd - dev: size erasesize name - mtd0: 00040000 00001000 "u-boot" - mtd1: 00010000 00001000 "u-boot-env" - mtd2: 00010000 00001000 "art" - mtd3: 00fa0000 00001000 "firmware" - mtd4: 002a0000 00001000 "kernel" - mtd5: 00d00000 00001000 "rootfs" - -Copy :file:`result/firmware.bin` to the device and now run (in this -example) - -.. code-block:: console - - flashcp -v firmware.bin /dev/mtd3 - - diff --git a/doc/intro.adoc b/doc/intro.adoc new file mode 100644 index 0000000..bc7c375 --- /dev/null +++ b/doc/intro.adoc @@ -0,0 +1,14 @@ +== Introduction + +Liminix is a Nix-based collection of software tailored for domestic wifi +router or IoT device devices, of the kind that OpenWrt or DD-WRT or +Gargoyle or Tomato run on. + +This is not NixOS-on-your-router: it's aimed at devices that are +underpowered for the full NixOS experience. It uses busybox tools, musl +instead of GNU libc, and s6-rc instead of systemd. + +The Liminix name comes from Liminis, in Latin the genitive declension of +"limen", or "of the threshold". Your router stands at the threshold of +your (online) home and everything you send to/receive from the outside +word goes across it. diff --git a/doc/intro.rst b/doc/intro.rst deleted file mode 100644 index 04a62c5..0000000 --- a/doc/intro.rst +++ /dev/null @@ -1,15 +0,0 @@ -Introduction -############ - -Liminix is a Nix-based collection of software tailored for domestic -wifi router or IoT device devices, of the kind that OpenWrt or DD-WRT -or Gargoyle or Tomato run on. - -This is not NixOS-on-your-router: it's aimed at devices that are -underpowered for the full NixOS experience. It uses busybox tools, -musl instead of GNU libc, and s6-rc instead of systemd. - -The Liminix name comes from Liminis, in Latin the genitive declension -of "limen", or "of the threshold". Your router stands at the threshold -of your (online) home and everything you send to/receive from the -outside word goes across it. diff --git a/doc/modules.adoc b/doc/modules.adoc new file mode 100644 index 0000000..32fdc0f --- /dev/null +++ b/doc/modules.adoc @@ -0,0 +1 @@ +== Module options diff --git a/doc/modules.rst b/doc/modules.rst deleted file mode 100644 index 5d1187e..0000000 --- a/doc/modules.rst +++ /dev/null @@ -1,4 +0,0 @@ -Module options -############## - -.. include:: modules-generated.inc.rst diff --git a/doc/outputs.rst b/doc/outputs.adoc similarity index 58% rename from doc/outputs.rst rename to doc/outputs.adoc index 477a69a..677cb38 100644 --- a/doc/outputs.rst +++ b/doc/outputs.adoc @@ -1,13 +1,10 @@ -Outputs -####### +== Outputs -Liminix *outputs* are artefacts that can be installed somehow on a -target device, or "installers" which run on the target device to -perform the installation. +Liminix _outputs_ are artefacts that can be installed somehow on a +target device, or "installers" which run on the target device to perform +the installation. There are different outputs because different target devices need different artefacts, or have different ways to get that artefact -installed. The options available for a particular device are described in -the section for that device. - -.. include:: outputs-generated.inc.rst +installed. The options available for a particular device are described +in the section for that device. diff --git a/doc/tutorial.adoc b/doc/tutorial.adoc new file mode 100644 index 0000000..37d21ec --- /dev/null +++ b/doc/tutorial.adoc @@ -0,0 +1,322 @@ +== Tutorial + +Liminix is very configurable, which can make it initially quite daunting +- especially if you're learning Nix or Linux or networking concepts at +the same time. In this section we build some "worked example" Liminix +images to introduce the concepts. If you follow the examples exactly, +they should work. If you change things as you go along, they may work +differently or not at all, but the experience should be educational +either way. + +=== Requirements + +You will need a reasonably powerful computer running Nix. Target devices +for Liminix are unlikely to have the CPU power and disk space to be able +to build it in situ, so the build process is based around +"cross-compilation" from another computer. The build machine can be any +reasonably powerful desktop/laptop/server PC running NixOS. Standalone +Nixpkgs installations on other Linux distributions - or on MacOS, or +even in a Docker container - also ought to work but are untested. + +=== Running in Qemu + +You can try out Liminix without even having a router to play with. Clone +the Liminix git repository and change into its directory + +[source,console] +---- +git clone https://gti.telent.net/dan/liminix +cd liminix +---- + +Now build Liminix + +[source,console] +---- +nix-build -I liminix-config=./examples/hello-from-qemu.nix \ + --arg device "import ./devices/qemu" -A outputs.default +---- + +In this command `+liminix-config+` points to the desired software +configuration (e.g. services, users, filesystem, secrets) and `+device+` +describes the hardware (or emulated hardware) to run it on. +`+outputs.default+` tells Liminix that we want the default image output +for flashing to the device: for the Qemu "hardware" it's an alias for +`+outputs.vmbuild+`, which creates a directory containing a root +filesystem image and a kernel. + +[TIP] +==== +The first time you run this it may take several hours, because it builds +all of the dependencies including a full MIPS gcc and library toolchain. +Once those intermediate build products are in the nix store, subsequent +builds will be much faster - practically instant, if nothing has +changed. +==== + +Now you can try it: + +[source,console] +---- +./result/run.sh +---- + +This starts the Qemu emulator with a bunch of useful options, to run the +Liminix configuration you just built. It connects the emulated device's +serial console and the +https://www.qemu.org/docs/master/system/monitor.html[QEMU monitor] to +stdin/stdout. + +You should now see Linux boot messages and after a few seconds be +presented with a root shell prompt. You can run commands to look at the +filesystem, see what processes are running, view log messages (in +:file:/run/log/current), etc. To kill the emulator, press ^P (Control P) +then c to enter the "QEMU Monitor", then type `+quit+` at the `+(qemu)+` +prompt. + +To see that it's running network services we need to connect to its +emulated network. Start the machine again, if you had stopped it, and +open up a second terminal on your build machine. We're going to run +another virtual machine attached to the virtual network, which will +request an IP address from our Liminix system and give you a shell you +can run ssh from. + +We use https://www.system-rescue.org/[System Rescue] in tty mode (no +graphical output) for this example, but if you have some other favourite +Linux Live CD ISO - or, for that matter, any other OS image that QEMU +can boot - adjust the command to suit. + +Download the System Rescue ISO: + +[source,console] +---- +curl https://fastly-cdn.system-rescue.org/releases/10.01/systemrescue-10.01-amd64.iso -O +---- + +and run it + +[source,console] +---- +nix-shell -p qemu --run " \ +qemu-system-x86_64 \ +-echr 16 \ +-m 1024 \ +-cdrom systemrescue-10.01-amd64.iso \ +-netdev socket,mcast=230.0.0.1:1235,localaddr=127.0.0.1,id=lan \ +-device virtio-net,disable-legacy=on,disable-modern=off,netdev=lan,mac=ba:ad:3d:ea:21:01 \ +-display none -serial mon:stdio" +---- + +System Rescue displays a boot menu at which you should select the +"serial console" option, then after a few moments it boots to a root +prompt. You can now try things out: + +* run `+ip a+` and see that it's been allocated an IP address in the +range 10.3.0.0/16. +* run `+ping 10.3.0.1+` to see that the Liminix VM responds +* run `+ssh root@10.3.0.1+` to try logging into it. + +Congratulations! You have installed your first Liminix system - albeit +it has no practical use and it's not even real. The next step is to try +running it on hardware. + +=== Installing on hardware + +For the next example, we're going to install onto an actual hardware +device. These steps have been tested using a GL.iNet GL-MT300A, which +has been chosen for the purpose because it's cheap and easy to unbrick +if necessary. + +[WARNING] +==== +There is always a risk of rendering your device unbootable by flashing +it with an image that doesn't work. The GL-MT300A has a builtin +"debrick" procedure in the boot monitor and is also comparatively simple +to attach serial cables to (soldering not required), so it is lower-risk +than some devices. Using some other Liminix-supported MIPS hardware +device also _ought_ to work here, but you accept the slightly greater +bricking risk if it doesn't. +==== + +See `+hardware+` for device support status. + +You may want to read and inwardly digest the Develoment Manual section +`+serial+` when you start working with Liminix on real hardware. You +won't _need_ serial access for this example, assuming it works, but it +allows you to see the boot monitor and kernel messages, and to login +directly to the device if for some reason it doesn't bring its network +up. + +Now we can build Liminix. Although we could use the same example +configuration as we did for Qemu, you might not want to plug a DHCP +server into your working LAN because it will compete with the real DHCP +service. So we're going to use a different configuration with a DHCP +client: this is `+examples/hello-from-mt300.nix+` + +It's instructive to compare the two configurations: + +[source,console] +---- +diff -u examples/hello-from-qemu.nix examples/hello-from-mt300.nix +---- + +You'll see a new `+boot.tftp+` stanza which you can ignore, +`+services.dns+` has been removed, and the static IP address allocation +has been replaced by a `+dhcp.client+` service. + +[source,console] +---- +nix-build -I liminix-config=./examples/hello-from-mt300.nix \ + --arg device "import ./devices/gl-mt300a" -A outputs.default +---- + +[TIP] +==== +The first time you run this it may take several hours. Again? Yes, even +if you ran the previous example. Qemu is set up as a big-endian system +whereas the MediaTek SoC on this device is little-endian - so it +requires building all of the dependencies including an entirely +different MIPS gcc and library toolchain to the other one. +==== + +This time in `+result/+` you will see a bunch of files. Most of them you +can ignore for the moment, but `+result/firmware.bin+` is the firmware +image you can flash. + +==== Flashing + +Again, there are a number of different ways you could do this: using +TFTP with a serial cable, through the stock firmware's web UI, or using +the https://docs.gl-inet.com/router/en/3/tutorials/debrick/[vendor's +"debrick" process]. The last of these options has a lot to recommend it +for a first attempt: + +* it works no matter what firmware is currently installed +* it doesn't require plugging a router into the same network as your +build system and potentially messing up your actual upstream +* no need to open the device and add cables + +You can read detailed instructions on the vendor site, but the short +version is: + +[arabic] +. turn the device off +. connect it by ethernet cable to a computer +. configure the computer to have static ip address 192.168.1.10 +. while holding down the Reset button, turn the device on +. after about five seconds you can release the Reset button +. visit http://192.168.1.1/ using a web browser on the connected +computer +. click on "Browse" and choose `+result/firmware.bin+` +. click on "Update firmware" +. wait a minute or so while it updates. + +There's no feedback from the web interface when the flashing is +finished, but what should happen is that the router reboots and starts +running Liminix. Now you need to figure out what address it got from +DHCP - e.g. by checking the DHCP server logs, or maybe by pinging +`+hello.lan+` or something. Once you've found it on the network you can +ping it and ssh to it just like you did the Qemu example, but this time +for real. + +[WARNING] +==== +Do not leave the default root password in place on any device exposed to +the internet! Although it has no writable storage and no default route, +a motivated attacker with some imagination could probably still do +something awful using it. +==== + +Congratulations Part II! You have installed your first Liminix system on +actual hardware - albeit that it _still_ has no practical use. + +Exercise for the reader: change the default password by editing +`+examples/hello-from-mt300.nix+`, and then create and upload a new +image that has it set to something less hopeless. + +=== Routing + +The third example `+examples/demo.nix+` is a fully-functional home "WiFi +router" - although you will have to edit it a bit before it will +actually work for you. Copy `+examples/demo.nix+` to `+my-router.nix+` +(or other name of your choice) and open it in your favourite text +editor. Everywhere that the text `+EDIT+` appears is either a place you +probably want to change or a place you almost certainly need to change. + +There's a lot going on in this configuration: + +* it provides a wireless access point using the `+hostapd+` service: in +this stanza you can change the ssid, the channel, the passphrase etc. +* the wireless lan and wired lan are bridged together with the +`+bridge+` service, so that your wired and wireless clients appear to be +on the same network. + +[TIP] +==== +If you were using a hardware device that provides both 2.4GHz and 5GHz +wifi, you'd probably find that it has two wireless devices (often called +wlan0 and wlan1). In Liminix we handle this by running two `+hostapd+` +services, and adding both of them to the network bridge along with the +wired lan. (You can see an example in `+examples/rotuer.nix+`) +==== + +* we use the combination DNS and DHCP daemon provided by the `+dnsmasq+` +service, which you can configure +* the upstream network is "PPP over Ethernet", provided by the `+pppoe+` +service. Assuming that your ISP uses this standard, they will have +provided you with a PPP username and password (sometimes this will be +listed as "PAP" or "CHAP") which you can edit into the configuration +* this example supports the +newfootnote:[https://datatracker.ietf.org/doc/html/rfc1883[RFC1883 +Internet Protocol, Version 6] was published in 1995, so only "new" +when Bill Clinton was US President] Internet Protocol v6 as well as +traditional IPv4. Configuring IPv6 seems to vary from one ISP to the +next: this example expects them to be providing IP address allocation +and "prefix delegation" using DHCP6. + +Build it using the same method as the previous example + +[source,console] +---- +nix-build -I liminix-config=./my-router.nix \ + --arg device "import ./devices/gl-mt300a" -A outputs.default +---- + +and then you can flash it to the device. + +==== Bonus: in-place updates + +This configuration uses a writable filesystem (see the line +`+rootfsType = "jffs2"+`), which means that once you've flashed it for +the first time, you can make further updates over SSH onto the running +router. To try this, make a small change (I'd suggest changing the +hostname) and then run + +[source,console] +---- +nix-build -I liminix-config=./my-router.nix \ + --arg device "import ./devices/gl-ar750" \ + -A outputs.systemConfiguration && \ +result/install.sh root@address-of-the-device +---- + +(This requires the device to be network-accessible from your build +machine, which for a test/demo system might involve a second network +device in your build system - USB ethernet adapters are cheap - or a bit +of messing around unplugging cables.) + +For more information about in-place-updates, see the manual section +`+Rebuilding the system+`. + +=== Final thoughts + +* These are demonstration configs for pedagogical purposes. If you'd +like to see some more realistic uses of Liminix, +`+examples/rotuer,arhcive,extneder.nix+` are based on some actual real +hosts in my home network. +* The technique used here for flashing was chosen mostly because it +doesn't need much infrastructure/tooling, but it is a bit of a faff +(requires physical access, vendor specific). There are slicker ways to +do it that need a bit more setup - we'll talk about that later as well. + +*Footnotes* diff --git a/doc/tutorial.rst b/doc/tutorial.rst deleted file mode 100644 index b8a02a3..0000000 --- a/doc/tutorial.rst +++ /dev/null @@ -1,327 +0,0 @@ -Tutorial -######## - -Liminix is very configurable, which can make it initially quite -daunting - especially if you're learning Nix or Linux or networking -concepts at the same time. In this section we build some "worked -example" Liminix images to introduce the concepts. If you follow the -examples exactly, they should work. If you change things as you go -along, they may work differently or not at all, but the experience -should be educational either way. - - -Requirements -************ - -You will need a reasonably powerful computer running Nix. Target -devices for Liminix are unlikely to have the CPU power and disk space -to be able to build it in situ, so the build process is based around -"cross-compilation" from another computer. The build machine can be -any reasonably powerful desktop/laptop/server PC running NixOS. -Standalone Nixpkgs installations on other Linux distributions - or on -MacOS, or even in a Docker container - also ought to work but are -untested. - - -Running in Qemu -*************** - -You can try out Liminix without even having a router to play with. -Clone the Liminix git repository and change into its directory - - -.. code-block:: console - - git clone https://gti.telent.net/dan/liminix - cd liminix - -Now build Liminix - -.. code-block:: console - - nix-build -I liminix-config=./examples/hello-from-qemu.nix \ - --arg device "import ./devices/qemu" -A outputs.default - -In this command ``liminix-config`` points to the desired software -configuration (e.g. services, users, filesystem, secrets) and -``device`` describes the hardware (or emulated hardware) to run it on. -``outputs.default`` tells Liminix that we want the default image -output for flashing to the device: for the Qemu "hardware" it's an -alias for ``outputs.vmbuild``, which creates a directory containing a -root filesystem image and a kernel. - -.. tip:: The first time you run this it may take several hours, - because it builds all of the dependencies including a full - MIPS gcc and library toolchain. Once those intermediate build - products are in the nix store, subsequent builds will be much - faster - practically instant, if nothing has changed. - -Now you can try it: - -.. code-block:: console - - ./result/run.sh - -This starts the Qemu emulator with a bunch of useful options, to run -the Liminix configuration you just built. It connects the emulated -device's serial console and the `QEMU monitor -`_ to -stdin/stdout. - -You should now see Linux boot messages and after a few seconds be -presented with a root shell prompt. You can run commands to look at -the filesystem, see what processes are running, view log messages (in -:file:/run/log/current), etc. To kill the emulator, press ^P -(Control P) then c to enter the "QEMU Monitor", then type ``quit`` at -the ``(qemu)`` prompt. - -To see that it's running network services we need to connect to its -emulated network. Start the machine again, if you had stopped it, and -open up a second terminal on your build machine. We're going to run -another virtual machine attached to the virtual network, which will -request an IP address from our Liminix system and give you a shell you -can run ssh from. - -We use `System Rescue `_ in tty -mode (no graphical output) for this example, but if you have some -other favourite Linux Live CD ISO - or, for that matter, any other OS -image that QEMU can boot - adjust the command to suit. - -Download the System Rescue ISO: - -.. code-block:: console - - curl https://fastly-cdn.system-rescue.org/releases/10.01/systemrescue-10.01-amd64.iso -O - -and run it - -.. code-block:: console - - nix-shell -p qemu --run " \ - qemu-system-x86_64 \ - -echr 16 \ - -m 1024 \ - -cdrom systemrescue-10.01-amd64.iso \ - -netdev socket,mcast=230.0.0.1:1235,localaddr=127.0.0.1,id=lan \ - -device virtio-net,disable-legacy=on,disable-modern=off,netdev=lan,mac=ba:ad:3d:ea:21:01 \ - -display none -serial mon:stdio" - -System Rescue displays a boot menu at which you should select the -"serial console" option, then after a few moments it boots to a root -prompt. You can now try things out: - -* run :command:`ip a` and see that it's been allocated an IP address in the range 10.3.0.0/16. - -* run :command:`ping 10.3.0.1` to see that the Liminix VM responds - -* run :command:`ssh root@10.3.0.1` to try logging into it. - -Congratulations! You have installed your first Liminix system - albeit -it has no practical use and it's not even real. The next step is to try -running it on hardware. - -Installing on hardware -********************** - -For the next example, we're going to install onto an actual hardware -device. These steps have been tested using a GL.iNet GL-MT300A, which -has been chosen for the purpose because it's cheap and easy to -unbrick if necessary. - -.. warning:: There is always a risk of rendering your device - unbootable by flashing it with an image that doesn't - work. The GL-MT300A has a builtin "debrick" procedure in - the boot monitor and is also comparatively simple to - attach serial cables to (soldering not required), so it - is lower-risk than some devices. Using some other - Liminix-supported MIPS hardware device also *ought* to - work here, but you accept the slightly greater bricking - risk if it doesn't. - -See :doc:`hardware` for device support status. - -You may want to read and inwardly digest the Develoment Manual section -:ref:`serial` when you start working with Liminix on real hardware. You -won't *need* serial access for this example, assuming it works, but it -allows you -to see the boot monitor and kernel messages, and to login directly to -the device if for some reason it doesn't bring its network up. - -Now we can build Liminix. Although we could use the same example -configuration as we did for Qemu, you might not want to plug a DHCP -server into your working LAN because it will compete with the real -DHCP service. So we're going to use a different configuration with a -DHCP client: this is :file:`examples/hello-from-mt300.nix` - -It's instructive to compare the two configurations: - -.. code-block:: console - - diff -u examples/hello-from-qemu.nix examples/hello-from-mt300.nix - -You'll see a new ``boot.tftp`` stanza which you can ignore, -``services.dns`` has been removed, and the static IP address allocation -has been replaced by a ``dhcp.client`` service. - -.. code-block:: console - - nix-build -I liminix-config=./examples/hello-from-mt300.nix \ - --arg device "import ./devices/gl-mt300a" -A outputs.default - -.. tip:: The first time you run this it may take several hours. - Again? Yes, even if you ran the previous example. Qemu is - set up as a big-endian system whereas the MediaTek SoC - on this device is little-endian - so it requires building - all of the dependencies including an entirely different - MIPS gcc and library toolchain to the other one. - -This time in :file:`result/` you will see a bunch of files. Most of -them you can ignore for the moment, but :file:`result/firmware.bin` is -the firmware image you can flash. - - -Flashing -======== - -Again, there are a number of different ways you could do this: using -TFTP with a serial cable, through the stock firmware's web UI, or -using the `vendor's "debrick" process -`_. The last -of these options has a lot to recommend it for a first attempt: - -* it works no matter what firmware is currently installed - -* it doesn't require plugging a router into the same network as your - build system and potentially messing up your actual upstream - -* no need to open the device and add cables - -You can read detailed instructions on the vendor site, but the short version is: - -1. turn the device off -2. connect it by ethernet cable to a computer -3. configure the computer to have static ip address 192.168.1.10 -4. while holding down the Reset button, turn the device on -5. after about five seconds you can release the Reset button -6. visit http://192.168.1.1/ using a web browser on the connected computer -7. click on "Browse" and choose :file:`result/firmware.bin` -8. click on "Update firmware" -9. wait a minute or so while it updates. - -There's no feedback from the web interface when the flashing is -finished, but what should happen is that the router reboots and -starts running Liminix. Now you need to figure out what address it got -from DHCP - e.g. by checking the DHCP server logs, or maybe by pinging -``hello.lan`` or something. Once you've found it on the -network you can ping it and ssh to it just like you did the Qemu -example, but this time for real. - -.. warning:: Do not leave the default root password in place on any - device exposed to the internet! Although it has no - writable storage and no default route, a motivated attacker - with some imagination could probably still do something - awful using it. - -Congratulations Part II! You have installed your first Liminix system on actual hardware - albeit that it *still* has no practical use. - -Exercise for the reader: change the default password by editing -:file:`examples/hello-from-mt300.nix`, and then create and upload a -new image that has it set to something less hopeless. - -Routing -******* - -The third example :file:`examples/demo.nix` is a fully-functional home -"WiFi router" - although you will have to edit it a bit before it will -actually work for you. Copy :file:`examples/demo.nix` to -:file:`my-router.nix` (or other name of your choice) and open it in -your favourite text editor. Everywhere that the text :code:`EDIT` -appears is either a place you probably want to change or a place you -almost certainly need to change. - -There's a lot going on in this configuration: - -* it provides a wireless access point using the :code:`hostapd` - service: in this stanza you can change the ssid, the channel, - the passphrase etc. - -* the wireless lan and wired lan are bridged together with the - :code:`bridge` service, so that your wired and wireless clients appear - to be on the same network. - -.. tip:: If you were using a hardware device that provides both 2.4GHz - and 5GHz wifi, you'd probably find that it has two wireless - devices (often called wlan0 and wlan1). In Liminix we handle - this by running two :code:`hostapd` services, and adding - both of them to the network bridge along with the wired lan. - (You can see an example in :file:`examples/rotuer.nix`) - -* we use the combination DNS and DHCP daemon provided by the - :code:`dnsmasq` service, which you can configure - -* the upstream network is "PPP over Ethernet", provided by the - :code:`pppoe` service. Assuming that your ISP uses this standard, - they will have provided you with a PPP username and password - (sometimes this will be listed as "PAP" or "CHAP") which you can edit - into the configuration - -* this example supports the new [#ipv6]_ Internet Protocol v6 - as well as traditional IPv4. Configuring IPv6 seems to - vary from one ISP to the next: this example expects them - to be providing IP address allocation and "prefix delegation" - using DHCP6. - -Build it using the same method as the previous example - - -.. code-block:: console - - nix-build -I liminix-config=./my-router.nix \ - --arg device "import ./devices/gl-mt300a" -A outputs.default - -and then you can flash it to the device. - - -Bonus: in-place updates -======================= - -This configuration uses a writable filesystem (see the line -:code:`rootfsType = "jffs2"`), which means that once you've flashed it -for the first time, you can make further updates over SSH onto the -running router. To try this, make a small change (I'd suggest changing -the hostname) and then run - -.. code-block:: console - - nix-build -I liminix-config=./my-router.nix \ - --arg device "import ./devices/gl-ar750" \ - -A outputs.systemConfiguration && \ - result/install.sh root@address-of-the-device - -(This requires the device to be network-accessible from your build -machine, which for a test/demo system might involve a second network -device in your build system - USB ethernet adapters are cheap - or -a bit of messing around unplugging cables.) - -For more information about in-place-updates, see the manual section :ref:`Rebuilding the system`. - - -Final thoughts -************** - -* These are demonstration configs for pedagogical purposes. If you'd - like to see some more realistic uses of Liminix, - :file:`examples/rotuer,arhcive,extneder.nix` are based on some - actual real hosts in my home network. - -* The technique used here for flashing was chosen mostly because it - doesn't need much infrastructure/tooling, but it is a bit of a faff - (requires physical access, vendor specific). There are slicker ways - to do it that need a bit more setup - we'll talk about that later as - well. - - - -.. rubric:: Footnotes - -.. [#ipv6] `RFC1883 Internet Protocol, Version 6 `_ was published in 1995, so only "new" when Bill Clinton was US President