use Linux kernel sources associated with openwrt by default

think
rt3200 test install
2024-12-24 12:30:15 +00:00 · 2024-12-24 12:30:15 +00:00 · 2024-12-23 23:59:52 +00:00 · 2024-12-23 22:37:07 +00:00 · 2024-12-23 22:37:07 +00:00 · 2024-12-23 22:37:07 +00:00
48 changed files with 628 additions and 292 deletions
@@ -132,4 +132,28 @@ and then run the generated `install.sh` script

    result/install.sh root@192.168.8.1

+2024-12-16

+Config options changed: if you had set config.hardware.dts.includes
+(maybe in an out-of-tree device port) to specify the search paths
+in which dtc finds include files, you will need to change this to 
+hardware.dts.includePaths.
+
+The "new" hardware.dts.includes option is now for dtsi files which
+should be merged into the device tree.
+
+2024-12-19
+
+Incremental updates changed again (but not massively). From hereon in,
+the preferred way to do an incremental update on an installed device
+with a writable filesystem is to build the updater output
+
+    nix-build  -I liminix-config=hosts/myhost.nix --argstr deviceName turris-omnia -A outputs.updater
+
+and then run the generated `update.sh` script. See 
+https://www.liminix.org/doc/admin.html#updating-an-installed-system
+
+2024-12-22
+
+outputs.zimage is now outputs.kernel.zImage. This is unlikely to
+affect many people at all but I mention it anyway.
@@ -6539,3 +6539,134 @@ using a Makefile

 idea 2: when a configuration contains levitate, something similar
 but necessarily more "manual" to do the analogous thing
+
+
+Sun Dec 15 18:55:55 GMT 2024
+
+Where we left off with this, rotuer was crashing randomly or failing
+to boot every time we tried to add log shipping, which is not very
+ideal. I started doing something with logging to /dev/pmsg0
+(CONFIG_PSTORE_PMSG) but I think (there seems not to be anything
+written down :-( ) that the gl-ar750 kernel needs it added to kconfig and device tree
+
+https://wiki.postmarketos.org/wiki/User:Knuxify/Enabling_pstore_and_ramoops
+
+we could add a new hardware.dts.dtsi = [] option so that any module
+could add a new chunk of dts. (Ideally we'd call it `includes`
+but that conflicts with the existing use of `includes` to specify
+search path. Maybe rename?)
+
+would we ever use it except in a hardware device definition?
+(Or user config?) I guess if we were consistent with names
+then we could set up nodes in the device file with status="disabled"
+and enable them in the module, except that dt doesn't consistently
+use status and in fact there isn't one for reserved-memory
+
+we could use global config to enable pstore_msg and check it in
+the device module to enable the needed hw support
+
+Tue Dec 17 23:39:28 GMT 2024
+
+I think we can just stick a tee in the fallback logger pipeline that
+writes to /dev/pmsg0
+
+Need to check it's a circular buffer
+
+do we want to do anything about recovering the log on boot?
+- we could just copy it to /run/log
+- if we have backfilling for shipped logs (we don't yet)
+  then we might want to ship it - but that may result in duplicate
+  logs if some of it was shipped before the crash
+
+perhaps we should truncate pmsg0 on orderly shutdown? or maybe it's
+good to see the late shutdown logs.
+
+Thu Dec 19 13:40:39 GMT 2024
+
+although we have PSTORE_foo in the omnia kconfig, I think this might
+be just because I copied it from RT3200
+
+Thu Dec 19 14:15:43 GMT 2024
+
+Omnia is not in ci.nix, and it's not trivial to add it because there
+is no output in the ci.nix configuration that makes sense for omnia.
+
+... OK, fixed by adding system-configuration as an independent module
+and importing in device config
+
+Thu Dec 19 21:59:47 GMT 2024
+
+The build-system shell script in outputs.systemConfiguration
+is ugly and requires we do bad things to avoid sucking build
+system stuff into the config
+
+I propose we make it a separate derivation.
+
+But first maybe we could improve some names
+
+Sun Dec 22 14:23:02 GMT 2024
+
+MT7622> echo $boot_default
+if env exists flag_recover ; then else run bootcmd ; fi ; run boot_recovery ; setenv replacevol 1 ; run boot_tftp_for
+ever
+MT7622> echo $bootcmd
+if pstore check ; then run boot_recovery ; else run boot_ubi ; fi
+MT7622> echo $boot_ubi
+ubi part ubi && run boot_production ; run boot_recovery
+MT7622> echo $boot_production
+led $bootled_pwr on ; run ubi_read_production && bootm $loadaddr#$bootconf ; led $bootled_pwr off
+MT7622> echo $ubi_read_production
+ubi read $loadaddr fit && iminfo $loadaddr && run ubi_prepare_rootfs
+MT7622> echo $ubi_prepare_rootfs
+if ubi check rootfs_data ; then else if env exists rootfs_data_max ; then ubi create rootfs_data $rootfs_data_max dynamic || ubi create rootfs_data - dynamic ; else ubi create rootfs_data - dynamic ; fi ; fi
+MT7622> echo $bootconf
+config-1
+MT7622> run boot_ubi
+UBI partition 'ubi' already selected
+No size specified -> Using max size (126976)
+Read 126976 bytes from volume fit to 0000000048000000
+
+## Checking Image at 48000000 ...
+Unknown image format!
+No size specified -> Using max size (7491584)
+Read 7491584 bytes from volume recovery to 0000000048000000
+## Loading kernel from FIT Image at 48000000 ...
+   Using 'config-1' configuration
+   Trying 'kernel-1' kernel subimage
+     Description:  ARM64 OpenWrt Linux-6.6.45
+
+Sun Dec 22 17:39:56 GMT 2024
+
+From the output above it looks like the device I have plugged in has
+an openwrt "recovery" image but not a "production" image
+
+It also looks like it will be quite hard work to persuade it to boot
+from usb or anything. It doesn't have any of the extlinux stuff
+but it does have uefi for what that's worth
+
+default boot_production command reads a ubi volume called 'fit' and
+calls bootm on what it finds
+
+we could define boot_production to ubifsmount liminix; ubifs load
+<addr> <filename> (which is a fit) and bootm it. *presumably* we could
+do this from the openwrt recovery image
+
+but could we install the whole system using said recovery image?  I
+expect we could do, it only requires getting a tarball onto it and
+unpacking it
+
+however, extlinux is not going to be helpful
+(actually it might be a bit, if we ask it to write the fit as
+well as/instead of the individual files)
+
+maybe we need separate concepts of "the filesystem contains
+stuff we need for boot" and "the stuff we need is the stuff
+that extlinux needs"
+
+each bootloader makes an output called bootfiles, and
+the bootablerootdir output copies from bootfiles
+
+Mon Dec 23 18:28:50 GMT 2024
+
+it might be worth moving ubi option decls into the hardware module, if
+they're hardware-dependent
@@ -102,12 +102,18 @@ in {
    systemd.services.sshd.wantedBy = pkgs.lib.mkForce [ "multi-user.target" ];

    virtualisation = {
-      forwardPorts = [ {
-        from = "host";
-        host.port = 7654;
-#        guest.address = "10.0.2.15";
-        guest.port =7654;
-      } ];
+      forwardPorts = [
+        {
+          from = "host";
+          host.port = 7654;
+          #        guest.address = "10.0.2.15";
+          guest.port =7654;
+        }
+        {
+          host.port = 2222;
+          guest.address = "10.0.2.15";
+          guest.port = 22;
+        }];
      qemu = {
        networkingOptions = [ ];
        options =
@@ -12,6 +12,7 @@ let
    "qemu-armv7l"
    "tp-archer-ax23"
    "zyxel-nwa50ax"
+    "turris-omnia"
  ];
  vanilla = ./vanilla-configuration.nix;
  for-device = name:
@@ -7,12 +7,10 @@
    and is "work in progress" in Liminix.

    .. note:: The factory flash image contains ECC errors that make it
-              incompatible with Liminix: you need to use the `OpenWrt
+              incompatible with Liminix: use the `OpenWrt
              UBI Installer <https://github.com/dangowrt/owrt-ubi-installer>`_ to
-              rewrite the partition layout before you can flash
-              Liminix onto it (or even use it with
-              :ref:`system-outputs-tftpboot`, if you want the wireless
-              to work).
+              rewrite the partition layout before you can use
+              Liminix with it

    Hardware summary
    ================
@@ -27,8 +25,33 @@
    Installation
    ============

-    Installation is currently a manual process (you need a :ref:`serial <serial>` conection and
-    TFTP) following the instructions at :ref:`system-outputs-ubimage`
+    Installation is currently a manual process.
+
+    Preparation
+    -----------
+
+    To prepare the device for Liminix you first need to use the
+    `OpenWrt UBI Installer
+    <https://github.com/dangowrt/owrt-ubi-installer>`_ image to
+    rewrite the flash layout. You can do this in onw of two ways:
+    either follow the instructions to do it through the vendor web
+    interface, or you can drop to U-boot and use TFTP
+
+    .. code-block:: console
+
+        MT7622> setenv ipaddr 10.0.0.6
+        MT7622> setenv serverip 10.0.0.1
+        MT7622> tftpboot 0x42000000  openwrt-mediatek-mt7622-linksys_e8450-ubi-initramfs-recovery-installer.itb
+        MT7622> bootm 0x42000000
+
+     This will write the new flash layout and then boot into a
+     "recovery" OpenWrt installation. Once it's finished booting into
+     Linux you can safely reboot
+
+     Installing Liminix
+     ------------------
+
+     This is a manual process: you need a :ref:`serial <serial>` conection and TFTP : follow the instructions at :ref:`system-outputs-ubimage`

 '';

@@ -52,14 +75,10 @@
        ../../modules/arch/aarch64.nix
        ../../modules/outputs/tftpboot.nix
        ../../modules/outputs/ubifs.nix
+        ../../modules/outputs/ubimage.nix
      ];
      config = {
    kernel = {
-      src = pkgs.pkgsBuildBuild.fetchurl {
-        name = "linux.tar.gz";
-        url = "https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.137.tar.gz";
-        hash = "sha256-PkdzUKZ0IpBiWe/RS70J76JKnBFzRblWcKlaIFNxnHQ=";
-      };
      extraPatchPhase = ''
          ${pkgs.openwrt.applyPatches.mediatek}
      '';
@@ -182,7 +201,8 @@
      in {
        ubi = {
          minIOSize = "2048";
-          eraseBlockSize = "126976";
+          logicalEraseBlockSize = "126976";
+          physicalEraseBlockSize = "131072";
          maxLEBcount = "1024"; # guessing
        };

@@ -197,7 +217,7 @@
        rootDevice = "ubi0:liminix";
        dts = {
          src = "${openwrt.src}/target/linux/mediatek/dts/mt7622-linksys-e8450-ubi.dts";
-          includes =  [
+          includePaths =  [
            "${openwrt.src}/target/linux/mediatek/dts"
            "${config.system.outputs.kernel.modulesupport}/arch/arm64/boot/dts/mediatek/"
          ];
@@ -5,11 +5,6 @@
  ];
  config = {
    kernel = {
-      src = pkgs.pkgsBuildBuild.fetchurl {
-        name = "linux.tar.gz";
-        url = "https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.137.tar.gz";
-        hash = "sha256-PkdzUKZ0IpBiWe/RS70J76JKnBFzRblWcKlaIFNxnHQ=";
-      };
      config = {
        MTD = "y";
        MTD_BLOCK = "y";
@@ -52,8 +52,9 @@

  '';

-  module = {pkgs, config, lim, ... }:
+  module = {pkgs, config, lim, lib, ... }:
    let
+      inherit (lib) mkIf;
      openwrt = pkgs.openwrt;
      firmwareBlobs = pkgs.pkgsBuildBuild.fetchFromGitHub {
        owner = "kvalo";
@@ -116,9 +117,12 @@
        rootDevice = "/dev/mtdblock5";
        dts = {
          src = "${openwrt.src}/target/linux/ath79/dts/qca9531_glinet_gl-ar750.dts";
-          includes =  [
+          includePaths =  [
            "${openwrt.src}/target/linux/ath79/dts"
          ];
+          includes = mkIf config.logging.persistent.enable [
+            ./pstore-ramoops.dtsi
+          ];
        };

        networkInterfaces =
@@ -157,12 +161,6 @@
        appendDTB = true;
      };
      kernel = {
-        src = pkgs.pkgsBuildBuild.fetchurl {
-          name = "linux.tar.gz";
-          url = "https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.137.tar.gz";
-          hash = "sha256-PkdzUKZ0IpBiWe/RS70J76JKnBFzRblWcKlaIFNxnHQ=";
-        };
-
        # Mainline linux 5.19 doesn't have device-tree support for
        # this device or even for the SoC, so we use the extensive
        # OpenWrt kernel patches
@@ -0,0 +1,9 @@
+/ {
+reserved-memory {
+		ramoops@03f00000 {
+			compatible = "ramoops";
+			reg = <0x03f00000 0x10000>;
+			pmsg-size = <0x10000>;
+		};
+	};
+};
@@ -81,7 +81,7 @@

        dts = {
          src = "${openwrt.src}/target/linux/ramips/dts/mt7620a_glinet_gl-mt300a.dts";
-          includes = [
+          includePaths = [
            "${openwrt.src}/target/linux/ramips/dts"
          ];
        };
@@ -114,11 +114,6 @@
     };

      kernel = {
-        src = pkgs.fetchurl {
-          name = "linux.tar.gz";
-          url = "https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.137.tar.gz";
-          hash = "sha256-PkdzUKZ0IpBiWe/RS70J76JKnBFzRblWcKlaIFNxnHQ=";
-        };
        extraPatchPhase = ''
          ${openwrt.applyPatches.ramips}
          ${openwrt.applyPatches.rt2x00}
@@ -78,7 +78,7 @@

        dts = {
          src = "${openwrt.src}/target/linux/ramips/dts/mt7628an_glinet_gl-mt300n-v2.dts";
-          includes = [
+          includePaths = [
            "${openwrt.src}/target/linux/ramips/dts"
          ];
        };
@@ -125,11 +125,6 @@
      };

      kernel = {
-        src = pkgs.fetchurl {
-          name = "linux.tar.gz";
-          url = "https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.137.tar.gz";
-          hash = "sha256-PkdzUKZ0IpBiWe/RS70J76JKnBFzRblWcKlaIFNxnHQ=";
-        };
        extraPatchPhase = ''
          ${openwrt.applyPatches.ramips}
        '';
@@ -66,7 +66,7 @@
        # *correct* but it does at least boot
        dts = lib.mkForce {
          src = "${config.system.outputs.kernel.modulesupport}/arch/mips/boot/dts/mti/malta.dts";
-          includes =  [
+          includePaths =  [
            "${config.system.outputs.kernel.modulesupport}/arch/mips/boot/dts/"
          ];
        };
@@ -50,11 +50,6 @@
      ];
      config = {
        kernel = {
-          src = pkgs.pkgsBuildBuild.fetchurl {
-            name = "linux.tar.gz";
-            url = "https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.137.tar.gz";
-            hash = "sha256-PkdzUKZ0IpBiWe/RS70J76JKnBFzRblWcKlaIFNxnHQ=";
-          };
          extraPatchPhase = ''
            ${pkgs.openwrt.applyPatches.ramips}
          '';
@@ -410,7 +405,7 @@
          rootDevice = "/dev/mtdblock3";
          dts = {
            src = "${openwrt.src}/target/linux/ramips/dts/mt7621_tplink_archer-ax23-v1.dts";
-            includes =  [
+            includePaths =  [
              "${openwrt.src}/target/linux/ramips/dts"
              "${config.system.outputs.kernel.modulesupport}/arch/arm64/boot/dts/mediatek/"
            ];
@@ -172,7 +172,6 @@
        ../../modules/arch/arm.nix
        ../../modules/outputs/tftpboot.nix
        ../../modules/outputs/mbrimage.nix
-        ../../modules/outputs/extlinux.nix
      ];

      config = {
@@ -224,11 +223,6 @@
            # WARNING: unmet direct dependencies detected for ARCH_WANT_LIBATA_LEDS
            ATA = "y";

-            PSTORE = "y";
-            PSTORE_RAM = "y";
-            PSTORE_CONSOLE = "y";
-#            PSTORE_DEFLATE_COMPRESS = "n";
-
            BLOCK = "y";
            MMC="y";
            PWRSEQ_EMMC="y";        # ???
@@ -341,14 +335,14 @@
            targets = ["ath9k" "ath10k_pci"];
          };
        in {
-          defaultOutput = "mtdimage";
+          defaultOutput = "updater";
          loadAddress = lim.parseInt "0x00800000"; # "0x00008000";
          entryPoint = lim.parseInt "0x00800000"; # "0x00008000";
          rootDevice = "/dev/mmcblk0p1";

          dts = {
            src = "${config.system.outputs.kernel.modulesupport}/arch/arm/boot/dts/marvell/armada-385-turris-omnia.dts";
-            includes =  [
+            includePaths =  [
              "${config.system.outputs.kernel.modulesupport}/arch/arm/boot/dts/marvell/"
            ];
          };
@@ -124,7 +124,7 @@
        hash = "sha256-ifriAjWzFACrxVWCANZpUaEZgB/0pdbhnTVQytx6ddg=";
      };
    in {
-      imports = [ 
+      imports = [
        # We include it to ensure the bridge functionality
        # is available on the target kernel.
        ../../modules/bridge
@@ -184,7 +184,7 @@
          # Actually, this is not what we want.
          # This DTS is insufficient.
          src = ./mt7621_zyxel_nwa50ax.dtsi;
-          includes = [
+          includePaths = [
            # Here's one weird trick to make `ubi` detection
            # out of the box.
            # We will write ubi on /dev/firmware_a:rootfs location
@@ -233,7 +233,7 @@
      services.zyxel-dual-image = config.boot.zyxel-dual-image.build {
        ensureActiveImage = "primary";
        # TODO: use mtd names rather…
-        # primary and secondary are always /dev/mtd3 by virtue of the 
+        # primary and secondary are always /dev/mtd3 by virtue of the
        # dtb being not too wrong…
        # TODO: remove this hack.
        primaryMtdPartition = "/dev/mtd3";
@@ -250,14 +250,8 @@
      #  IMAGE/ramboot-factory.bin := append-kernel | pad-to $$(KERNEL_SIZE) | append-ubi

      kernel = {
-        src = pkgs.fetchurl {
-          name = "linux.tar.gz";
-          url = "https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.137.tar.gz";
-          hash = "sha256-PkdzUKZ0IpBiWe/RS70J76JKnBFzRblWcKlaIFNxnHQ=";
-        };
        extraPatchPhase = ''
          ${openwrt.applyPatches.ramips}
-
        '';
        config = {

@@ -131,50 +131,54 @@ human-readable format, use :command:`s6-tai64nlocal`.
  1970-01-02 21:51:48.832588765 wan.link.pppoe sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x667a9594> <pcomp> <accom
  p>]

+Log persistence
+---------------

+Logs written to :file:`/run/log/` will not survive a reboot or crash,
+as it is an ephemeral filesystem.

-Updating an installed system (JFFS2)
-************************************
+On supported hardware you can enable logging to `pstore
+<https://www.kernel.org/doc/Documentation/ABI/testing/pstore>`_ which
+means the most recent log messages will be preserved on reboot.  Set
+the config option ``logging.persistent.enable = true`` to log messages
+to :file:`/dev/pmsg0` as well as to the regular log. This is a
+circular buffer, so when it fills up newer messages will overwrite the
+oldest messages.

-
-Adding packages
-===============
-
-If your device is running a JFFS2 root filesystem, you can build
-extra packages for it on your build system and copy them to the
-device: any package in Nixpkgs or in the Liminix overlay is available
-with the ``pkgs`` prefix:
+To check the previous messages after a (planned or forced) reboot,
+you need to mooun the pstore filesystem.

 .. code-block:: console

-    nix-build -I liminix-config=./my-configuration.nix \
-     --arg device "import ./devices/mydevice" -A pkgs.tcpdump
-
-    nix-shell -p min-copy-closure root@the-device result/
-
-Note that this only copies the package to the device: it doesn't update
-any profile to add it to ``$PATH``
+    # mount -t pstore pstore /sys/fs/pstore/
+    # ls -l /sys/fs/pstore/
+    -r--r--r--    1     43071 pmsg-ramoops-0
+    # cat /sys/fs/pstore/pmsg-ramoops-0
+    @40000000000000282c997d29 mydevice klogd <6>[   30.793756] int: port 2(wlan0) entered blocking state
+    [log messages from before the reboot follow]


-.. _rebuilding the system:

-Rebuilding the system
-=====================
+Updating an installed system
+****************************

-Liminix has a mechanism for in-place updates of a running system which
-is analogous to :command:`nixos-rebuild`, but its operation is a
-bit different because it expects to run on a build machine and then
-copy to the host device. To use this, build the `outputs.systemConfiguration`
-target and then run the :command:`result/install.sh` script it generates.
+If your system has a writable root filesystem (JFFS2, btrfs etc -
+anything but squashfs), we have mechanisms for in-places updates
+analogous to :command:`nixos-rebuild`, but the operation is a bit
+different because it expects to run on a build machine and then copy
+to the host device using :command:`ssh`.
+
+To use this, build the ``outputs.updater``
+target and then run the :command:`update.sh` script it generates.

 .. code-block:: console

    nix-build -I liminix-config=./my-configuration.nix \
       --arg device "import ./devices/mydevice" \
-       -A outputs.systemConfiguration
-    ./result/install.sh root@the-device 
+       -A outputs.updater
+    ./result/bin/update.sh root@the-device 

-The install script uses min-copy-closure to copy new or changed
+The update script uses min-copy-closure to copy new or changed
 packages to the device, then (perhaps) reboots it. The reboot
 behaviour can be affected by flags:

@@ -187,7 +191,6 @@ behaviour can be affected by flags:
  the services that have updated store paths (and anything that
  depends on them), but will not affect services that haven't changed.

-
 It doesn't delete old packages automatically: to do that run
 :command:`min-collect-garbage`, which will delete any packages not in
 the current system closure. Note that Liminix does not have the NixOS
@@ -207,6 +210,26 @@ Caveats

 .. _levitate:  

+Adding packages
+===============
+
+If you simply wish to add a package without any change to services,
+you can call :command:`min-copy-closure` directly to install
+any package in Nixpkgs or in the Liminix overlay
+
+.. code-block:: console
+
+    nix-build -I liminix-config=./my-configuration.nix \
+     --arg device "import ./devices/mydevice" -A pkgs.tcpdump
+
+    nix-shell -p min-copy-closure root@the-device result/
+
+Note that this only copies the package and its dependencies to the
+device: it doesn't update any profile to add it to ``$PATH``
+
+
+.. _rebuilding the system:
+
 Reinstalling on a running system
 ********************************

@@ -22,7 +22,6 @@ in rec {
    ../modules/outputs/ubimage.nix
    ../modules/outputs/jffs2.nix
    ../modules/outputs/ext4fs.nix
-    ../modules/outputs/extlinux.nix
  ];

  kernel.config = {
@@ -30,7 +30,6 @@
   ./outputs/vmroot.nix
   ./ppp
   ./ramdisk.nix
-   ./squashfs.nix
   ./ssh
   ./users.nix
   ./vlan
@@ -13,6 +13,12 @@ in
  options = {
    boot = { };
    hardware = {
+      ubi = {
+        minIOSize = mkOption { type = types.str; };
+        logicalEraseBlockSize = mkOption { type = types.str; }; # LEB
+        physicalEraseBlockSize = mkOption { type = types.str; }; # PEB
+        maxLEBcount = mkOption { type = types.str; }; # LEB
+      };
      dts = {
        src = mkOption {
          type = types.nullOr types.path;
@@ -24,11 +30,16 @@ in
            only for QEMU.
          '';
        };
-        includes = mkOption {
+        includePaths = mkOption {
          default = [ ];
          description = "List of directories to search for DTS includes (.dtsi files)";
          type = types.listOf types.path;
        };
+        includes = mkOption {
+          default = [ ];
+          description = "\"dtsi\" fragments to include in the generated device tree";
+          type = types.listOf types.path;
+        };
      };
      defaultOutput = mkOption {
        description = "\"Default\" output: what gets built for this device when outputs.default is requested. Typically this is \"mtdimage\" or \"vmroot\"";
@@ -6,7 +6,7 @@
 { lib, pkgs, config, ...}:
 let
  inherit (lib) mkOption types ;
-  inherit (pkgs) liminix;
+  inherit (pkgs) liminix openwrt;

  mergeConditionals = conf : conditions :
    # for each key in conditions, if it is present in conf
@@ -21,8 +21,8 @@ let
 in {
  options = {
    kernel = {
-      src = mkOption { type = types.path; } ;
-      version = mkOption { type = types.str; default = "5.15.137";} ;
+      src = mkOption { type = types.path; default = openwrt.kernelSrc; } ;
+      version = mkOption { type = types.str; default = openwrt.kernelVersion;} ;
      modular = mkOption {
        type = types.bool;
        default = true;
@@ -68,19 +68,15 @@ in {
    };
  };
  config = {
-    system.outputs =
+    system.outputs.kernel =
      let
        mergedConfig = mergeConditionals
          config.kernel.config
          config.kernel.conditionalConfig;
-        k = liminix.builders.kernel.override {
-          config = mergedConfig;
-          inherit (config.kernel) version src extraPatchPhase;
-          targets = config.kernel.makeTargets;
-        };
-      in {
-        kernel = k.vmlinux;
-        zimage = k.zImage;
+      in liminix.builders.kernel.override {
+        config = mergedConfig;
+        inherit (config.kernel) version src extraPatchPhase;
+        targets = config.kernel.makeTargets;
      };

    kernel = rec {
@@ -11,9 +11,12 @@ let
 in
 {
  imports = [
-    ./squashfs.nix
+    ./outputs/squashfs.nix
    ./outputs/vmroot.nix
-    ./outputs/extlinux.nix
+    ./outputs/boot-extlinux.nix
+    ./outputs/boot-fit.nix
+    ./outputs/uimage.nix
+    ./outputs/updater
  ];
  options = {
    system.outputs = {
@@ -27,17 +30,7 @@ in
          kernel
          ******

-          Kernel vmlinux file (usually ELF)
-        '';
-      };
-      zimage = mkOption {
-        type = types.package;
-        internal = true;
-        description = ''
-          zimage
-          ******
-
-          Kernel in compressed self-extracting package
+          Kernel package (multi-output derivation)
        '';
      };
      dtb = mkOption {
@@ -50,16 +43,6 @@ in
          Compiled device tree (FDT) for the target device
        '';
      };
-      uimage = mkOption {
-        type = types.package;
-        internal = true;
-        description = ''
-          uimage
-          ******
-
-          Combined kernel and FDT in uImage (U-Boot compatible) format
-        '';
-      };
      tplink-safeloader = mkOption {
        type = types.package;
      };
@@ -82,6 +65,12 @@ in
          directory of files to package into root filesystem
        '';
      };
+      bootfiles = mkOption {
+        type = types.nullOr types.package;
+        internal = true;
+        default = null;
+        # description = "";
+      };
      bootablerootdir = mkOption {
        type = types.package;
        internal = true;
@@ -104,18 +93,11 @@ in
    system.outputs = rec {
      dtb = liminix.builders.dtb {
        inherit (config.boot) commandLine;
-        dts = config.hardware.dts.src;
-        includes = config.hardware.dts.includes ++ [
+        dts = [config.hardware.dts.src] ++ config.hardware.dts.includes;
+        includes = config.hardware.dts.includePaths ++ [
          "${o.kernel.headers}/include"
        ];
      };
-      uimage = liminix.builders.uimage {
-        commandLine = concatStringsSep " " config.boot.commandLine;
-        inherit (config.boot) commandLineDtbNode;
-        inherit (config.hardware) loadAddress entryPoint alignment;
-        inherit (config.boot) imageFormat;
-        inherit (o) kernel dtb;
-      };
      rootdir =
        let
          inherit (pkgs.pkgsBuildBuild) runCommand;
@@ -132,8 +114,8 @@ in
        let inherit (pkgs.pkgsBuildBuild) runCommand;
        in runCommand "add-slash-boot" { } ''
          cp -a ${o.rootdir} $out
-          ${if config.boot.loader.extlinux.enable
-            then "(cd $out && chmod -R +w . && rmdir boot && cp -a ${o.extlinux} boot)"
+          ${if o.bootfiles != null
+            then "(cd $out && chmod -R +w . && rmdir boot && cp -a ${o.bootfiles} boot)"
            else ""
           }
         '';
@@ -12,19 +12,15 @@ let
  cmdline = concatStringsSep " " config.boot.commandLine;
  wantsDtb = config.hardware.dts ? src && config.hardware.dts.src != null;
 in {
-  options.system.outputs.extlinux = mkOption {
-    type = types.package;
-    # description = "";
-  };
  options.boot.loader.extlinux.enable = mkEnableOption "extlinux";

  config = mkIf cfg.enable {
-    system.outputs.extlinux = pkgs.runCommand "extlinux" {} ''
+    system.outputs.bootfiles = pkgs.runCommand "extlinux" {} ''
      mkdir $out
      cd $out
      ${if wantsDtb then "cp ${o.dtb} dtb" else "true"}
      cp ${o.initramfs} initramfs
-      cp ${o.zimage} kernel
+      cp ${o.kernel.zImage} kernel
      mkdir extlinux
      cat > extlinux/extlinux.conf << _EOF
      menu title Liminix
@@ -37,7 +33,7 @@ in {
      _EOF
    '';
    filesystem = dir {
-      boot = symlink config.system.outputs.extlinux;
+      boot = symlink config.system.outputs.bootfiles;
    };
  };
 }
@@ -0,0 +1,27 @@
+{
+  config
+, pkgs
+, lib
+, ...
+}:
+let
+  inherit (lib) mkIf mkEnableOption mkOption types concatStringsSep;
+  inherit (pkgs.pseudofile) dir symlink;
+  cfg = config.boot.loader.fit;
+  o = config.system.outputs;
+  cmdline = concatStringsSep " " config.boot.commandLine;
+  wantsDtb = config.hardware.dts ? src && config.hardware.dts.src != null;
+in {
+  options.boot.loader.fit.enable = mkEnableOption "FIT in /boot";
+
+  config = mkIf cfg.enable {
+    system.outputs.bootfiles = pkgs.runCommand "boot-fit" {} ''
+      mkdir $out
+      cd $out
+      cp ${o.uimage} fit
+    '';
+    filesystem = dir {
+      boot = symlink config.system.outputs.bootfiles;
+    };
+  };
+}
@@ -9,6 +9,7 @@ let
  inherit (pkgs) runCommand;
 in
 {
+  imports = [ ./system-configuration.nix ];
  options = {
    boot.initramfs = {
      enable = mkEnableOption "initramfs";
@@ -22,14 +23,6 @@ in
          filesystem
        '';
      };
-      systemConfiguration = mkOption {
-        type = types.package;
-        description = ''
-          pkgs.systemconfig for the configured filesystem,
-          contains 'activate' and 'init' commands
-        '';
-        internal = true;
-      };
    };
  };
  config = mkIf config.boot.initramfs.enable {
@@ -53,8 +46,6 @@ in
          file /init ${pkgs.preinit}/bin/preinit 0755 0 0
          SPECIALS
        '';
-      systemConfiguration =
-        pkgs.systemconfig config.filesystem.contents;
    };
  };
 }
@@ -0,0 +1,28 @@
+{
+  config
+, pkgs
+, lib
+, ...
+}:
+let
+  inherit (lib) mkEnableOption mkOption mkIf types;
+  inherit (pkgs) runCommand;
+in
+{
+  options = {
+    system.outputs = {
+      systemConfiguration = mkOption {
+        type = types.package;
+        description = ''
+          pkgs.systemconfig for the configured filesystem,
+          contains 'activate' and 'init' commands
+        '';
+        internal = true;
+      };
+    };
+  };
+  config = {
+    system.outputs.systemConfiguration =
+        pkgs.systemconfig config.filesystem.contents;
+  };
+}
@@ -61,7 +61,7 @@ in {
          o = config.system.outputs;
          image = let choices = {
            uimage = o.uimage;
-            zimage = o.zimage;
+            zimage = o.kernel.zImage;
          }; in choices.${cfg.kernelFormat};
          bootCommand = let choices = {
            uimage = "bootm";
@@ -13,18 +13,6 @@ in
    ./initramfs.nix
  ];

-  options.system.outputs.rootubifs = mkOption {
-    type = types.package;
-    internal = true;
-  };
-
-  options.hardware.ubi = {
-    minIOSize = mkOption { type = types.str; };
-    logicalEraseBlockSize = mkOption { type = types.str; }; # LEB
-    physicalEraseBlockSize = mkOption { type = types.str; }; # PEB
-    maxLEBcount = mkOption { type = types.str; }; # LEB
-  };
-
  config = mkIf (config.rootfsType == "ubifs") {
    kernel.config = {
      MTD_UBI="y";
@@ -33,7 +21,7 @@ in
    };
    boot.initramfs.enable = true;
    system.outputs = {
-      rootubifs =
+      rootfs =
        let
          inherit (pkgs.pkgsBuildBuild) runCommand mtdutils;
          cfg = config.hardware.ubi;
@@ -82,12 +82,11 @@ automatically, you can try booting it by hand to see if it works:
 .. code-block:: console

    uboot> ubifsmount ubi0:liminix
-    uboot> ubifsload ''${loadaddr} boot/uimage
+    uboot> ubifsload ''${loadaddr} boot/fit
    uboot> bootm ''${loadaddr}

 Once you've done this and you're happy with it, reset the device to
-U-Boot. You don't need to recreate the volume but you do need to
-repeat step 3.
+return to U-Boot.

 5) Instructions for configuring autoboot are likely to be very
 device-dependent. On the Linksys E8450/Belkin RT3200, the environment
@@ -96,7 +95,10 @@ you could do

 .. code-block:: console

-    uboot> setenv boot_production 'led $bootled_pwr on ; ubifsmount ubi0:liminix; ubifsload ''${loadaddr} boot/uimage; bootm ''${loadaddr}'
+    uboot> setenv orig_boot_production $boot_production
+    uboot> setenv boot_production 'led $bootled_pwr on ; ubifsmount ubi0:liminix && ubifsload ''${loadaddr} boot/fit && bootm ''${loadaddr}'
+    uboot> saveenv
+    uboot> reset

 On other devices, some detective work may be needed. Try running
 `printenv` and look for likely commands, try looking at the existing
@@ -111,16 +113,13 @@ boot process, maybe even try looking for documentation for that device.
    };
  };

-  config = mkIf (config.rootfsType == "ubifs") {
-    system.outputs = {
-      ubimage =
-        let o = config.system.outputs; in
-        pkgs.runCommand "ubimage" {} ''
-          mkdir $out
-          cd $out
-          ln -s ${o.rootfs} rootfs
-          ln -s ${instructions} env.scr
-       '';
-    };
-  };
+  config.system.outputs.ubimage =
+    assert config.rootfsType == "ubifs";
+    let o = config.system.outputs; in
+    pkgs.runCommand "ubimage" {} ''
+      mkdir $out
+      cd $out
+      ln -s ${o.rootfs} rootfs
+      ln -s ${instructions} env.scr
+   '';
 }
@@ -18,6 +18,9 @@ in
      eraseBlockSize = mkOption { type = types.str; }; # LEB
      maxLEBcount = mkOption { type = types.str; }; # LEB
    };
+    options.system.outputs.ubivolume = mkOption {
+      type = types.package;
+    };

    config = mkIf (config.rootfsType == "ubifs") {
      kernel.config = {
@@ -28,7 +31,7 @@ in
      };
      boot.initramfs.enable = true;

-      system.outputs.rootfs =
+      system.outputs.ubivolume =
      let
        inherit (pkgs.pkgsBuildBuild) runCommand;
        ubiVolume = ({ name, volumeId, image, flags ? [] }:
@@ -81,7 +84,7 @@ in
          ]);

        disk = ubiDisk {
-          initramfs = config.system.outputs.rootubifs; # liminix.builders.squashfs config.filesystem.contents; #           # assert this is a proper FIT.
+          initramfs = config.system.outputs.rootfs; # ???
        };

      in
@@ -0,0 +1,30 @@
+{
+  config,
+  pkgs,
+  lib,
+  ...
+}:
+let
+  inherit (lib) mkOption types concatStringsSep;
+  inherit (pkgs) liminix writeText;
+  o = config.system.outputs;
+in
+{
+  options.system.outputs.uimage = mkOption {
+    type = types.package;
+    internal = true;
+    description = ''
+      uimage
+      ******
+
+      Combined kernel and FDT in uImage (U-Boot compatible) format
+    '';
+  };
+  config.system.outputs.uimage = liminix.builders.uimage {
+    commandLine = concatStringsSep " " config.boot.commandLine;
+    inherit (config.boot) commandLineDtbNode;
+    inherit (config.hardware) loadAddress entryPoint alignment;
+    inherit (config.boot) imageFormat;
+    inherit (o) kernel dtb;
+  };
+}
@@ -0,0 +1,36 @@
+{
+  config
+, pkgs
+, lib
+, ...
+}:
+let
+  inherit (lib) mkIf;
+  o = config.system.outputs;
+  inherit (pkgs) runCommand;
+  inherit (lib) mkOption types;
+  inherit (pkgs.buildPackages) min-copy-closure;
+in
+{
+  imports = [ ../system-configuration.nix ];
+  options.system.outputs.updater = mkOption {
+    type = types.package;
+    description = ''
+      updater
+      ******
+
+      For configurations with a writable filesystem, create a shell
+      script that runs on the build system and updates the device
+      over the network to the new configuration
+    '';
+  };
+
+  config.system.outputs.updater =
+    runCommand "buildUpdater" { } ''
+      mkdir -p $out/bin
+      substitute ${./update.sh} $out/bin/update.sh \
+         --subst-var-by toplevel ${o.systemConfiguration} \
+         --subst-var-by min_copy_closure ${min-copy-closure}
+      chmod +x  $out/bin/update.sh
+    '';
+}
@@ -1,22 +1,19 @@
 #!/usr/bin/env bash

-# this shell script can be run on the build system to
-# min-copy-closure the system configuration onto the device
-# and reboot/restart services as requested
+# this shell script is run on the build system to min-copy-closure the
+# system configuration onto the device and reboot/restart services as
+# requested

 die() {
    echo "$@"
    exit 1
 }

-# add min-copy-closure to the path. removing junk characters
-# inserted by default.nix (q.v.)
-min_copy_closure=@min-copy-closure@; PATH=${min_copy_closure//_/}/bin:$PATH

+PATH=@min_copy_closure@/bin:$PATH
 ssh_command=${SSH_COMMAND-ssh}

 reboot="reboot"
-
 case "$1" in
    "--no-reboot")
 	unset reboot
@@ -34,10 +31,11 @@ shift
 test -n "$target_host" || \
    die "Usage: $0 [--no-reboot] [--fast] target-host"

-toplevel=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && readlink `pwd` )
+toplevel=$(realpath @toplevel@)
 test -e $toplevel/etc/nix-store-paths || die "missing etc/nix-store-paths, is this really a system configuration?"
 echo installing from systemConfiguration $toplevel to host $target_host

+$ssh_command $target_host uname -a || die "Can't ssh to $target_host"
 min-copy-closure $target_host $toplevel
 set -x
 $ssh_command $target_host $toplevel/bin/install
@@ -31,7 +31,7 @@ on a system with pre-existing firmware and OS.
  config = mkIf (config.rootfsType == "ubifs") {

    system.outputs.zyxel-nwa-fit =
-      let 
+      let
        o = config.system.outputs;
        # 8129kb padding.
        paddedKernel = pkgs.runCommand "padded-kernel" {} ''
@@ -39,7 +39,7 @@ on a system with pre-existing firmware and OS.
          dd if=/dev/zero of=$out bs=1 count=1 seek=8388607
        '';
        firmwareImage = pkgs.runCommand "firmware-image" {} ''
-          cat ${paddedKernel} ${o.rootfs} > $out
+          cat ${paddedKernel} ${o.ubivolume} > $out
        '';
        dts = pkgs.writeText "image.its" ''
        /dts-v1/;
@@ -7,10 +7,27 @@ let
    s6-linux-init
    stdenvNoCC;
  inherit (lib.lists) unique concatMap;
+  inherit (lib) concatStrings;
+  inherit (builtins) map;
  inherit (pkgs.pseudofile) dir symlink;
  inherit (pkgs.liminix.services) oneshot bundle longrun;
  inherit (lib) mkIf mkEnableOption mkOption types;
  cfg = config.logging;
+
+  logger =
+    let pipecmds =
+          ["${s6}/bin/s6-log -bpd3 -- ${cfg.script} 1"] ++
+          (lib.optional cfg.persistent.enable
+            "/bin/tee /dev/pmsg0") ++
+          (lib.optional cfg.shipping.enable
+            "${pkgs.logshipper}/bin/logtap ${cfg.shipping.socket} logshipper-socket-event");
+    in ''
+      #!${execline}/bin/execlineb -P
+      ${execline}/bin/redirfd -w 1 /dev/null
+      ${execline}/bin/redirfd -rnb 0 fifo
+      ${concatStrings (map (l: "pipeline { ${l} }\n") pipecmds)}
+      ${s6}/bin/s6-log -- ${cfg.directory}
+    '';
  s6-rc-db =
    let
      # In the default bundle we need to have all the services
@@ -106,21 +123,7 @@ let
        mode = "0600";
      };
      notification-fd = { file = "3"; };
-      run = {
-        file = ''
-              #!${execline}/bin/execlineb -P
-              ${execline}/bin/redirfd -w 1 /dev/null
-              ${execline}/bin/redirfd -rnb 0 fifo
-              ${if cfg.shipping.enable then ''
-                pipeline { ${s6}/bin/s6-log -bpd3 -- ${cfg.script} 1 }
-                pipeline { ${pkgs.logshipper}/bin/logtap ${cfg.shipping.socket} logshipper-socket-event }
-                ${s6}/bin/s6-log -- ${cfg.directory}
-              '' else ''
-                ${s6}/bin/s6-log -bpd3 -- ${cfg.script} ${cfg.directory}
-              ''}
-          '';
-        mode = "0755";
-      };
+      run = { file = logger; mode = "0755"; };
    };
    getty = dir {
      run = {
@@ -212,6 +215,9 @@ let
 in {
  options = {
    logging = {
+      persistent = {
+        enable = mkEnableOption "store logs across reboots";
+      };
      shipping = {
        enable = mkEnableOption "unix socket for log shipping";
        socket = mkOption {
@@ -263,6 +269,11 @@ in {
    )];

  config = {
+    kernel.config = mkIf config.logging.persistent.enable {
+      PSTORE = "y";
+      PSTORE_PMSG = "y";
+      PSTORE_RAM = "y";
+    };
    filesystem = dir {
      etc = dir {
        s6-rc = dir {
@@ -5,7 +5,7 @@ let
  extraPkgs = import ./pkgs/default.nix {
    inherit (final) lib callPackage;
  };
-  inherit (final) fetchpatch;
+  inherit (final) fetchpatch lib;
  luaHost =
    let
      l = prev.lua5_3.overrideAttrs(o: {
@@ -73,14 +73,15 @@ extraPkgs // {
        });
    in chrony'.override {
      gnutls = null;
-      nss = null;
-      nspr = null;
-      readline = null;
      libedit = null;
      libseccomp = null;
      # should texinfo be in nativeBuildInputs instead of
      # buildInputs?
      texinfo = null;
+    } // lib.optionalAttrs (lib.versionOlder lib.version "24.10") {
+      nss = null;
+      nspr = null;
+      readline = null;
    };

  # clevis without luks/tpm
@@ -217,11 +218,11 @@ extraPkgs // {
    ] ++ final.lib.optionals (o ? patches) o.patches;
  });

-  mtdutils = prev.mtdutils.overrideAttrs(o: {
+  mtdutils = (prev.mtdutils.overrideAttrs(o: {
    patches = (if o ? patches then o.patches else []) ++ [
      ./pkgs/mtdutils/0001-mkfs.jffs2-add-graft-option.patch
    ];
-  });
+  })).override { util-linux = final.util-linux-small ; };

  nftables = prev.nftables.overrideAttrs(o: {
    configureFlags = [
@@ -3,7 +3,6 @@

 ;; e.g. (import-macros { : expect= } :anoia.assert)

-
 (fn expect [assertion]
  (let [msg (.. "expectation failed: " (view assertion))]
    `(when (not ,assertion)
@@ -82,7 +82,6 @@
  (let [p (find-executable "yes" (os.getenv "PATH"))]
    (expect (string.match p "coreutils.+bin/yes$"))))

-
 {
 : mktree
 : rmtree
@@ -1,8 +1,4 @@
-;; importing assert.fnl macros here would be circular, so we can't use
-;; the full test functionality
-(macro define-tests [& body]
-  (when _G.RUNNING_TESTS
-    `(do ,(unpack body))))
+(import-macros { :  define-tests : expect : expect= } :anoia.assert)

 (fn assoc [tbl k v & more]
  (tset tbl k v)
@@ -32,6 +28,18 @@
 (fn dirname [path]
  (string.match path "(.*)/[^/]-$"))

+(fn append-path [dirname filename]
+  (let [base (or (string.match dirname "(.*)/$") dirname)
+        result []]
+    (each [component (string.gmatch filename "([^/]+)")]
+      (if (and (= component "..") (> (# result) 0))
+          (table.remove result)
+          (= component "..")
+          (error "path traversal attempt")
+          true
+          (table.insert result component)))
+    (.. base "/" (table.concat result "/"))))
+
 (fn system [s]
  (match (os.execute s)
    res (do (print (.. "Executed \"" s "\", exit code " (tostring res)))  res)
@@ -59,16 +67,28 @@
             (and present (. a k))))))

 (define-tests
- (assert (table= {:a 1 :b 2} {:b 2 :a 1}))
- (assert (not (table= {:a 1 :b 2 :k :l}  {:b 2 :a 1})))
- (assert (not (table= {:a 1 :b 2}  {:b 2 :a 1 :k :l})))
+ (expect (table= {:a 1 :b 2} {:b 2 :a 1}))
+ (expect (not (table= {:a 1 :b 2 :k :l}  {:b 2 :a 1})))
+ (expect (not (table= {:a 1 :b 2}  {:b 2 :a 1 :k :l})))

- (assert (table= {:a 1 :b {:l 17}} {:b {:l 17} :a 1}))
- (assert (table= {:a [4 5 6 7] } {:a [4 5 6 7]}))
- (assert (not (table= {:a [4 5 6 7] } {:a [4 5 6 7 8]})))
- (assert (not (table= {:a [4 5 7 6] } {:a [4 5 6 7 ]})))
+ (expect (table= {:a 1 :b {:l 17}} {:b {:l 17} :a 1}))
+ (expect (table= {:a [4 5 6 7] } {:a [4 5 6 7]}))
+ (expect (not (table= {:a [4 5 6 7] } {:a [4 5 6 7 8]})))
+ (expect (not (table= {:a [4 5 7 6] } {:a [4 5 6 7 ]})))

- (assert (table= {} {}))
+ (expect (table= {} {}))
+
+ (let [traps (fn [b p]
+               (match (pcall append-path b p)
+                 (true f) (error "didn't trap path traversal")
+                 (false err) (expect (string.match err "path traversal"))))]
+   (expect= (append-path "/tmp" "hello") "/tmp/hello")
+   (expect= (append-path "/tmp/" "hello") "/tmp/hello")
+   (expect= (append-path "/tmp/" "///hello") "/tmp/hello")
+   (expect= (append-path "/tmp/" "///hello/../fish") "/tmp/fish")
+   (traps "/tmp/" "../hello")
+   (expect= (append-path "/tmp/" "hello/../goodbye") "/tmp/goodbye")
+   (traps "/tmp/" "hello/../../goodbye"))
 )

 (fn dig [tree path]
@@ -177,17 +197,17 @@
        b64 (base64 :url)]

    (let [a (b64:decode "YWxsIHlvdXIgYmFzZQ==")]
-      (assert (= a "all your base") (view a)))
+      (expect= a "all your base"))
    (let [a (b64:decode "ZmVubmVsIHRoaW5n")]
-      (assert (= a "fennel thing")  a))
+      (expect= a "fennel thing"))
    (let [a (b64:decode "TWFueSBoYW5kcyBtYWtlIGxpZ2h0IHdvcms=")]
-      (assert (= a "Many hands make light work") (view  a)))
+      (expect= a "Many hands make light work"))
    (let [a (b64:encode "hello world")]
-      (assert (= a "aGVsbG8gd29ybGQ=") a))
+      (expect= a "aGVsbG8gd29ybGQ="))

    (fn check [plain enc]
-      (let [a (b64:encode plain)] (assert (= a enc) (.. "encode " a)))
-      (let [a (b64:decode enc)] (assert (= a plain) (.. "decode " a))))
+      (let [a (b64:encode plain)] (expect (= a enc) (.. "encode " a)))
+      (let [a (b64:decode enc)] (expect (= a plain) (.. "decode " a))))

    (check ""  "")
    (check "f"  "Zg==")
@@ -198,12 +218,10 @@
    (check "foobar"  "Zm9vYmFy")

    (let [x (b64:decode "REtOdUtNS05BNEJWLXdfcUhtNU9YV2liOUxkX3RTdVJTQWVUR0dkWldBdVEyaURObDZ2b3pSbEJwMzlzOEltdkhWdmpzZmMiLCJ5IjoiQVlDY1QwOGZrNFZWZ2lZSVIxbkU4UlJGaGZOSGdBUEFzckRITmJtRGNfUGtWZmdDR0xTMTIweU5SNncwdjd5RUY4WDN1OGpvazhkU0pqN0hnWjZCZHAzcSJ9LCJraWQiOiJlalVDaXBCUE9BeDRWQ1dQdUtkVGlYNDNadW5XTDNjSWN6V1h1RVZyTVNFIn0")]
-      (assert (string.match x "}$") x))
+      (expect (string.match x "}$") x))

    ))

-
-
    ;; doesn't work if the padding is missing
    ;; (let [a (from-base64 "TWFueSBoYW5kcyBtYWtlIGxpZ2h0IHdvcms")]
    ;;   (assert (= a "Many hands make light work") (view  a)))
@@ -212,6 +230,7 @@


 {
+ : append-path
 : assoc
 : base64
 : base64url
@@ -2,6 +2,8 @@
  stdenv
 , dtc
 , lib
+, runCommand
+, writeText
 }:
 { dts
 , includes
@@ -10,14 +12,20 @@
  cppDtSearchFlags = builtins.concatStringsSep " " (map (f: "-I${f}") includes);
  dtcSearchFlags = builtins.concatStringsSep " " (map (f: "-i${f}") includes);
  cmdline = lib.concatStringsSep " " commandLine;
+  chosen = writeText "chosen.dtsi" "/{ chosen { bootargs = ${builtins.toJSON cmdline}; }; };";
+  combined = writeText "combined-dts-fragments"
+    (lib.concatStrings
+      (builtins.map
+        (f: "#include \"${f}\"\n")
+        (dts ++ [ chosen ])));
 in stdenv.mkDerivation {
  name = "dtb";
  phases = [ "buildPhase" ];
  nativeBuildInputs = [ dtc ];
  buildPhase = ''
-    ${stdenv.cc.targetPrefix}cpp -nostdinc -x assembler-with-cpp ${cppDtSearchFlags} -undef -D__DTS__  -o dtb.tmp ${dts}
-    echo '/{ chosen { bootargs = ${builtins.toJSON cmdline}; }; };'  >> dtb.tmp
+    ${stdenv.cc.targetPrefix}cpp -nostdinc -x assembler-with-cpp ${cppDtSearchFlags} -undef -D__DTS__  -o dtb.tmp ${combined}
    dtc ${dtcSearchFlags} -I dts  -O dtb -o $out dtb.tmp
+    # dtc -I dtb  -O dts  $out
    test -e $out
  '';
 }
@@ -29,6 +29,17 @@ let
  '';
 in {
  inherit src;
+
+  # The kernel sources typically used with this version of openwrt
+  # You can find this in `include/kernel-5.15` or similar in the
+  # openwrt sources
+  kernelSrc = pkgsBuildBuild.fetchurl {
+    name = "linux.tar.gz";
+    url = "https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.137.tar.gz";
+    hash = "sha256-PkdzUKZ0IpBiWe/RS70J76JKnBFzRblWcKlaIFNxnHQ=";
+  };
+  kernelVersion = "5.15.137";
+
  applyPatches.ath79 = doPatch "ath79";
  applyPatches.ramips = doPatch "ramips";
  applyPatches.mediatek = doPatch "mediatek"; # aarch64
@@ -56,38 +56,33 @@ let
                    else "";
          in "unlink(${qpathname}); ${cmd} ${chown}";
    in mapAttrsToList (makeFile prefix) attrset;
-  activateScript = attrset: writeText "makedevs.c" ''
-    #include "defs.h"
-    int main(int argc, char* argv[]) {
-      chdir(argv[1]);
-      ${(builtins.concatStringsSep "\n" (visit "." attrset))}
-    }
-  '';
 in attrset:
-  let makedevs = activateScript attrset;
+  let
+    activateScript = writeText "activate.c" ''
+      #include "defs.h"
+      int main(int argc, char* argv[]) {
+        chdir(argv[1]);
+        ${(builtins.concatStringsSep "\n" (visit "." attrset))}
+      }
+    '';
  in stdenv.mkDerivation {
-    name="make-stuff";
+    name="system-configuration";
    src = ./.;

    CFLAGS = "-Os";
    LDFLAGS  = "-static -Xlinker -static";

    postConfigure = ''
-      cp ${makedevs} makedevs.c
+      cp ${activateScript} activate.c
    '';
-    makeFlags = ["makedevs"];
+    makeFlags = ["activate"];
    installPhase = ''
-      closure=${closureInfo { rootPaths = [ makedevs ]; }}
+      closure=${closureInfo { rootPaths = [ activateScript ]; }}
      mkdir -p $out/bin $out/etc
      cp $closure/store-paths $out/etc/nix-store-paths
-      $STRIP --remove-section=.note  --remove-section=.comment --strip-all makedevs -o $out/bin/activate
+      $STRIP --remove-section=.note  --remove-section=.comment --strip-all activate -o $out/bin/activate
      ln -s ${s6-init-bin}/bin/init $out/bin/init
      cp -p ${writeFennel "restart-services" {} ./restart-services.fnl} $out/bin/restart-services
-      # obfuscate the store path of min-copy-closure so that the output
-      # closure doesn't include a bunch of build system stuff
-      f=${buildPackages.min-copy-closure}; f=$(echo $f | sed 's/\(.....\)/\1_/g')
-      substitute ${./build-system-install.sh} $out/install.sh --subst-var-by min-copy-closure $f
-      chmod +x $out/install.sh
      cat > $out/bin/install <<EOF
      #!/bin/sh -e
      prefix=\''${1-/}
@@ -16,10 +16,18 @@

 (print (.. "TFTP serving from " options.base-directory))

-(fn merge-pathname [directory filename]
-  (if (directory:match "/$")
-      (.. directory  filename)
-      (.. directory "/" filename)))
+;; this is a copy of anoia append-path
+(fn  merge-pathname [dirname filename]
+  (let [base (or (string.match dirname "(.*)/$") dirname)
+        result []]
+    (each [component (string.gmatch filename "([^/]+)")]
+      (if (and (= component "..") (> (# result) 0))
+          (table.remove result)
+          (= component "..")
+          (error "path traversal attempt")
+          true
+          (table.insert result component)))
+    (.. base "/" (table.concat result "/"))))

 (->
 (tftp:listen
@@ -1,4 +1,4 @@
-(local { : system : assoc : split : dup : table= : dig } (require :anoia))
+(local { : system : assoc : split : dup : table= : dig : append-path } (require :anoia))
 (local svc (require :anoia.svc))
 (import-macros { :  define-tests : expect : expect= } :anoia.assert)

@@ -13,14 +13,14 @@
  (when (not (table= old-tree new-tree))
    (io.stderr:write "new ssh keys\n")
    (each [username pubkeys (pairs new-tree)]
-      (with-open [f (assert (io.open (.. path "/" username) :w))]
+      (with-open [f (assert (io.open (append-path path username) :w))]
        ;;  the keys are "1" "2" "3" etc, so pairs not ipairs
        (each [_ k (pairs pubkeys)]
          (f:write k)
          (f:write "\n")))))
  (each [k v (pairs old-tree)]
    (when (not (. new-tree k))
-      (os.remove (.. path "/" k))))
+      (os.remove (append-path path k))))
  new-tree)

 (define-tests
@@ -3,7 +3,6 @@ let
  liminix = (import ./default.nix {
    device = (import ./devices/qemu);
    liminix-config = ./vanilla-configuration.nix;
-    inherit nixpkgs;
  });
  here = builtins.toString ./.;
 in liminix.buildEnv.overrideAttrs (o: {
@@ -1,12 +1,18 @@
-set timeout 10
+set timeout 40
+fconfigure stdout -buffering  none

 set when [lindex $argv 0];

+send_user "\n\n\n\n#################################\n running $when"
+
+
 spawn socat -,echo=0,icanon=1 unix-connect:vm/monitor
 set monitor_id $spawn_id
+fconfigure $monitor_id -buffering none

 spawn socat unix-connect:vm/console -
 set console_id $spawn_id
+fconfigure $console_id -buffering  none

 proc chat {instr outstr} {
    expect {
@@ -19,14 +25,20 @@ proc adddevice { } {
    global monitor_id console_id spawn_id
    set spawn_id $monitor_id

+    send_user "\n#### inserting usb device"
+    send "\r\n"
    chat "QEMU" "device_add usb-storage,bus=xhci.0,drive=usbstick\n"
-    chat "(qemu)" "version\r"

    set spawn_id $console_id
+
    expect {
      "sda: sda1" { }
-      timeout { exit 1 }
+      timeout {
+          puts stderr "timeout waiting for disk"
+          exit 1
+      }
    }
+    send_user "done inserting usb device"       
 }

 if { $when eq "early" } {
@@ -35,20 +47,25 @@ if { $when eq "early" } {

 expect "BusyBox"
 chat "#" "PS1=RE\\ADY_\\ ; stty -echo \r"
+# script needs klogd output to tell when sda1 appears. Then
+# run s6-rc -b  -a list to wait for booting to finish
 chat "READY_"  "tail -f /run/log/current & \rs6-rc -b  -a list\r"

-chat "mount" "\r"
-
 if { $when eq "late" } {
  adddevice
 }

-send "\r"
-set timeout 20
+send_user "\n\n\nwaiting for mount to happen"

-chat "READY_" "sleep 5; grep /srv /proc/mounts && hostname\r"
-
-expect {
-  "inout" { }
-  timeout { exit 1 }
+set FINISHED 0
+set EXIT "1" 
+while { $FINISHED < 10 } {
+  chat "READY_" "grep /srv /proc/mounts\r\n"
+  expect {
+    "backup-disk" { set FINISHED 20; set EXIT 0; }
+    "READY_" { send_user "waiting ...\n" ; send "\r\n"; sleep 6 }
+  }
+  set FINISHED [ expr $FINISHED + 1 ]
 }
+
+exit $EXIT
@@ -18,13 +18,17 @@ mkfs.ext2 -L backup-disk ./vm/stick.e2fs
 dd if=/dev/zero of=./vm/stick.img bs=1M count=38
 dd if=./vm/stick.e2fs of=./vm/stick.img bs=512 seek=34 conv=notrunc
 parted -s ./vm/stick.img -- mklabel gpt mkpart backup-disk ext2 34s -0M
+sync
+cp  ./vm/stick.img  ./vm/stick.img.orig

 {

+
 ${img}/run.sh  --background ./vm --flag -device --flag usb-ehci,id=xhci --flag -drive  --flag if=none,id=usbstick,format=raw,file=$(pwd)/vm/stick.img
 expect ${./script.expect} late
 kill $(cat ./vm/pid)

+cp  ./vm/stick.img.orig ./vm/stick.img
 ${img}/run.sh  --background ./vm --flag -device --flag usb-ehci,id=xhci --flag -drive  --flag if=none,id=usbstick,format=raw,file=$(pwd)/vm/stick.img
 expect ${./script.expect} early

@@ -4,7 +4,7 @@ let
 in {
  imports = [
    ../../vanilla-configuration.nix
-    ../../modules/squashfs.nix
+    ../../modules/outputs/squashfs.nix
    ../../modules/outputs/jffs2.nix
  ];
  config.rootfsType = "jffs2";
@@ -17,6 +17,8 @@ serverstatedir=$(mktemp -d -t routeros-XXXXXX)
 # python scapy drags in matplotlib which doesn't enjoy running in
 # a sandbox with no $HOME, hence this environment variable
 export MPLCONFIGDIR=$(mktemp -d -t routeros-XXXXXX)
+export XDG_CONFIG_HOME=/tmp
+export XDG_CACHE_HOME=/tmp

 . ${../test-helpers.sh}
Author	SHA1	Message	Date
Arnout Engelen	f8a275d1a3	use Linux kernel sources associated with openwrt by default	2024-12-24 12:30:15 +00:00
dan	0ee9c76c33	think	2024-12-24 12:30:15 +00:00
dan	bc20f4c6b7	rt3200 test install	2024-12-23 23:59:52 +00:00
dan	848214d104	add ubivolume output	2024-12-23 22:37:07 +00:00
dan	ede8f12d2b	declare options.hardware.ubi unconditionally this is so it can be defined in device modules even when ubifs is not included in the configuration	2024-12-23 22:37:07 +00:00
dan	6cd5b90678	outputs.rootubifs -> ubifs	2024-12-23 22:37:07 +00:00
dan	db4f098c02	add fit bootloader this is for the belkin rt3200, whose uboot doesn't do extlinux but can load a fit from a ubifs. It adds the a kernel+dtb as /boot/fit	2024-12-23 11:21:58 +00:00
dan	1347937345	rename file	2024-12-23 10:31:22 +00:00
dan	a7b5f80674	rename extlinux output to bootfiles this is in preparation for introducing other non-extlinux modules that populate /boot	2024-12-23 00:09:31 +00:00
dan	5c78338d71	make mtdutils use no-systemd util-linux	2024-12-22 23:24:11 +00:00
dan	ed02d02767	bump NEWS	2024-12-22 21:12:36 +00:00
dan	f07a38b0fd	extract uimage output module into own file	2024-12-22 21:10:07 +00:00
dan	ac189f2977	outputs.zimage -> outputs.kernel.zImage remove config option/derivation in favour of accessing as output of the kernel derivation (matches what we do with e.g. modulesupport)	2024-12-22 17:27:59 +00:00
dan	ebb4d4a831	think	2024-12-22 16:03:24 +00:00
dan	6bfbdf352d	bordervm: expose ssh on port 2222	2024-12-22 16:01:38 +00:00
dan	4ea1cf7f32	rt3200 better install docs	2024-12-20 22:26:40 +00:00
dan	f60b74f415	add a new updater output this is so that we don't have to obfuscate store paths in systemConfiguration to avoid dragging in build system deps. breaking-ish change to workflows, docs updated	2024-12-20 00:05:07 +00:00
dan	812e35b7b9	systemconfig: improve filenames/pathnames no more make-stuff	2024-12-19 22:28:30 +00:00
dan	172f368633	fix markup	2024-12-19 21:59:04 +00:00
dan	1af9a39db1	omnia: delete pstore config we're probably not using	2024-12-19 20:59:52 +00:00
dan	420552ce98	add omnia to ci	2024-12-19 20:56:05 +00:00
dan	56c667cfd5	extract systemConfiguration into its own output module	2024-12-19 20:55:10 +00:00
dan	f9b4f0bc9c	move modules/squashfs.nix into outputs/	2024-12-19 14:33:50 +00:00
dan	ba5e4704a0	add short note about persistent logs	2024-12-18 23:08:28 +00:00
dan	3357d21d7f	enlarge pmsg buffer to full size of ramoops region 4k was a piddly amount and we weren't using the rest of it for anything else	2024-12-18 21:16:49 +00:00
dan	ffaca615ba	copy logs to /dev/pmsg0 when ogging.persistent.enabled	2024-12-18 21:11:58 +00:00
dan	77cd4492b2	unbreak nix-shell	2024-12-17 23:26:56 +00:00
dan	81f5550bf0	config.logging.persistent enables /dev/pmsg0 - whatever's written to /dev/pmsg0 appears as /sys/fs/pstore/pmsg-ramoops-0 after reboot - only works on devices with the relevant device tree support (gl-ar750 and whatever has it by default) - nothing in the system is actually writing this file yet - or reading it at boot time, for that matter	2024-12-17 23:24:31 +00:00
dan	b52133a28b	add hardware.dts.includes option	2024-12-17 20:36:14 +00:00
dan	44caefcd3b	rename config.hardware.dts.includes -> includePaths (1) it's a better name (2) I want to use `includes` to specify dtsi files	2024-12-17 17:41:53 +00:00
dan	6e6b8790eb	think	2024-12-17 17:24:52 +00:00
dan	2e5a8a572e	tufted: more robust merge-pathname impl	2024-12-17 17:24:40 +00:00
dan	464d046b5a	append-path spec behaviour for repeated /	2024-12-17 17:24:16 +00:00
dan	ac8b971cc0	new fn append-path in anoia complains if you try to ../../../	2024-12-11 17:26:44 +00:00
dan	13087d17e3	use assert macros in anoia/init.fnl there is no circularity (maybe there was once?)	2024-12-11 17:25:39 +00:00
dan	5572c0ecb0	rewrite parts of inout test to be differently wrong	2024-12-09 23:36:22 +00:00
dan	4cbe3ba683	add some debug output in inout test	2024-12-09 21:00:11 +00:00
dan	20f4a12689	inout: improve robustness, maybe?	2024-12-07 16:02:42 +00:00
dan	33e5c436d5	add environment variables that scapy now needs (24.11)	2024-12-04 21:16:36 +00:00
dan	cde30bcd54	in nixos 24.11 chrony no longer expects nss/nspr/readline	2024-12-03 21:39:54 +00:00