liminix/doc/development.adoc

= For Developers

== Contributions

Patches welcome!

* if you have an obvious bug fix, new package, documentation
  improvement or other uncontroversial small patch, send it straight
  in.

* if you have a large new feature or design change in mind, please
  please _get in touch_ to talk about it before you commit time to
  implementing it. Perhaps it isn't what we were expecting, most
  likely we will have ideas or advice on what it should do or how it
  should be done.

Liminix development is not tied to Github or any other particular
forge. How to send changes:

1. Push your Liminix repo with your changes to a git repository
somewhere on the Internet that I can clone from. It can be on Codeberg
or Gitlab or Sourcehut or Forgejo or Gitea or Github or a bare repo in
your own personal web space or any kind of hosting you like.

2. Email devel@liminix.org with the URL of the repo and the branch
name.

If that's not an option, I’m also happy for you to send your changes
direct to the list itself, as an incremental git bundle or using git
format-patch. We'll work it out somehow.

The main branch of Liminix is hosted at
<https://gti.telent.net/dan/liminix>, with a mirror at
<https://github.com/telent/liminix>. You can clone from either of
those repos.

=== Code of Conduct

Liminix is dedicated to providing a harassment-free experience for everyone. We do not tolerate harassment of participants in any form.

The Liminix
https://gti.telent.net/dan/liminix/src/commit/7bcf6b15c3fdddafeda13f65b3cd4a422dc52cd3/CODE-OF-CONDUCT.md[Code
of Conduct] applies to all Liminix spaces, including the IRC channel,
mailing lists, and any other forums both online and off. Anyone who
violates this code of conduct may be sanctioned or expelled from these
spaces at the discretion of the project leadership.

=== Nix language style

This section describes some Nix language style points that we attempt to
adhere to in this repo. Some are more aspirational than actual.

* indentation and style is according to `nixfmt-rfc-style`
* favour `+callPackage+` over raw `+import+` for calling derivations or
any function that may generate one - any code that might need `+pkgs+`
or parts of it.
* prefer `+let inherit (quark) up down strange charm+` over
`+with quark+`, in any context where the scope is more than a single
expression or there is more than one reference to `+up+`, `+down+` etc.
`+with pkgs; [ foo bar baz]+` is OK,
`+with lib; stdenv.mkDerivation { ... }+` is usually not.
* `+<liminix>+` is defined only when running tests, so don't refer to it
in "application" code
* the parameters to a derivation are sorted alphabetically, except for
`+lib+`, `+stdenv+` and maybe other non-package "special cases"
* where a `+let+` form defines multiple names, put a newline after the
token `+let+`, and indent each name two characters
* to decide whether some code should be a package or a module? Packages
are self-contained - they live in `+/nix/store/eeeeeee-name+` and don't
directly change system behaviour by their presence or absense. modules
can add to `+/etc+` or `+/bin+` or other global state, create services,
all that side-effecty stuff. Generally it should be a package unless it
can't be.

=== Copyright

The Nix code in Liminix is MIT-licenced (same as Nixpkgs), but the code
it combines from other places (e.g. Linux, OpenWrt) may have a variety
of licences. Copyright assignment is not expected:
just like when submitting to the Linux kernel you retain the copyright
on the code you contribute.

== Development tools

In this section we describe some tools to make the edit/build/run
development cycle less painful than flashing a new image on a hardware
device every time.

// FIXME if this is still true we should fix it
In general, packages and tools that run on the "build" machine are
available in the `+buildEnv+` derivation and can most easily be added to
your environment by running `+nix-shell+`.

=== Emulated devices

Liminix has a number of emulated device descriptions which generate
images suitable for running on your build machine using the free
http://www.qemu.org[QEMU machine emulator]. They are

* `qemu`(MIPS)
* `qemu-armv7l`(32 bit ARM)
* `qemu-aarch64` (64 bit ARM)

This is useful for developing userland without needing to keep
flashing or messing with U-Boot: it also enables testing against
emulated network peers using
https://wiki.qemu.org/Documentation/Networking#Socket[QEMU socket
networking], which may be preferable to letting Liminix loose on your
actual LAN. To build,

[source,console]
----
nix-build -I liminix-config=path/to/your/configuration.nix --arg device "import ./devices/qemu" -A outputs.default
----

This creates a `+result/+` directory containing a `+vmlinux+` and a
`+rootfs+`, and a shell script `+run.sh+` which invokes QEMU to run
that kernel with that filesystem. It connects the Liminix serial console
and the https://www.qemu.org/docs/master/system/monitor.html[QEMU
monitor] to stdin/stdout. Use `^P` (not `^A`) to switch to the monitor.

// FIXME should add a `connect.sh` script instead of requiring nix-shell here

If you run with `+--background /path/to/some/directory+` as the first
parameter, it will fork into the background and open Unix sockets in
that directory for console and monitor. Use `+nix-shell --run
connect-vm+` to connect to either of these sockets, and ^O to
disconnect.

[[qemu-networking]]
==== Networking

VMs can network with each other using QEMU socket networking. We observe
these conventions, so that we can run multiple emulated instances and
have them wired up to each other in the right way:

* multicast 230.0.0.1:1234 : access (interconnect between router and
"isp")
* multicast 230.0.0.1:1235 : lan
* multicast 230.0.0.1:1236 : world (the internet)

Any VM started by a `+run.sh+` script is connected to "lan" and
"access". The emulated upstream (see below) runs PPPoE and is
connected to "access" and "world".

==== Upstream connection

In pkgs/routeros there is a derivation to install and configure
https://mikrotik.com/software[Mikrotik RouterOS] as a PPPoE access
concentrator connected to the `+access+` and `+world+` networks, so that
Liminix PPPoE client support can be tested without actual hardware.

This is made available as the `+routeros+` command in `+buildEnv+`, so
you can do something like:

....
mkdir ros-sockets
nix-shell
nix-shell$ routeros ros-sockets
nix-shell$ connect-vm ./ros-sockets/console
....

to start it and connect to it. Note that by default it runs in the
background. It is connected to "access" and "world" virtual networks and
runs a PPPoE service on "access" - so a Liminix VM with a PPPOE client
can connect to it and thus reach the virtual internet. [ check, but
pretty sure this is not the actual internet ]

[.title-ref]#Liminix does not provide RouterOS licences and it is your
own responsibility if you use this to ensure you're compliant with the
terms of Mikrotik's licencing. It may be supplemented or replaced in
time with configurations for RP-PPPoE and/or Accel PPP.#

=== Hardware devices

==== TFTP

[[tftpserver]]
How you get your image onto hardware will vary according to the device,
but is likely to involve taking it apart to add wires to serial console
pads/headers, then using U-Boot to fetch images over TFTP. The OpenWrt
documentation has a
https://openwrt.org/docs/techref/hardware/port.serial[good explanation]
of what you may expect to find on the device.

[[tufted]]
`tufted` is a rudimentary TFTP server which runs from the command
line, has an allowlist for client connections, and follows symlinks,
so you can have your device download images direct from the
`+./result+` directory without exposing `+/nix/store/+` to the
internet or mucking about copying files to `+/tftproot+`. If the
permitted device is to be given the IP address 192.168.8.251 you might
do something like this:

[source,console]
----
nix-shell --run "tufted -a 192.168.8.251 result"
----

Now add the device and server IP addresses to your configuration:

[source,nix]
----
boot.tftp = {
  serverip = "192.168.8.111";
  ipaddr = "192.168.8.251";
};
----

and then build the derivation for `+outputs.default+` or
`+outputs.mtdimage+` (for which it will be an alias on any device where
this is applicable). You should find it has created

* `+result/firmware.bin+` which is the file you are going to flash
* `+result/flash.scr+` which is a set of instructions to U-Boot to
download the image and write it to flash after erasing the appropriate
flash partition.

NOTE: TTL serial connections typically have no form of flow control and so
don't always like having massive chunks of text pasted into them - and
U-Boot may drop characters while it's busy. So don't necessarily expect
to copy-paste the whole of `+boot.scr+` into a terminal emulator and
have it work just like that. You may need to paste each line one at a
time, or even retype it.

==== Running from RAM

For a faster edit-compile-test cycle, you can build a TFTP-bootable
image which boots directly from RAM (using phram) instead of needing
to be flashed first. In your device configuration add

[source,nix]
----
imports = [
  ./modules/tftpboot.nix
];
----

and then build `+outputs.tftpboot+`. This creates a file `+result/boot.scr+`, which you can copy and paste into U-Boot to
transfer the kernel and filesystem over TFTP and boot the kernel from
RAM.

[[bng]]
==== Networking

You probably don't want to be testing a device that might serve DHCP,
DNS and routing protocols on the same LAN as you (or your colleagues,
employees, or family) are using for anything else, because it will
interfere. You also might want to test the device against an "upstream"
connection without having to unplug your regular home router from the
internet so you can borrow the cable/fibre/DSL.

`+bordervm+` is included for this purpose. You will need

* a Linux machine with a spare (PCI or USB) ethernet device which you
can dedicate to Liminix
* an L2TP service such as https://www.aa.net.uk/broadband/l2tp-service/

You need to "hide" the Ethernet device from the host so that QEMU has
exclusive use of it. For PCI this means configuring it for VFIO
passthru; for USB you need to unload the module(s) it uses. I have
this segment in my build machine's `configuration.nix` which you may
be able to adapt:

[source,nix]
----
boot = {
  kernelParams = [ "intel_iommu=on" ];
  kernelModules = [
    "kvm-intel" "vfio_virqfd" "vfio_pci" "vfio_iommu_type1" "vfio"
  ];

  postBootCommands = ''
    # modprobe -i vfio-pci
    # echo vfio-pci > /sys/bus/pci/devices/0000:01:00.0/driver_override
  '';
  blacklistedKernelModules = [
    "r8153_ecm" "cdc_ether"
  ];
};
services.udev.extraRules = ''
  SUBSYSTEM=="usb", ATTRS{idVendor}=="0bda", ATTRS{idProduct}=="8153", OWNER="dan"
'';
----

Then you can execute `+run-border-vm+` in a `+buildEnv+` shell, which
starts up QEMU using the NixOS configuration in
`+bordervm-configuration.nix+`.

Inside the VM

* your Liminix checkout is mounted under `+/home/liminix/liminix+`
* TFTP is listening on the ethernet device and serving
`+/home/liminix/liminix+`. The server IP address is 10.0.0.1
* a PPPOE-L2TP relay is running on the same ethernet card. When the
connected Liminix device makes PPPoE requests, the relay spawns L2TPv2
Access Concentrator sessions to your specified L2TP LNS. Note that
authentication is expected at the PPP layer not the L2TP layer, so the
PAP/CHAP credentials provided by your L2TP service can be configured
into your test device - bordervm doesn't need to know about them.

To configure bordervm, you need a file called `+bordervm.conf.nix+`
which you can create by copying and appropriately editing
`+bordervm.conf-example.nix+`

NOTE: If you make changes to the bordervm configuration after executing
`+run-border-vm+`, you need to remove the `+border.qcow2+` disk image
file otherwise the changes won't get picked up.

== Running tests

You can run all of the tests by evaluating `+ci.nix+`, which is the
input I use in Hydra.

[source,console]
----
nix-build -I liminix=`pwd`  ci.nix -A pppoe # run one job
nix-build -I liminix=`pwd`  ci.nix -A all # run all jobs
----

== Porting to new hardware

// FIXME add this

TBD

== Troubleshooting

=== Diagnosing unexpectedly large images

Sometimes you can add a package and it causes the image size to balloon
because it has dependencies on other things you didn't know about. Build
the `+outputs.manifest+` attribute, which is a JSON representation of
the filesystem, and you can run `+nix-store --query+` on it.

[source,console]
----
nix-build -I liminix-config=path/to/your/configuration.nix \
  --arg device "import ./devices/qemu" -A outputs.manifest \
  -o manifest
nix-store -q --tree manifest
----