welcome-back-victoria.md (7913B)
1 Date: Tue Oct 21 18:52:38 2025 2 Title: Centralised logging with Liminix and VictoriaLogs 3 Topics: liminix 4 5 It's a year since I wrote [Log off](https://www.liminix.org/2024/10/06/log-off.html), in which I 6 described some ongoing-at-the-time work to make Liminix devices log over the 7 network to a centralised log repo. It's also, and this is entirely a coincidence, 8 a year since I made any kind of progress on it: since that time all my log messages have continued to be written to ramdisk that will 9 be lost forever like tears in the rain. 10 11 This situation was not ideal. I had some time and energy recently to 12 see if I could finish it up and, well, I haven't done that exactly but 13 whereas last time I only *believed* it was substantially finished, 14 this time I believe it is substantially finished. 15 16 It goes a little something like this: 17 18 ## Tap the log pipeline 19 20 Each service in Liminix is connected to its own log process, which is 21 (for 98% of the services) connected to the "fallback logger" which 22 writes the logs to disk (ramdisk) and takes care of log rotation 23 etc. This is standard s6 stuff, we're not innovating here. 24 25 Into the middle of this pipeline we insert a program called 26 [logtap](https://gti.telent.net/dan/liminix/src/branch/main/pkgs/logtap/logtap.c) 27 which copies its input to its output and also to a fifo - but only 28 writes to the fifo if the previous writes worked (i.e. it doesn't back 29 up or stop working if the tap is not connected). The standard output from 30 logtap goes on to the default logger, so local logging is unaffected - 31 which is important if the network is down or hasn't come up yet. 32 33 This is a change from last year's version, which used a unix domain 34 socket instead of a fifo. Two reasons: first, we need to know which 35 messages were sent successfully and which weren't. It was difficult to 36 tell reliably and without latency whether there was anything at the 37 other end of the socket, whereas we learn almost instantly when a fifo 38 write fails. Second, it makes it easier to implement a shipper because 39 it can just open the fifo and read from it, instead of having to call 40 socket functions. 41 42 ## Hang a reader on the tap 43 44 The log shipper opens the other end of the fifo and ... ships the 45 logs. I've chosen 46 [VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaLogs) 47 (wrapped in an HTTPS reverse proxy) as my centralised log service, so 48 my log shipper has to conect with HTTPS to the service endpoint and 49 send ["jsonline" log messages](https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api). In 50 fact, my log shipper just speaks pidgin HTTP on [file descriptors 6 and 7](https://cr.yp.to/proto/ucspi.txt) and leverages 51 [s6-tlsclient](https://skarnet.org/software/s6-networking/s6-tlsclient.html) 52 to do the actual TCP/TLS heavy lifting. 53 54 This is all new since last year when we were just splatting raw logs 55 over a socket connection instead of doing this fancy JSON stuff. It 56 did mean writing a parser for [TAI64N](https://gti.telent.net/dan/liminix/src/branch/main/pkgs/anoia/tai64.fnl) external timestamps and some 57 functions to convert it to UTC: as a matter of principle (read: 58 stubbornness) I do appreciate that my log message timestamps won't go 59 forwards and backwards arbitrarily when leap seconds are decreed, but 60 I guess almost nobody else (at least, neither VictoriaLogs nor Zinc) 61 thinks it's important. 62 63 ```nix 64 # in liminix config 65 logging.shipping = { 66 enable = true; 67 command = 68 let certc = config.services.client-cert; 69 in '' 70 export CERTFILE=$(output_path ${certc} certificate) 71 export CAFILE=$(output_path ${certc} ca-certificate) 72 export KEYFILE=$(output_path ${certc} key) 73 ${pkgs.s6-networking}/bin/s6-tlsclient -j -y -k loghost.example.org \ 74 10.0.0.1 443 \ 75 ${pkgs.logshippers}/bin/victorialogsend https://loghost.example.org/insert/jsonline 76 ''; 77 dependencies = [services.qemu-hyp-route services.client-cert]; 78 }; 79 ``` 80 81 ## ... using the TLS cert you previously requested 82 83 Before the log shipper can start, it needs to get its TLS client 84 certificate, by making a CSR and sending it to 85 [Certifix](https://gti.telent.net/dan/certifix). The 86 [certifix-client](https://gti.telent.net/dan/liminix/src/branch/main/pkgs/certifix-client/certifix-client.fnl) 87 is _almost_ the same as last year's version except that it uses 88 [lua-http](https://daurnimator.github.io/lua-http/0.4/) instead of 89 [fetch-freebsd](https://github.com/jrmarino/fetch-freebsd) as the http 90 interface. This is because last year's version wasn't work 91 when asked to traverse the baroque maze of iptables forwarding and 92 QEMU Slirp networking that lies between my Liminix test network and my 93 VictoriaLogs instance. After a long time staring at pcap dumps I gave 94 up trying to work out why and just rewrote that bit. 95 96 It's important to have an (at least vaguely) accurate clock before 97 attempting HTTPS, because the server certificate has a "not valid 98 before" field, so OpenSSL won't like it if you say it's still 1970. 99 100 ``` 101 # in liminix config 102 services.client-cert = svc.tls-certificate.certifix-client.build { 103 caCertificate = builtins.readFile /var/lib/certifix/certs/ca.crt; 104 subject = "C=GB,ST=London,O=Example Org,OU=devices,CN=${config.hostname}"; 105 secret = builtins.readFile /var/lib/certifix/challengePassword; 106 serviceUrl = "https://loaclhost.lan:19613/sign"; 107 dependencies = [ config.services.ntp ] ; 108 }; 109 ``` 110 111 112 ## ... to connect to an HTTPS reverse proxy 113 114 Originally I planned to put a Lets Encrypt cert in front of 115 Victorialogs, but that would need 500k of CA certificate bundle on 116 each device, which is quite a lot on devices with little flash. So it 117 makes more sense to use the Certifix CA here too. 118 119 Persuading the OpenSSL command line tools to make a CSR with a 120 `challengePassword` was probably as much work as writing something 121 with luaossl would have been - it was certainly messier - but the 122 point is I didn't know that when I started. 123 124 125 ```nix 126 # in nixos configuration.nix 127 systemd.services."loghost-certificate" = 128 let 129 dir = "/var/lib/certifix"; 130 pw = builtins.readFile "${dir}/private/challengePassword"; 131 in { 132 script = '' 133 set -eu 134 cd ${dir} 135 PATH=${pkgs.openssl}/bin:${pkgs.curl}/bin:$PATH 136 openssl req -config <(printf '[req]\nprompt=no\nattributes=attrs\ndistinguished_name=DN\n[DN]"C=GB\nST=London\nO=Example Org\nCN=loghost\n[attrs]\nchallengePassword=${pw}') -newkey rsa:2048 -addext "extendedKeyUsage = serverAuth" -addext "subjectAltName = DNS:loghost.lan,DNS:loghost,DNS:loghost.example.org" -nodes -keyout private/loghost.key --out certs/loghost.csr 137 curl --cacert certs/ca.crt -H 'content-type: application/x-pem-file' --data-binary @certs/loghost.csr https://localhost:19613/sign -o certs/loghost.crt 138 ''; 139 serviceConfig = { 140 Type = "oneshot"; 141 User = "root"; 142 ReadWritePaths = ["/var/lib/certifix"]; 143 StateDirectory = "certifix"; 144 }; 145 startAt = "monthly"; 146 }; 147 ``` 148 149 The proxy itself is just Nginx with `ssl_verify_client` set, but 150 certifix-client holds the https connection open so remember to disable 151 proxy buffering or you aren't getting your logs in any kind of timely 152 fashion. 153 154 ```nix 155 # in nixos configuration.nix 156 services.nginx.virtualHosts."loghost.example.org" = { 157 forceSSL = true; 158 sslTrustedCertificate = /var/lib/certifix/certs/ca.crt; 159 sslCertificateKey = "/var/lib/certifix/private/loghost.key"; 160 sslCertificate = "/var/lib/certifix/certs/loghost.crt"; 161 162 extraConfig = '' 163 ssl_verify_client on; 164 proxy_buffering off; 165 proxy_request_buffering off; 166 ''; 167 168 locations."/".proxyPass = "http://127.0.0.1:9428/"; 169 }; 170 ``` 171 172 Just as I did last year, I'm going to finish by claiming that this is 173 basically finished and it just needs installing on some real 174 devices. Hopefully I'm right this time, though.