diff --git a/THOUGHTS.txt b/THOUGHTS.txt index 8cc0e2b..b5c1c1d 100644 --- a/THOUGHTS.txt +++ b/THOUGHTS.txt @@ -6187,3 +6187,340 @@ video editing * there are ppp credentials onscreen at 13:00 * should finish at 33:00 + + +Sat Sep 7 22:21:12 BST 2024 + +what is causing these messages? + +@400000000001620127a096b3 watch-for-modem-modeswitch /nix/store/4q3swc0mg28ja30anap4id3gics3h7fk-lua-tty-mips-unknown-linux-mus +l/bin/lua: ...lr-uevent-watch-mips-unknown-linux-musl/bin/uevent-watch:4: unexpected symbol near '/' +@400000000001620127a0ec7e watch-for-modem-modeswitch stack traceback: +@400000000001620127a11c42 watch-for-modem-modeswitch [C]: in function 'dofile' +@400000000001620127a14dec watch-for-modem-modeswitch (command line):1: in main chunk +@400000000001620127a41b1c watch-for-modem-modeswitch [C]: in ? +@ + +Sun Sep 8 10:15:56 BST 2024 + +If we could produce logs in JSON then we could push them to zinc (or +elasticsearch, which has the same api). We'd like fields for +timestamp, message, pid, host. + +* we can add host when we post to elasticsearch, no need to repeat it +on every field + +* there is no (sensible) way to get the pid of the other end of a pipe. +But we could print it from the sender before execing the process. But then +it'll only appear once instead of every entry. Maybe we could log the +logger pid as well, then we can correlate + +TBH given that we already have to process the log lines to get them +into zinc, and that we already can unambiguously parse the log line +(provided we disallow whitespace in the service name, and we mandate +that the message is always the final field) there's not much value in +producing a different json. + +Actually the logger pid probably won't help us tell when the service +has been restarted, because the logger won't be restarted at the time +time due to fdholder stuff + +so perhaps there are no logging changes we can easily/reasonably make +and we should just write a log processor that ships to a collector. + +- open connection to zinc (s6-tlsclient) +- send http headers +- while not eof(stdin) + - read line + - split fields + - send command, send data + +{ "index" : { "_index" : "olympics" } } +{"Year": 1896, "City": "Athens", "Sport": "Aquatics", "Discipline": "Swimming", "Athlete": "HAJOS, Alfred", "Country": "HUN", "Gender": "Men", "Event": "100M Freestyle", "Medal": "Gold", "Season": "summer"} + + +we can't calculate content-length. maybe we can use chunks + +Transfer-Encoding: chunked + +size-of-chunk-in-hex CRLF +chunk-data CRLF + +0 CRLF +CRLF + +to generate test data: +$ nix-shell -p s6 --run " sort --random-sort ~/src/liminix/THOUGHTS.txt | head -1 | sed 's/^/servicename /g' |tr -cd '[a-z0-9 ]' | s6-tai64n" + +Sun Sep 8 16:39:55 BST 2024 + +* how do we add incz to the logging infra and configure it? +* how do we get zinc on loaclhost to be visible to test lan (port forward + on border?) +* shall we rig up a service on loaclhost so that zinc starts at boot? + +Mon Sep 9 17:58:46 BST 2024 + +We can use this as a log processor. However, a log processor doesn't +ship the segment until the log writer has finished with it, therefore, +some latency is introduced. + +We can write logs to the network as they are generated. However, what if: + +- the network is not available +- the collector is not keeping up + +s6-log says "if a processor fails, s6-log will try it again after some +cooldown time.". Laurent says "if a processor fails [if you're using +-b] then the rotation cannot happen, and s6-log will stop reading +until the processor succeeds. Without -b, logs keep accumulating in +RAM, and s6-log may crash if it runs oom before the processor +succeeds" + +.... so, maybe we shouldn't use log processors here. + +shipping finished log segments outside of the s6-log framework is +quite straightforward. the issue is how to send the in-progress log. +Challenges + +- if we are sending the in-progress log, how _not_ to resend all the +same entries when the segment is rotated +- how do we know when the segment is rotated and should start reading +the new file + +Maybe + +1) we have a logshipper service that listens on a unix socket +2) s6-log is hooked to the logshipper-client logger, which checks + for the unix socket and only writes data to it if it exists. + Probably it should check periodically for the socket to exist + and not just try it on every write + (Would be good if it could tell whether the socket had a + listener or not. Maybe abstract sockets) + + + (and also writes to stdout so the s6 logging chain is unbroken) +3) when logshipper is ready, it reads all past log entries whose + timestamps are from before when it started, and writes them. + and/or it could write a cookie to the log, then it would know + to stop reading the logs when it encounters its own cookie + +Mon Sep 16 19:54:35 BST 2024 + +incz won't work as-is because it uses stdin/stdout for communicating +with http and reads the logs from a filename. unless we make it use +/proc/self/fd/3 for the filename? Even then, ideally we kind of want +to adapt it for streaming + +Wed Sep 18 18:23:38 BST 2024 + +we can run + +socat tcp-listen:19612,reuseaddr,fork | s6-log -b /var/log/clients + +on the log collection host (or use openssl-listen if you're going to set up ssl certs) + +then our logshipper program can basically be "open socket and cat to s6-tcpclient" (though it would be better if we can add the hostname in the process) + +let's make the logging script a config option + + +pipeline { s6-ipcserver -1 /run/uncaught-logs/shipping } +pipeline { s6-tcpclient loghost:19612 } +fdmove -c 1 7 +cat + +Wed Sep 18 20:32:54 BST 2024 + +s6-log will create its directory but the parent must exist. incidentally, +putting /run/uncaught-logs in pseudofiles is pointless because /run is a +mountpoint + +ip addr add 10.0.2.15/24 dev lan + +s6-rc -d change sshd; s6-rc -u change sshd; + +Sun Sep 22 21:13:15 BST 2024 + +This works for the collector (but note that it collects logs from +*anywhere* that can write to that port, so please firewall responsibly) + + + systemd.services."s6-log-collector" = { + after = [ "network.target" ]; + wantedBy = [ "multi-user.target" ]; + serviceConfig = { + Type = "exec"; + WorkingDirectory = "/var/log"; + ExecStart = '' + ${pkgs.bash}/bin/sh -c "${pkgs.socat}/bin/socat tcp4-listen:17345,reuseaddr,fork stdout | ${pkgs.s6}/bin/s6-log -b /var/log/remote" + ''; + }; + }; + + +Tue Sep 24 18:14:42 BST 2024 + +"Raw" TCP is not the ideal transport for logs because I don't want the +whole internet able to write to my log server, and writing +only-from-the-LAN iptables rules is messy with a gazillion ipv6 +addresses to account for. + +. SSL with client certificates would be nice, but there is the issue +of how to get the private key onto the device and sign it. My idea is + +1) device generates a private key on first boot (or every boot, if no +persistent storage). Private key includes some field with a value that +was set at build time (PSK, effectively) + +2) there is an API-driven CA signing thingy that the client can use to +get a cert based on their key. It checks the field for the presence +of the PSK. It should probably expose HTTPS only so that the client +can be sure it's getting signed by the correct CA + + +3) the log collector refuses connections unless the client is signed by the +local CA + +This is quite a lot of work insofar as it would appear to require +writing the CA. + +We could alternatively do something much more ad hoc where the client +just writes the PSK to the server when it opens the stream, before +sending any data. We'd have to write the server end, then, instead of +just using socat - but that is probably less work than an API-driven CA. +On the other hand, TLS logs would also be encrypted which is a good thing +if the LAN is not trusted. + +Sat Sep 28 16:04:15 BST 2024 + +OK, so we wrote the CA. + +To do HTTPS on the client we need + +1) to generate a csr +2) to https it to the server +3) store the generated thingy as a service output + +looking at x86-64 sizes for ballpark + +-r-xr-xr-x 1 root root 987K Jan 1 1970 /nix/store/s45wy1ssim1dkxzligx09xjp4n0668i2-openssl-3.0.14-bin/bin/openssl +-r-xr-xr-x 1 root root 263K Jan 1 1970 /nix/store/z28bxdnsw2gr1xwx7qj6px9iz5sr84i9-lua-5.3.6-env/lib/lua/5.3/_openssl.so + +suggesting that we'd use _less_ disk doing the whole thing in lua than + +Sun Sep 29 10:20:49 BST 2024 + +We need luaossl support for setting attributes in a CSR + +https://www.rfc-editor.org/rfc/rfc2986#page-5 + + Attributes { ATTRIBUTE:IOSet } ::= SET OF Attribute{{ IOSet }} + + CRIAttributes ATTRIBUTE ::= { + ... -- add any locally defined attributes here -- } + + Attribute { ATTRIBUTE:IOSet } ::= SEQUENCE { + type ATTRIBUTE.&id({IOSet}), + values SET SIZE(1..MAX) OF ATTRIBUTE.&Type({IOSet}{@type}) + } + +I don't understand this 100% but it looks like the raw data is _not_ +the same format as an x509 attribute. See e.g. +https://github.com/golang/go/commit/e78e654c1de0a7bfe0314d6954d42b046f14f1bb#diff-a789286d7e257f148c437404f8cf5d3379688597381ff13352e62ac406be295aL1712 +in support of my hypothesis (background: iiuc, "critical" is a boolean flag, +but x509 attributes aren't allowed to be booleans) + +https://en.wikipedia.org/wiki/Certificate_signing_request + + 388:d=2 hl=2 l= 35 cons: cont [ 0 ] + 390:d=3 hl=2 l= 33 cons: SEQUENCE + 392:d=4 hl=2 l= 9 prim: OBJECT :challengePassword + 403:d=4 hl=2 l= 20 cons: SET + 405:d=5 hl=2 l= 18 prim: UTF8STRING :loves labours lost + 425:d=1 hl=2 l= 13 cons: SEQUENCE + 427:d=2 hl=2 l= 9 prim: OBJECT :sha256WithRSAEncryption + + +(csr:getAttribute "challengePassword") +(csr:getAttributeNames) +(csr:setAttribute "challengePassword" :IA5STRING ["loves labours lost"]) + +how do we know the asn1 type of the attribute values? it looks like they're +defined by the object: see e.g. https://www.rfc-editor.org/rfc/rfc2985#page-16 + + A challenge-password attribute must have a single attribute value. + + ChallengePassword attribute values generated in accordance with this + version of this document SHOULD use the PrintableString encoding + whenever possible. If internationalization issues make this + impossible, the UTF8String alternative SHOULD be used. PKCS #9- + attribute processing systems MUST be able to recognize and process + all string types in DirectoryString values. + +crypto/asn1/tbl_standard.h: {NID_pkcs9_challengePassword, 1, -1, PKCS9STRING_TYPE, 0}, +include/openssl/asn1.h.in:# define PKCS9STRING_TYPE (DIRSTRING_TYPE|B_ASN1_IA5STRING) + +I assume there's something in openssl that will do lookups in this table +to give us the type for the oid, then maybe something in luaossl that +would lua-ize it? +5 + +or we could put that burden on the caller, as x509.name:add does + +Sun Sep 29 20:46:16 BST 2024 + + +OBJ_txt2nid("challengePassword"); works with short/long names + +the get call is complicated because there can be multiple +attributes with the same type. There probably aren't but ... + +(csr:getAttribute "challengePassword") => multivals attr, index + +(csr:getAttribute "challengePassword" index) => multivals attr, index + +(csr:addAttribute "challengePassword" :IA5String ["loves labours"]) + +(csr:clearAttribute index) + + +Tue Oct 1 21:55:25 BST 2024 + +on server, we need to reconfigure socat to give it our CA cert and +expect peer to authenticate + +on client, I am not sure if we can persuade s6-tlsclient to use the same +file for cert and private key. perhaps certifix-client could write +two separate files with --out-key and --out-certificate + + +CAFILE=ca.crt KEYFILE=client.key CERTFILE=client.crt s6-tlsclient -k localhost -y -v localhost 19612 socat 'fd:6!!fd:7' - + + +socat ssl-l:19612,reuseaddr,fork,cert=server-combined.pem,cafile=ca.crt stdout + +suggest creating /var/lib/s6-log-collector/{private,cert} on loaclhost +wherein we keep the server and ca keys, then socat and the signing +server can both see them + + +ip addr add 10.0.2.15/24 dev lan + +Sat Oct 5 22:35:41 BST 2024 + +We had it working in a VM, and the service is installed on loaclhost + +TODO + +1) make a module-based service for client-cert +caCertificateFile +secretFile +subject +url + +2) make the shipping service a consumer-for + +3) can we reduce the verbiage in the shipping service somehow? + +4) rebuild an actual device with all this stuff