From 36e396fd30bf098fa2459f11f4dd4430f7a8396e Mon Sep 17 00:00:00 2001 From: Dirk Engling Date: Fri, 30 Oct 2026 17:29:08 +0100 Subject: Start documenting design decisions and measurments --- DESIGN | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 DESIGN (limited to 'DESIGN') diff --git a/DESIGN b/DESIGN new file mode 100644 index 0000000..ff37706 --- /dev/null +++ b/DESIGN @@ -0,0 +1,99 @@ +opentracker design notes +======================== + +Decisions that cost measurement to reach, and the alternatives that were +tried and rejected. + +What the tracker is optimised for +--------------------------------- + +UDP carries most of the traffic. An HTTP request costs about ten times a UDP +announce, 26 microseconds of tracker CPU against 2.5. Nearly all of that is +the kernel setting up and tearing down a connection rather than tracker +code. + +A connection carries one request and closes. Keep-alive works and is rarely +asked for. + +Statistics are approximate on purpose. Counters are per thread and summed +when read, so a /stats read can under-report once. Nothing drifts. + + +Decisions +--------- + +One UDP socket per worker, where the kernel spreads datagrams over a group +bound to one address. Cost per datagram then stays flat as workers are added, +0.71 to 0.72 microseconds from one worker to eight. Sharing one socket grows +it eightfold over the same range, 0.71 to 6.07. + +listen.udp.workers defaults to 1. One worker beats none by 19 to 37 per cent +everywhere, because a blocking read costs one system call where the event +loop costs two. Beyond one the gain depends on the kernel balancing, so +raising it is the operator's decision. + +A thread parked in a blocking read is woken by a signal, not by closing or +shutting down its socket: + + mechanism Linux FreeBSD macOS + close() blocked blocked 0 ms + shutdown(SHUT_RD) 1 ms 1 ms blocked + SO_RCVTIMEO works works works + pthread_kill 1 ms 1 ms 1 ms + +Only the signal works everywhere. A receive timeout also works, but wakes +every thread on every tick for the life of the process to catch something +that happens once: at eight workers it doubled idle context switches, 255 to +495 over thirty seconds. + +Each binary is built in one compiler invocation. Without object files a build +cannot link objects compiled with different FEATURES, which was a real +failure mode. It costs incremental rebuilds, 0.075 seconds becoming 1.56. + +A socket is read or written, never both. http_interest_set replaces an +add-and-drop pair so that the invariant cannot be broken by accident, and the +event layer hears about a change once instead of twice. + +Accept waits until the client's request has arrived: TCP_DEFER_ACCEPT on +Linux, the accf_data filter on FreeBSD. Measured against a flood of silent +connections, the filter serves more requests than no filter, not fewer, so it +is not the denial of service vector that FreeBSD-SA-02:26 once described. + + +Paths not taken +--------------- + +accf_http, the HTTP-aware accept filter. It duplicates a parser the tracker +has anyway, puts more kernel code in the path, and holds a connection that +sends "GET /" and stops, which accf_data hands straight over. + +recvmmsg and sendmmsg. No measurable gain on either FreeBSD machine, between +-4 and +10 per cent, and they do not exist on Darwin. + +Link time optimisation. It cuts .text by 13 per cent, 93,497 bytes to 81,630, +and changes throughput not at all. A tracker waits on memory and system +calls, not on instruction fetch: the announce path is 6.6 KB of code against +a 32 KB instruction cache at worst. + +A directory for object files. bmake has no pattern rules and the Makefile +builds under FreeBSD's make today. Building each binary in one invocation +clears the objects out of the tree without giving that up. + +A src/ directory. 52 files sharing an ot_ prefix are a module list rather +than clutter, and build output was most of what made the tree look crowded. + + +Platform differences that cost us code +-------------------------------------- + +SO_REUSEPORT means three different things. Linux spreads datagrams over the +group; FreeBSD permits the bind and needs SO_REUSEPORT_LB to spread. Darwin +hands every datagram to whichever socket bound last, so a group there starves +all but one reader. + +Waking a blocked reader differs per kernel. see the table above. + +Deferring accept is TCP_DEFER_ACCEPT, or accf_data, or nothing. + +make is bmake on FreeBSD and GNU make elsewhere, and the Makefile has to +build under both. -- cgit v1.3