1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
|
opentracker design notes
========================
Decisions that cost measurement to reach, and the alternatives that were
tried and rejected.
What the tracker is optimised for
---------------------------------
UDP carries most of the traffic. An HTTP request costs about ten times a UDP
announce, 26 microseconds of tracker CPU against 2.5. Nearly all of that is
the kernel setting up and tearing down a connection rather than tracker
code.
A connection carries one request and closes. Keep-alive works and is rarely
asked for.
Statistics are approximate on purpose. Counters are per thread and summed
when read, so a /stats read can under-report once. Nothing drifts.
Decisions
---------
One UDP socket per worker, where the kernel spreads datagrams over a group
bound to one address. Cost per datagram then stays flat as workers are added,
0.71 to 0.72 microseconds from one worker to eight. Sharing one socket grows
it eightfold over the same range, 0.71 to 6.07.
listen.udp.workers defaults to 1. One worker beats none by 19 to 37 per cent
everywhere, because a blocking read costs one system call where the event
loop costs two. Beyond one the gain depends on the kernel balancing, so
raising it is the operator's decision.
A thread parked in a blocking read is woken by a signal, not by closing or
shutting down its socket:
mechanism Linux FreeBSD macOS
close() blocked blocked 0 ms
shutdown(SHUT_RD) 1 ms 1 ms blocked
SO_RCVTIMEO works works works
pthread_kill 1 ms 1 ms 1 ms
Only the signal works everywhere. A receive timeout also works, but wakes
every thread on every tick for the life of the process to catch something
that happens once: at eight workers it doubled idle context switches, 255 to
495 over thirty seconds.
Each binary is built in one compiler invocation. Without object files a build
cannot link objects compiled with different FEATURES, which was a real
failure mode. It costs incremental rebuilds, 0.075 seconds becoming 1.56.
A socket is read or written, never both. http_interest_set replaces an
add-and-drop pair so that the invariant cannot be broken by accident, and the
event layer hears about a change once instead of twice.
Accept waits until the client's request has arrived: TCP_DEFER_ACCEPT on
Linux, the accf_data filter on FreeBSD. Measured against a flood of silent
connections, the filter serves more requests than no filter, not fewer, so it
is not the denial of service vector that FreeBSD-SA-02:26 once described.
Paths not taken
---------------
accf_http, the HTTP-aware accept filter. It duplicates a parser the tracker
has anyway, puts more kernel code in the path, and holds a connection that
sends "GET /" and stops, which accf_data hands straight over.
recvmmsg and sendmmsg. No measurable gain on either FreeBSD machine, between
-4 and +10 per cent, and they do not exist on Darwin.
Link time optimisation. It cuts .text by 13 per cent, 93,497 bytes to 81,630,
and changes throughput not at all. A tracker waits on memory and system
calls, not on instruction fetch: the announce path is 6.6 KB of code against
a 32 KB instruction cache at worst.
A directory for object files. bmake has no pattern rules and the Makefile
builds under FreeBSD's make today. Building each binary in one invocation
clears the objects out of the tree without giving that up.
A src/ directory. 52 files sharing an ot_ prefix are a module list rather
than clutter, and build output was most of what made the tree look crowded.
Platform differences that cost us code
--------------------------------------
SO_REUSEPORT means three different things. Linux spreads datagrams over the
group; FreeBSD permits the bind and needs SO_REUSEPORT_LB to spread. Darwin
hands every datagram to whichever socket bound last, so a group there starves
all but one reader.
Waking a blocked reader differs per kernel. see the table above.
Deferring accept is TCP_DEFER_ACCEPT, or accf_data, or nothing.
make is bmake on FreeBSD and GNU make elsewhere, and the Makefile has to
build under both.
|
