Notes to My Younger Self: The Wins and the Reverts
At minimum frame size, a saturated gigabit wire gives you roughly 672 ns per packet — round it to a 1-microsecond budget — and when the machine enforcing that budget is an embedded Raspberry Pi CM4, the compiler feels like an adversary long before it becomes an ally. Building AncileNet™ (45k+ lines, v0.6.0, 800+ tests) taught me that real performance work is mostly ruthless subtraction.
A note on the numbers below: the per-packet timings and throughput deltas come from Criterion benchmarks on an x86_64 dev machine, not the CM4 — they show the direction and size of each change, not the target hardware's absolute ceiling. The CM4-specific evidence is the capture ring: a saturated link went from ~33% buffer drops to zero.
01. What I Would Do Again (The Winning Bets)
Zero-Copy Stack Lifetimes: slicing raw frames (&'a [u8]) straight off DMA ring buffers kept the hot path 100% allocation-free.
Dual-Plane Kernel Offload: O(1) CIDR drops installed at kernel prerouting priority -300, ahead of conntrack, paired with userspace NFQueue heuristics.
Privacy-First Cryptographic Profiling: pairing JA4/JA4S TLS client-server fingerprints removed any need for invasive MITM decryption.
Proactive NatJack Defense: automated /proc/sys kernel posture checks (/api/posture) plus real-time LAN address-conflict alerts (ip_mac_conflict), aimed at the modern NAT attack classes covered at Black Hat USA 2026 (CVE-2026-56179 / CVE-2026-56181 / CVE-2026-63913).
02. What I Got Wrong, and What I Didn't Build
Benchmarks Blind to the Feature: a single runtime-policy rule — the product's headline parental-control feature — cost 42% of packet throughput, and ten rules cost 63%. None of the benchmark workloads initialized a policy store, so every run exercised the rules-disabled fast path and the regression went unnoticed. Fixing the benchmark first, then normalizing rule targets once at load and matching against borrowed slices, took the ten-rule case from 20 heap allocations per packet to zero — a +52.5% cumulative gain across six changes in that pass.
The Optimization That Wasn't: hand-packing FlowKey's derived Hash into a single write, instead of the usual five hasher writes, looked like a clear win. It measured 14.3% slower and got reverted. That negative result now sits in the test's doc comment, so the next person with the same idea finds the measurement instead of redoing the work.
On-Device Threat Feed Parsing: compiling multi-megabyte blocklists directly on router RAM caused memory spikes. Moving that work to an upstream release pipeline — ed25519-signed manifests, precompiled Bloom/FST filters — fixed it for good.
Silent DNS Dropping: a silently dropped packet means a 5–10s browser hang. Injecting a synthetic NXDOMAIN (RCODE=3) instead gives a clean 0 ms fallback.
Fixed-Size Ring Buffers: a statically sized buffer dropped ~33% of packets on a saturated link; deriving the capture buffer from system RAM at startup instead brought that to zero.
The Feature I Didn't Build: once you're measuring drops, a ring that resizes itself under load looks like the obvious next step — and it would have been a vulnerability instead. pcap_set_buffer_size only accepts an inactive handle, so any resize means reopening capture and dropping packets during the gap; a control loop driven by load would hand an attacker a window of degraded inspection for nothing more than a traffic burst. So the ring is sized once, at startup, and left alone after that. Sustained loss instead produces a specific recommendation, and distinguishes buffer overrun — where a bigger ring would help — from interface-level drops, where it wouldn't.
03. The Core Takeaway
Line rate inside a ~7.7 MB binary on embedded hardware doesn't come from stacking abstractions — it comes from respecting the hardware, killing lock contention before it can start, and designing with mechanical sympathy in mind.
Production Footprint
800+ Tests
100% passing test suite (unit, integration & seeded fuzz)
~7.7 MB
Static Musl + Mimalloc zero-dependency binary size
45k+ LOC
Memory-safe Rust with zero hot-path heap allocations
0 Drops
Memory-derived capture ring sizing on RPi CM4
Hot-Path Memory
Heap Allocs
Zero-Copy Stack
Rule Evaluation
Runtime Strings
Zero-Alloc +52.5%
Blocklist Updates
On-Device Parse
Signed Manifests
DNS Blocking UX
5-10s Timeout
Instant NXDOMAIN
NAT Attack Defense
Unaudited Kernel
NatJack Posture
