All posts

Topic

Systems & Infra

Systems programming, Linux kernel internals, io_uring, memory management, CPU architecture, SIMD, CXL memory pooling, and datacenter hardware. Performance-critical engineering at the metal.

64 posts · ~468 min of reading

Sep 2, 2026

A cloud block store has to answer one question on every I/O: where does logical block N physically live? Answering it per-block costs gigabytes of DRAM per terabyte of attached capacity, and that DRAM is the real cost center of the storage fleet. RASK argues the fix is to stop indexing blocks and index ranges instead, which sounds trivial until you hit overlapping writes and fragmentation. Here is the design, plus my own simulation of where the memory actually goes.

storage indexing systems 9 min

Aug 31, 2026

Every systems course teaches the Young/Daly optimal checkpoint interval. A wave of 2025-2026 LLM training papers quietly ignores it and checkpoints far more often than the formula allows. I simulated why, then verified the trick that makes it safe — torn snapshots repaired by replaying the optimizer on the host — and found a 1.9x traffic reduction the papers leave on the table.

distributed-training fault-tolerance checkpointing 7 min

Aug 31, 2026

For twenty years the sea of nodes was the received wisdom for optimizing compilers. V8 finished replacing it with a CFG-based IR and halved compile time. I tried to reproduce the two mechanical arguments — traversal order and memory locality — and only one of them survives in isolation.

compilers jit javascript 8 min

Aug 30, 2026

Format-aware compressors win big and then rot, because every reader needs the matching decoder. OpenZL embeds the resolved transform graph in each frame so one universal decoder handles all of them. I rebuilt its core thesis on a 10MB record table: structure-aware transforms at zstd -3 hit 2.84x, beating xz -9 on the raw bytes (2.35x) at roughly 60x the throughput — and once the structure is exposed, the entropy backend stops mattering.

compression data-engineering systems 7 min

Aug 28, 2026

CFS could tell you how much CPU a task got, never how soon. EEVDF splits those into two orthogonal knobs — weight sets the share, request size sets the deadline. I simulated it: dropping one task's request from 700us to 100us cut its worst-case wait 4x while its share moved 0.20 percentage points.

linux-kernel scheduling latency 7 min

Aug 27, 2026

Requests-per-minute limits are the industry's fairness mechanism, and they are terrible: to get a tighter fairness gap than a proper fair scheduler, my simulation had to throw away 78% of the GPU. VTC (OSDI '24) ports weighted fair queueing to continuous batching, and the part that matters is not the counter — it is the one line that erases a returning client's banked credit. Without it, an idle client comes back and takes 83.5% of the machine.

llm-inference scheduling fairness 8 min

Aug 27, 2026

Linking is the last stage of the build that refuses to use your machine. lld sits on roughly one core for four of the five seconds it takes to link Firefox. The reason is not that the work is inherently sequential — it is that archive semantics are defined as a left-to-right scan, so parallelizing symbol resolution changes which object files end up in your binary. mold's answer is to replace the scan with a reachability walk, and I reproduced both the win and the corner-case divergence it causes.

compilers linkers parallelism 8 min

Aug 27, 2026

On datacenter workloads, hardware prefetchers run at 24% accuracy — the useless prefetches are 44% of all DRAM traffic. Themis fixes this not with a smarter predictor but with a page-table bit: profile which 4KB pages are prefetch-hostile, mark them in the PTE, and let the prefetcher read the hint off the TLB. I checked the paper's arithmetic, derived where its magic constant comes from, and found the failure mode it never mentions — huge pages.

microarchitecture prefetching operating-systems 8 min

Aug 26, 2026

For twenty years Linux gave you two page sizes: 4K, or a 2M hugepage that inflates a lightly-touched heap by up to 100x. Multi-size THP fills in the orders between them — and on arm64, the contiguous PTE bit turns 16 page table entries into one TLB entry. The interesting engineering is in the access/dirty bits.

linux-kernel memory-management arm64 7 min

Aug 26, 2026

The eBPF verifier tracks your registers bit by bit in a domain called tnums. Multiplication in that domain was replaced upstream in August 2025, and the commit message quietly admits the new version is sometimes less precise. I enumerated all 43 million 8-bit operand pairs to find out where.

ebpf static-analysis abstract-interpretation 8 min

Aug 25, 2026

Every serious attempt at memory-safe C has broken the ABI: fat pointers grow to 128 or 256 bits, structs change layout, and nothing links against anything. Fil-C keeps pointers at 64 bits by storing the capability in an invisible parallel allocation the C program cannot address, then pays for the resulting dangling-capability problem with a concurrent on-the-fly garbage collector. The result compiles CPython, OpenSSH, and Emacs unmodified at roughly 1.5x to 4x native speed.

memory-safety compilers garbage-collection 8 min

Aug 25, 2026

Verifying pointer-manipulating systems code means proving two very different things: that aliasing is sound, and that the arithmetic is right. Verus's central design bet is to refuse to send the first problem to the SMT solver at all — aliasing is discharged by a linear type checker, leaving Z3 a purely functional program to reason about. That split is why a verified hypervisor security module and a verified concurrent page-table subsystem now exist, and why the remaining engineering cost has moved somewhere surprising: quantifier triggers.

formal-verification rust smt-solvers 7 min

Aug 25, 2026

A per-CPU free list needs no atomics — but userspace cannot safely read "which CPU am I on" and then act on the answer, because the scheduler can migrate you between the two. rseq closes that gap by letting the kernel rewind your instruction pointer. Then Linux 6.19 made rseq 15% faster and instantly broke every TCMalloc binary on the planet.

linux-kernel concurrency lock-free 8 min

Aug 24, 2026

Tricolor mark-sweep is a graph flood, and a graph flood is a microarchitectural disaster: Go's collector spent over a third of its marking cycles stalled on memory it could not predict. Green Tea, default in Go 1.26, changes the unit of work from the object to the 8 KiB span. The entire performance case rests on one empirical bet about queue discipline, and it is possible to measure whether that bet pays off.

garbage-collection go runtime 8 min

Aug 19, 2026

A CHERI capability needs a base, a top, and an address, but only gets 128 bits. I reimplemented the CHERI Concentrate encoder from the ISAv9 spec, derived the exact worst-case padding a malloc pays (1/256), and found where the ratifying RISC-V spec quietly moved a constant that changes how far out-of-bounds your pointers may legally roam.

cheri memory-safety hardware 8 min

Aug 19, 2026

Optical circuit switches let you rewire a cluster mid-collective, but a 3D MEMS crossbar takes 15 ms to settle — a lifetime next to a 1.7 µs step. I worked through Bridge (arXiv:2605.12766), verified its subring structure in Python, and derived the closed-form condition it never states: the optimal number of reconfigurations depends on exactly two things, ln n and the dimensionless ratio δ/c. Solve (x−1)e^x + 1 = δ/c and you have it.

networking distributed-systems hpc 7 min

Aug 18, 2026

Linux's DAMON answers 'which memory is cold and for how long' at an overhead set by a region count, not by memory size. I simulated its adaptive region sampler and found the case where its own auto-tuner reports healthy while the access map is noise.

linux memory-management kernel 8 min

Aug 18, 2026

In RNS-CKKS the scarce resource is modulus bits, and a rescale spends a whole prime whether or not the operation needed one. I built the key-switch cost model, reproduced Grafting's measured 1.83x speedup from pure limb counting, and found the case where finer-grained levels make things 4x worse.

homomorphic-encryption cryptography ckks 7 min

Aug 17, 2026

Uniform probing was proven optimal for open addressing in 1985, and conjectured optimal in the worst case too. A 2025 paper broke the conjecture with two constructions that never move an element after inserting it. I implemented both against uniform probing and found where the crossover actually is: 99.9% load, not sooner.

algorithms data-structures hash-tables 7 min

Aug 17, 2026

Every zkVM has to prove that its reads return the last thing written. For a decade that meant permutation checks and grand products. Twist and Shout replace both with one-hot addresses and increments. I implemented the core sum-checks, counted the prover's multiplications, and found the locality optimization pays off in RAM, not registers.

cryptography snarks zkvm 8 min

Aug 16, 2026

Blackwell's tcgen05 instructions break two rules that every GPU matrix-multiply kernel has been built around: accumulators no longer live in registers, and the MMA is no longer collective. I worked through the PTX ISA's Tensor Memory chapter and derived the arithmetic that forced both changes — accumulator traffic per FLOP scales as K/4, and every new tcgen05 kind lands at exactly 2x Hopper's per-SM accumulator bandwidth. That number is the whole design.

gpu cuda blackwell 7 min

Aug 16, 2026

SimplePIR made private database queries fast by turning the database itself into an LWE matrix, then charged every client a 400 MB download for the privilege. YPIR deletes that download entirely. I re-derived the hint arithmetic, then benchmarked the online kernel and found the memory-bandwidth story only holds on wide-vector cores.

cryptography lattices performance 8 min

Aug 15, 2026

A JIT that emits code faster than it can parse its own input sounds like cheating. Copy-and-patch gets there by refusing to compile at all: it precompiles every operation with holes in it at build time, then memcpy's the pieces together and patches the holes. It is the whole reason CPython now ships a JIT, and I measured it at 2.2 nanoseconds per micro-op.

compilers jit python 7 min

Aug 15, 2026

PagedAttention solved KV cache fragmentation by making the cache non-contiguous in virtual memory, and every attention kernel since has paid for that decision in rewrites, register spills, and block-table bookkeeping. vAttention points out that the fragmentation was always a physical-memory problem, and fixes it with CUDA's virtual memory APIs instead. I re-derived the paper's page-size tables and its batch-size result from first principles; both hold, and the arithmetic reveals why they had to patch NVIDIA's driver to ship it.

llm-inference gpu virtual-memory 9 min

Aug 12, 2026

A SNARK prover's real cost is bits committed, not gates executed. Committing a single boolean into a 254-bit field wastes 253 of those bits. Binius removes the waste by replacing prime fields with a tower of binary fields, where a bit and a 128-bit word share one arithmetic.

cryptography zero-knowledge snarks 7 min

Aug 8, 2026

Bufferbloat has a fix nobody could deploy: mark congestion aggressively enough that queues stay under a millisecond. The problem is that any queue tuned that way starves every Reno and CUBIC flow sharing it. L4S solves the deployment problem with one algebraic trick, two queues fed by one controller whose output is squared for the legacy queue and scaled linearly for the new one. This post derives the coupling, then checks the RFC's reference parameters against the qdisc that actually shipped in Linux.

networking congestion-control linux-kernel 8 min

Aug 6, 2026

Linux 6.12 and 6.15 shipped two features that look unrelated: TCP receive directly into GPU memory, and io_uring zero-copy receive into userspace pages. Both rest on one abstraction that inverts a thirty-year assumption in the network stack, that a packet buffer is a struct page the kernel can dereference. The consequence is a receive path where the payload is unreadable to the kernel that routes it, and header/data split stops being an optimization and becomes a correctness requirement.

linux-kernel networking io-uring 8 min

Jul 27, 2026

Tensor parallelism has a hard ceiling that most people never hit, because you only hit it when your KV cache is measured in millions of tokens. Once TP width exceeds the number of KV heads, adding GPUs stops reducing per-GPU KV traffic entirely, and the arithmetic says so plainly. Helix Parallelism breaks the ceiling by using a different sharding strategy for attention than for the FFN, in the same layer, on the same GPUs, microseconds apart.

llm-inference gpu distributed-systems 8 min

Jul 26, 2026

A single-token forward pass through a 1B parameter model should take 740 microseconds on an H100. Production engines take two to three times that, and the reason is not arithmetic, not memory bandwidth, and not the model. It is the roughly one hundred kernel launches per forward pass, each of which drains the memory pipeline dry. Two 2025 projects independently arrived at the same fix: collapse the entire model into one kernel launch and build a scheduler inside it.

gpu cuda llm-inference 8 min

Jul 26, 2026

Everyone repeats that LLM inference is memory bound. Run the arithmetic on a batched 70B serving workload and the opposite falls out: compute is the binding constraint, and yet aggregate GPU utilization sits near 40 percent. The gap is not kernel quality, it is that compute, memory, and NVLink operations execute one after another inside a single device. NanoFlow splits each batch into nano-batches so those three resources run at once, and lands 1.91x over TensorRT-LLM.

llm-inference gpu systems 8 min

Jul 26, 2026

FDP gives the host eight tags to steer writes into physically separate NAND regions, and the pitch is write amplification approaching 1.0 with no application rewrite. The FAST '26 characterization of two shipping FDP drives found the same workload hitting 1.03 on one device and 3.12 on another, both advertising FDP support. The failures are structural: a noisy handle propagates GC pressure to its neighbors, long sequential streams get reclaimed prematurely, and F2FS tags 99% of user data with a single hint that collapses the whole interface back to a conventional SSD.

storage nvme ssd 8 min

Jul 23, 2026

Data-parallel training assumes a fat, low-latency fabric between every GPU. DiLoCo throws that assumption out: workers train independently for hundreds of steps, then exchange a single pseudo-gradient through an outer optimizer. Streaming DiLoCo pushes it further, cutting peak bandwidth another two orders of magnitude so you can train a frontier model over ordinary internet links.

distributed-training llm-training optimization 6 min

Jul 17, 2026

Prompting a model to 'return valid JSON' is a wish, not a guarantee. Constrained decoding turns it into an invariant by masking logits against a finite-state machine or pushdown automaton. The hard part is doing it without adding latency to every single token, which is where compiled indices and context-token classification come in.

llm inference structured-output 8 min

Jul 14, 2026

Pipeline parallelism wastes GPU time in bubbles, idle gaps at the warmup and cooldown of every training step. By splitting the backward pass into its input-gradient and weight-gradient halves and deferring the optimizer's sync, ZB-H1/H2 schedules drive the bubble toward zero, buying up to 31% more throughput under synchronous semantics.

distributed-training llm-training pipeline-parallelism 8 min

Jul 13, 2026

TCP was designed for the wide-area internet, yet it carries the overwhelming majority of datacenter RPC traffic, where its byte-stream abstraction, connection state, and loss-based recovery actively destroy tail latency. Homa is a message-based, connectionless transport that uses receiver-driven flow control and shortest-remaining-processing-time scheduling to cut 99th-percentile latency by an order of magnitude, and it is now being upstreamed into the Linux kernel.

networking datacenter transport-protocol 7 min

Jul 11, 2026

How database engines use WebAssembly to run user-defined functions with near-native speed while maintaining memory safety, deterministic execution, and zero-trust isolation within the query pipeline.

webassembly databases query-engines 7 min

Jul 9, 2026

How DPDK eliminates syscall overhead, interrupt storms, and kernel scheduling jitter to process 100+ million packets per second on commodity hardware, and why the kernel's networking stack becomes the bottleneck at scale.

dpdk kernel-bypass networking 7 min

Jul 9, 2026

How eBPF-based continuous profilers achieve always-on stack trace collection in production with sub-1% CPU overhead using frame pointer unwinding, BPF ring buffers, and adaptive sampling — replacing the traditional trade-off between observability and performance.

ebpf profiling observability 6 min

Jul 9, 2026

Intel disabled TSX across its entire consumer lineup after a decade of security vulnerabilities. ARM's Transactional Memory Extension (TME) is now the last standing hardware TM implementation in commodity silicon. This post dissects how HTM works at the microarchitectural level, why Intel's design was fundamentally flawed, and what ARM's approach changes.

concurrency hardware transactional-memory 8 min

Jul 9, 2026

Project Loom promised millions of concurrent tasks on a handful of OS threads. Synchronized blocks broke that promise by pinning continuations to carriers. JEP 491 in Java 24 eliminates pinning entirely through object monitor reimplementation, closing the last major gap in virtual thread adoption.

java concurrency virtual-threads 7 min

Jul 9, 2026

How modern database systems replace decades-old histogram-based cardinality estimators with neural models — covering CardBench, MSCN, and NeuroCard architectures that reduce join order estimation errors from 1000x to under 3x.

databases query-optimization machine-learning 6 min

Jul 9, 2026

The default malloc implementation in glibc uses a single arena with coarse-grained locking that collapses under thread contention. Modern allocators like jemalloc, mimalloc, and Scudo achieve nanosecond-scale allocation through thread-local free lists, size-class sharding, and virtual memory tricks that eliminate fragmentation without sacrificing throughput.

memory-allocator jemalloc mimalloc 7 min

Jul 9, 2026

ARM's Memory Tagging Extensions (MTE) catch use-after-free and buffer overflows in hardware with 3-5% overhead, compared to 100%+ for software sanitizers. How tag coloring, lock-and-key validation, and probabilistic detection achieve what decades of software-only approaches could not.

memory-safety arm mte 8 min

Jul 9, 2026

mmap() seems like the perfect database buffer pool: let the OS handle page caching, avoid copies, and get a simple pointer interface. In practice, it introduces catastrophic stalls, uncontrollable eviction, and subtle corruption. Here is why every serious DBMS builds its own buffer manager, and the systems-level reasons mmap fails at scale.

databases memory-management operating-systems 7 min

Jul 9, 2026

Modern databases like ScyllaDB and Redpanda abandon thread pools entirely, pinning one thread to each CPU core with zero shared mutable state between them. The result is predictable tail latency at microsecond scale, but the programming model demands rethinking everything from memory allocation to request routing.

systems-architecture performance concurrency 7 min

Jul 8, 2026

How bcachefs uses copy-on-write B-trees with six-point journal entries to achieve crash consistency without the write amplification of traditional journaling, and why this design changes the calculus for next-generation storage engines.

filesystems btrees copy-on-write 7 min

Jul 8, 2026

How Microsoft Research's 1.58-bit LLM architecture replaces floating-point matrix multiplications with integer additions, matching full-precision performance while fundamentally changing inference hardware requirements.

llm-inference quantization hardware-efficiency 7 min

Jul 8, 2026

How the Bw-Tree achieves latch-free concurrent access to B+ tree indexes through delta chains, an indirection mapping table, and epoch-based garbage collection, enabling linear scalability on modern many-core hardware.

data-structures databases concurrency 8 min

Jul 8, 2026

How CXL 3.1 fabric-attached memory eliminates stranded DRAM across server fleets, enabling dynamic memory composition with sub-200ns additional latency through hardware-coherent interconnects.

cxl memory-pooling disaggregated-memory 7 min

Jul 8, 2026

Exploring epoch-based reclamation (EBR), the technique that lets lock-free data structures deallocate memory without garbage collection, from the foundational quiescent-state mechanism through Crossbeam's production implementation to recent advances like PEBR and Hyaline.

concurrency memory-management lock-free 7 min

Jul 8, 2026

How io_uring eliminated the system call overhead that plagued Linux I/O for decades, and why submission queue polling lets the kernel do I/O without ever context-switching.

io_uring linux-kernel systems-programming 6 min

Jul 8, 2026

Linux's legacy page reclaim scanned the entire active list to find cold pages. Multi-Gen LRU replaces that O(n) scan with generation-based aging — cutting memory-pressure stalls by 40% in real workloads.

linux memory-management kernel 7 min

Jul 8, 2026

The Piecewise Geometric Model index uses linear regression segments to predict key positions, achieving O(log log n) point queries with orders-of-magnitude less space than B-trees. Here's the theory, the recursive structure, and why production systems are starting to care.

learned-indexes data-structures databases 8 min

Jul 8, 2026

How SGLang's RadixAttention uses a radix tree to automatically detect and reuse shared prefixes in KV caches, eliminating redundant computation and achieving up to 10x throughput gains for multi-turn LLM workloads.

llm-serving kv-cache radix-tree 6 min

Jul 8, 2026

Inside Ribbon Filters, the Gaussian elimination based probabilistic data structure that replaced Bloom filters in RocksDB, achieving information-theoretic space optimality at the cost of a linear algebra construction step.

data-structures databases probabilistic 8 min

Jul 8, 2026

How Roaring Bitmaps achieve intersection, union, and cardinality on billion-element sets in microseconds by partitioning integers into typed containers, each optimized for its density regime, and accelerated with SIMD vectorization.

data-structures databases indexing 7 min

Jul 8, 2026

Linux 6.12 merged sched_ext, a framework that lets you write CPU scheduler policies as BPF programs, load them at runtime, and swap them without rebooting. Here's how it works, why it matters, and what Meta learned running it in production.

linux-kernel ebpf scheduling 6 min

Jul 8, 2026

How simdjson exploits branch-free SIMD instructions to parse JSON at hardware speeds, achieving multi-gigabyte throughput by treating structural character discovery as a bitwise parallel classification problem.

simd json parsing 6 min

Jul 8, 2026

Flat hash maps based on Swiss Table design now dominate C++, Rust, Go, and Zig standard libraries. The key insight is not a better hash function or collision strategy, it is using SIMD to probe 16 slots in a single instruction, turning the control byte array into a hardware-accelerated Bloom filter.

data-structures performance simd 7 min

Jul 8, 2026

Push-based pipelines, morsel-driven parallelism, and selection vectors: the three architectural bets that let an in-process database saturate modern hardware without a cluster.

databases query-execution simd 6 min

Jul 8, 2026

Traditional deserialization copies bytes into heap-allocated objects, burning CPU and memory bandwidth. Zero-copy formats flip this model: the serialized bytes ARE the in-memory data structure, accessed directly through pointer arithmetic and alignment guarantees.

serialization systems performance 7 min

Jul 6, 2026

EAGLE-3 gets up to 6.5x decoding speedup by abandoning the feature-prediction objective that defined its predecessors. The interesting part is why feature prediction became the bottleneck, and how a trick called training-time test fixes the train/inference mismatch it leaves behind.

llm-inference speculative-decoding performance 6 min

Jul 5, 2026

Prefill and decode have opposite hardware profiles, and serving them on the same GPUs wastes both. A practical tour of DistServe and Mooncake, the two papers behind the biggest architecture shift in LLM inference.

llm-inference kv-cache distributed-systems 7 min

Jul 5, 2026

A 2024 NSDI paper showed that a FIFO queue, one bit per object, and a lazy hand pointer can out-perform LRU, ARC, and friends on web workloads, while removing the lock that makes LRU a scalability bottleneck. Here is how SIEVE works and why its simplicity is the whole point.

caching algorithms systems 7 min