Topic
Distributed
Distributed systems fundamentals: consensus, simulation testing, fault injection, reliability engineering, and coordination primitives for large-scale services.
29 posts · ~215 min of reading
Oct 5, 2026
JumpBackHash: Constant-Time Consistent Hashing by Walking Active Indices Backwards
JumpBackHash (Ertl, arXiv:2403.18682) replaces JumpHash's O(log n) floating-point loop with an integer-only walk over power-of-two intervals, consuming under 5/3 PRG outputs on average. I ported it and every formula reproduced. On an M3 Pro (n ≥ 100) it runs at 7.5–20 ns versus JumpHash's 59–221 ns. But its bit-sharing optimizations break something the paper never tests: on multi-node resizes, each departing shard's keys spread to survivors with χ²/dof up to 1035, versus 1.0 for JumpHash.
Sep 29, 2026
Eg-walker: Replaying Event Graphs Instead of Choosing Between OT and CRDTs
Eg-walker (Gentle & Kleppmann, EuroSys 2025 best paper, arXiv 2409.14252) stores collaborative text edits as an immutable event graph and builds CRDT state only transiently, during merges — then throws it away. The result: OT-style steady-state memory (just the text), CRDT-style merge complexity O((k+m) log(k+m)), and a trace where OT needs an hour and eg-walker needs 24 ms. I built a miniature walker with prepare/effect versions and retreat/advance, and verified convergence over every valid replay order — after my first, naive concurrency tie-break diverged, which is itself the lesson: the walker's inner ordering logic must be a real CRDT.
Sep 28, 2026
WaferLLM: The PLMR Model, Interleaved Cannon Shifts, and LLM Inference on 850,000 Cores
WaferLLM (OSDI '25) is the first LLM inference system built for wafer-scale chips like the Cerebras WSE-2: 850,000 cores, 48 KB of SRAM each, arranged in a 2D mesh with no shared memory. The paper's PLMR device model explains why GPU-style allgather, SUMMA, and even Cannon's algorithm all break on this hardware, and the fixes — an interleaved cyclic shift that bounds every communication step to two physical hops, a K-tree allreduce for GEMV, and shift-based KV cache placement — deliver 606x faster GEMV than an A100 and 10-20x end-to-end speedups over multi-GPU SGLang. The constraints are as interesting as the wins.
Sep 27, 2026
Autobahn: Seamless BFT — Lanes, f+1 Proofs of Availability, and Committing a Whole Backlog in One Cut
Autobahn (SOSP '24) attacks the failure mode nobody benchmarks: what happens *after* the network blip ends. View-based BFT protocols like HotStuff suffer 'hangovers' that outlast the disruption; DAG-based protocols recover gracefully but pay ~2× latency every day. Autobahn separates dissemination into per-replica lanes certified by just f+1 votes, then has consensus order a 'cut' of lane tips — so one post-blip slot commits the entire accumulated backlog. Same throughput as Bullshark, half the latency.
Sep 20, 2026
TernFS: An Exabyte Filesystem Built on 256 Shards That Never Talk to Each Other
XTX Markets open-sourced TernFS in September 2025: a distributed filesystem running at 500+ PB across 30,000 disks with zero bytes lost. Its core bet is aggressive simplification — files are immutable, metadata lives in 256 RocksDB shards that never communicate, cross-directory renames funnel through a single serial coordinator, and durability comes from Reed-Solomon (10,4) with 4 KiB CRC32-C interleaving. The math behind the design holds up: 1.4× storage overhead vs 3× for mirroring, per-source reads of ~0.67 GB to evacuate a 20 TB drive, and CRC linearity over XOR that lets checksums compose with parity.
Sep 14, 2026
LazyLog: Deferring Total Order to Read Time for 1-RTT Shared Log Appends
LazyLog (SOSP '24) observes that shared-log applications need linearizable total order at read time, not append time. By binding records to global positions lazily — after acknowledging appends but before reads arrive — its Erwin implementations cut append latency 3.8x versus Corfu and two orders of magnitude versus Scalog, while keeping reads on a fast path whenever readers lag writers by even one ordering interval.
Sep 7, 2026
Diskless Kafka: What Happens When the Log's Replication Protocol Becomes a PUT
KIP-1150 and KIP-1163 propose a leaderless Kafka topic type where any broker accepts produces, batches from many partitions share one WAL object, and a batch coordinator assigns offsets after upload. I worked through the design and ran the cost model: the cross-AZ advantage converges to ~41x on AWS, and the KIP's own P50 latency target carries ~2x headroom over the sum of its stated components.
Aug 22, 2026
Rateless IBLTs: Set Reconciliation Without Knowing the Difference Size
Every set reconciliation scheme in production makes you guess the answer before you compute it. Invertible Bloom Lookup Tables need the difference size up front; guess low and decoding fails outright, guess high and you burn bandwidth. Rateless IBLTs (SIGCOMM 2024) remove the guess entirely by turning the sketch into an infinite stream of coded symbols with a 1/(1+αi) mapping density. I re-derived the closed-form index generator and simulated the peeling decoder to check the 1.35x overhead claim.
Aug 20, 2026
Two Slots and a Square Root: Optimal Weak Isolation Testing
Checking whether a database actually delivered Read Committed is polynomial time, which sounds like the end of the story until you notice the state of the art was degree six. AWDIT (PLDI 2025) gets Read Committed and Read Atomic to n^1.5 and Causal Consistency to n*k, and proves you cannot do better without fast matrix multiplication. I reimplemented the core pass and fuzzed 20,000 histories to find out where the cleverness actually lives.
Aug 19, 2026
Reusable Subrings: Scheduling Collectives When Rewiring Costs Milliseconds
Optical circuit switches let you rewire a cluster mid-collective, but a 3D MEMS crossbar takes 15 ms to settle — a lifetime next to a 1.7 µs step. I worked through Bridge (arXiv:2605.12766), verified its subring structure in Python, and derived the closed-form condition it never states: the optimal number of reconfigurations depends on exactly two things, ln n and the dimensionless ratio δ/c. Solve (x−1)e^x + 1 = δ/c and you have it.
Aug 14, 2026
Running TPC-H on NCCL: The Broadcast That Refuses to Scale
A distributed SQL engine built on the collective communication library used for ML training runs all 22 TPC-H queries at 1TB in 0.53 seconds on 40 H100s. The abstract leads with that. Their own breakdown figure leads somewhere else: 8 GPUs in one machine do it in 1.13 seconds, so 5x the hardware bought 2.13x. This post derives why, verifies the paper's shuffle-vs-broadcast crossover condition, finds a non-monotonicity in it the paper doesn't mention, and computes the cost-per-query that makes scaling out the worse deal.
Aug 10, 2026
Disaggregated State in Flink 2.0: Making 23ms Object Storage Behave Like a 68µs Disk
Streaming engines have kept state on local disk because remote storage is two orders of magnitude slower. Flink 2.0 moves state to DFS anyway, and the trick that makes it work is not caching, it is Little's Law.
Aug 10, 2026
Kernel-Initiated RDMA: Why Large-Scale MoE Decode Broke the CPU Proxy
Every GPU that sends a network message has to get a work request into a NIC queue, and for years a CPU thread did that on the GPU's behalf. Large-scale expert parallelism broke that arrangement, but not for the reason usually given. The proxy round trip is only 2.3% of a decode layer's budget. This post derives the number that actually matters, checks DeepEP's published latency table against its own bandwidth column, and finds that one of the DeepSeek-V3 hardware paper's design justifications compares against a baseline that never occurs.
Aug 7, 2026
Uncertified DAGs: How Mysticeti Cut BFT Consensus Latency by Deleting the Certificates
DAG-based Byzantine consensus bought throughput by paying for it in latency: every block gets explicitly certified by a supermajority before it can even be considered for ordering, which costs three message delays before the three-delay consensus starts. Mysticeti removes the certification round entirely and recovers the same guarantee by reading patterns in the DAG structure. The result went to production on a 106-validator network and cut median latency 4.75x.
Aug 4, 2026
Asynchronous RL for Reasoning Models: How Decoupling Rollout from Training Breaks PPO's Math
Synchronous RL post-training wastes most of its inference fleet waiting on the single longest reasoning trace in each batch. Decoupling generation from training recovers that idle time, but it quietly invalidates the assumption PPO's importance ratio is built on, and naive async training drops AIME24 accuracy from 42.0 to 23.3. AReaL fixes the schedule and the objective together.
Aug 3, 2026
Prequal: The Load Balancer That Wins by Not Balancing Load
Weighted round-robin can hold every replica's CPU inside its allocation on a one minute chart and still time out a quarter of your queries. Prequal (NSDI '24) replaces load equalization with a probe pool and a two signal lexicographic rule, cutting YouTube tail latency 2x and driving errors at 1.74x allocation from 25 percent to exactly zero. The interesting part is why the near perfect load balancer is the loser.
Aug 3, 2026
TokenWeave: Why Hiding Tensor-Parallel Communication Needs Wave Arithmetic, Not Finer Tiles
Tensor-parallel LLM inference burns up to 20% of its latency in AllReduce, and every framework ships with compute-communication overlap turned off by default. The reason is not laziness: splitting work finer to create overlap costs more than the communication it hides. TokenWeave (MLSys 2026) fixes this with two unglamorous ideas, an unequal split sized to GPU wave boundaries and a fused AllReduce-RMSNorm kernel that runs on 8 SMs out of 132.
Jul 27, 2026
Helix Parallelism: Why Tensor Parallelism Hits a Wall at Million-Token Context
Tensor parallelism has a hard ceiling that most people never hit, because you only hit it when your KV cache is measured in millions of tokens. Once TP width exceeds the number of KV heads, adding GPUs stops reducing per-GPU KV traffic entirely, and the arithmetic says so plainly. Helix Parallelism breaks the ceiling by using a different sharding strategy for attention than for the FFN, in the same layer, on the same GPUs, microseconds apart.
Jul 15, 2026
Accord: Leaderless Strict-Serializable Transactions in One Round Trip
Every distributed transaction protocol you have used pays a latency tax: either a leader that funnels all writes through one node, or two round trips to order operations. Accord, the protocol behind Apache Cassandra's general-purpose transactions, delivers strict serializability with no leader and one wide-area round trip on the happy path. The trick is a reorder buffer that turns bounded clock skew into a consensus guarantee.
Jul 9, 2026
CALM Theorem: When Distributed Systems Can Skip Coordination Entirely
The CALM theorem proves that monotone programs never need coordination for consistency, giving us a compiler-verifiable criterion for when distributed systems can safely avoid locks, consensus, and barriers.
Jul 9, 2026
CRDTs and Automerge 2.0: How Columnar Encoding Achieves 1000x Compression for Conflict-Free Collaboration
Conflict-Free Replicated Data Types promise coordination-free merging, but naive implementations explode in memory. Automerge 2.0's columnar encoding compresses operation histories by three orders of magnitude, making CRDTs practical for real-time collaborative editing at scale.
Jul 9, 2026
Hybrid Logical Clocks: Bridging Physical Time and Causality in Distributed Databases
How Hybrid Logical Clocks combine NTP-synchronized physical time with Lamport causality to provide globally meaningful timestamps without coordination, enabling snapshot isolation and serializable transactions across geo-distributed databases.
Jul 9, 2026
Mixture-of-Experts: Expert Parallelism, All-to-All Routing, and Auxiliary-Loss-Free Load Balancing
How MoE models like Mixtral and DeepSeek-V3 route tokens to sparse expert networks across GPU clusters, and why auxiliary-loss-free routing solves the capacity collapse problem without degrading model quality.
Jul 9, 2026
RDMA Consensus: One-Sided Reads and Sub-Microsecond Replication
How DARE, Hermes, and Mu exploit one-sided RDMA verbs to achieve consensus in under 2 microseconds, eliminating CPU involvement on followers and redefining the latency floor for replicated state machines.
Jul 9, 2026
Shared Log Architecture: How Cloud-Native Databases Disaggregate Storage from Compute
The shared log pattern decouples compute from storage by treating a replicated, append-only log as the single source of truth. This deep dive explores how systems like Aurora DSQL, Neon, and FoundationDB leverage this architecture to achieve independent scaling, instant recovery, and strong consistency without distributed two-phase commit.
Jul 8, 2026
Ring Attention: Distributing Million-Token Contexts Across Devices
How Ring Attention eliminates the memory wall for long-context transformers by overlapping blockwise attention computation with KV-cache communication in a ring topology, enabling near-linear context scaling across devices.
Jul 7, 2026
S3 Conditional Writes: The CAS Primitive That Killed the Coordination Sidecar
For years, every database built on object storage needed a DynamoDB table or a ZooKeeper cluster on the side just to answer "who is the writer?" In late 2024, S3 quietly shipped If-Match and If-None-Match support on PutObject, turning the object store itself into a compare-and-swap register. Here is why that one HTTP header changes how you architect storage systems, and how projects like SlateDB use it for formally verified writer fencing.
Jul 5, 2026
Deterministic Simulation Testing: Finding Distributed Systems Bugs Before They Exist
FoundationDB shipped a distributed database with essentially zero customer-reported bugs by running its entire cluster inside a single-threaded simulation driven by one random seed. TigerBeetle and Antithesis have since pushed the idea further. Here is how deterministic simulation testing works, why a seed is worth a thousand log files, and where the technique's real limits are.
Jul 5, 2026
Two Workloads in a Trench Coat: Prefill/Decode Disaggregation in LLM Serving
Prefill and decode have opposite hardware profiles, and serving them on the same GPUs wastes both. A practical tour of DistServe and Mooncake, the two papers behind the biggest architecture shift in LLM inference.