Topic
Distributed
Distributed systems fundamentals: consensus, simulation testing, fault injection, reliability engineering, and coordination primitives for large-scale services.
22 posts · ~166 min of reading
Aug 22, 2026
Rateless IBLTs: Set Reconciliation Without Knowing the Difference Size
Every set reconciliation scheme in production makes you guess the answer before you compute it. Invertible Bloom Lookup Tables need the difference size up front; guess low and decoding fails outright, guess high and you burn bandwidth. Rateless IBLTs (SIGCOMM 2024) remove the guess entirely by turning the sketch into an infinite stream of coded symbols with a 1/(1+αi) mapping density. I re-derived the closed-form index generator and simulated the peeling decoder to check the 1.35x overhead claim.
Aug 20, 2026
Two Slots and a Square Root: Optimal Weak Isolation Testing
Checking whether a database actually delivered Read Committed is polynomial time, which sounds like the end of the story until you notice the state of the art was degree six. AWDIT (PLDI 2025) gets Read Committed and Read Atomic to n^1.5 and Causal Consistency to n*k, and proves you cannot do better without fast matrix multiplication. I reimplemented the core pass and fuzzed 20,000 histories to find out where the cleverness actually lives.
Aug 19, 2026
Reusable Subrings: Scheduling Collectives When Rewiring Costs Milliseconds
Optical circuit switches let you rewire a cluster mid-collective, but a 3D MEMS crossbar takes 15 ms to settle — a lifetime next to a 1.7 µs step. I worked through Bridge (arXiv:2605.12766), verified its subring structure in Python, and derived the closed-form condition it never states: the optimal number of reconfigurations depends on exactly two things, ln n and the dimensionless ratio δ/c. Solve (x−1)e^x + 1 = δ/c and you have it.
Aug 14, 2026
Running TPC-H on NCCL: The Broadcast That Refuses to Scale
A distributed SQL engine built on the collective communication library used for ML training runs all 22 TPC-H queries at 1TB in 0.53 seconds on 40 H100s. The abstract leads with that. Their own breakdown figure leads somewhere else: 8 GPUs in one machine do it in 1.13 seconds, so 5x the hardware bought 2.13x. This post derives why, verifies the paper's shuffle-vs-broadcast crossover condition, finds a non-monotonicity in it the paper doesn't mention, and computes the cost-per-query that makes scaling out the worse deal.
Aug 10, 2026
Disaggregated State in Flink 2.0: Making 23ms Object Storage Behave Like a 68µs Disk
Streaming engines have kept state on local disk because remote storage is two orders of magnitude slower. Flink 2.0 moves state to DFS anyway, and the trick that makes it work is not caching, it is Little's Law.
Aug 10, 2026
Kernel-Initiated RDMA: Why Large-Scale MoE Decode Broke the CPU Proxy
Every GPU that sends a network message has to get a work request into a NIC queue, and for years a CPU thread did that on the GPU's behalf. Large-scale expert parallelism broke that arrangement, but not for the reason usually given. The proxy round trip is only 2.3% of a decode layer's budget. This post derives the number that actually matters, checks DeepEP's published latency table against its own bandwidth column, and finds that one of the DeepSeek-V3 hardware paper's design justifications compares against a baseline that never occurs.
Aug 7, 2026
Uncertified DAGs: How Mysticeti Cut BFT Consensus Latency by Deleting the Certificates
DAG-based Byzantine consensus bought throughput by paying for it in latency: every block gets explicitly certified by a supermajority before it can even be considered for ordering, which costs three message delays before the three-delay consensus starts. Mysticeti removes the certification round entirely and recovers the same guarantee by reading patterns in the DAG structure. The result went to production on a 106-validator network and cut median latency 4.75x.
Aug 4, 2026
Asynchronous RL for Reasoning Models: How Decoupling Rollout from Training Breaks PPO's Math
Synchronous RL post-training wastes most of its inference fleet waiting on the single longest reasoning trace in each batch. Decoupling generation from training recovers that idle time, but it quietly invalidates the assumption PPO's importance ratio is built on, and naive async training drops AIME24 accuracy from 42.0 to 23.3. AReaL fixes the schedule and the objective together.
Aug 3, 2026
Prequal: The Load Balancer That Wins by Not Balancing Load
Weighted round-robin can hold every replica's CPU inside its allocation on a one minute chart and still time out a quarter of your queries. Prequal (NSDI '24) replaces load equalization with a probe pool and a two signal lexicographic rule, cutting YouTube tail latency 2x and driving errors at 1.74x allocation from 25 percent to exactly zero. The interesting part is why the near perfect load balancer is the loser.
Aug 3, 2026
TokenWeave: Why Hiding Tensor-Parallel Communication Needs Wave Arithmetic, Not Finer Tiles
Tensor-parallel LLM inference burns up to 20% of its latency in AllReduce, and every framework ships with compute-communication overlap turned off by default. The reason is not laziness: splitting work finer to create overlap costs more than the communication it hides. TokenWeave (MLSys 2026) fixes this with two unglamorous ideas, an unequal split sized to GPU wave boundaries and a fused AllReduce-RMSNorm kernel that runs on 8 SMs out of 132.
Jul 27, 2026
Helix Parallelism: Why Tensor Parallelism Hits a Wall at Million-Token Context
Tensor parallelism has a hard ceiling that most people never hit, because you only hit it when your KV cache is measured in millions of tokens. Once TP width exceeds the number of KV heads, adding GPUs stops reducing per-GPU KV traffic entirely, and the arithmetic says so plainly. Helix Parallelism breaks the ceiling by using a different sharding strategy for attention than for the FFN, in the same layer, on the same GPUs, microseconds apart.
Jul 15, 2026
Accord: Leaderless Strict-Serializable Transactions in One Round Trip
Every distributed transaction protocol you have used pays a latency tax: either a leader that funnels all writes through one node, or two round trips to order operations. Accord, the protocol behind Apache Cassandra's general-purpose transactions, delivers strict serializability with no leader and one wide-area round trip on the happy path. The trick is a reorder buffer that turns bounded clock skew into a consensus guarantee.
Jul 9, 2026
CALM Theorem: When Distributed Systems Can Skip Coordination Entirely
The CALM theorem proves that monotone programs never need coordination for consistency, giving us a compiler-verifiable criterion for when distributed systems can safely avoid locks, consensus, and barriers.
Jul 9, 2026
CRDTs and Automerge 2.0: How Columnar Encoding Achieves 1000x Compression for Conflict-Free Collaboration
Conflict-Free Replicated Data Types promise coordination-free merging, but naive implementations explode in memory. Automerge 2.0's columnar encoding compresses operation histories by three orders of magnitude, making CRDTs practical for real-time collaborative editing at scale.
Jul 9, 2026
Hybrid Logical Clocks: Bridging Physical Time and Causality in Distributed Databases
How Hybrid Logical Clocks combine NTP-synchronized physical time with Lamport causality to provide globally meaningful timestamps without coordination, enabling snapshot isolation and serializable transactions across geo-distributed databases.
Jul 9, 2026
Mixture-of-Experts: Expert Parallelism, All-to-All Routing, and Auxiliary-Loss-Free Load Balancing
How MoE models like Mixtral and DeepSeek-V3 route tokens to sparse expert networks across GPU clusters, and why auxiliary-loss-free routing solves the capacity collapse problem without degrading model quality.
Jul 9, 2026
RDMA Consensus: One-Sided Reads and Sub-Microsecond Replication
How DARE, Hermes, and Mu exploit one-sided RDMA verbs to achieve consensus in under 2 microseconds, eliminating CPU involvement on followers and redefining the latency floor for replicated state machines.
Jul 9, 2026
Shared Log Architecture: How Cloud-Native Databases Disaggregate Storage from Compute
The shared log pattern decouples compute from storage by treating a replicated, append-only log as the single source of truth. This deep dive explores how systems like Aurora DSQL, Neon, and FoundationDB leverage this architecture to achieve independent scaling, instant recovery, and strong consistency without distributed two-phase commit.
Jul 8, 2026
Ring Attention: Distributing Million-Token Contexts Across Devices
How Ring Attention eliminates the memory wall for long-context transformers by overlapping blockwise attention computation with KV-cache communication in a ring topology, enabling near-linear context scaling across devices.
Jul 7, 2026
S3 Conditional Writes: The CAS Primitive That Killed the Coordination Sidecar
For years, every database built on object storage needed a DynamoDB table or a ZooKeeper cluster on the side just to answer "who is the writer?" In late 2024, S3 quietly shipped If-Match and If-None-Match support on PutObject, turning the object store itself into a compare-and-swap register. Here is why that one HTTP header changes how you architect storage systems, and how projects like SlateDB use it for formally verified writer fencing.
Jul 5, 2026
Deterministic Simulation Testing: Finding Distributed Systems Bugs Before They Exist
FoundationDB shipped a distributed database with essentially zero customer-reported bugs by running its entire cluster inside a single-threaded simulation driven by one random seed. TigerBeetle and Antithesis have since pushed the idea further. Here is how deterministic simulation testing works, why a seed is worth a thousand log files, and where the technique's real limits are.
Jul 5, 2026
Two Workloads in a Trench Coat: Prefill/Decode Disaggregation in LLM Serving
Prefill and decode have opposite hardware profiles, and serving them on the same GPUs wastes both. A practical tour of DistServe and Mooncake, the two papers behind the biggest architecture shift in LLM inference.