Posts

I was reading this post by Su Jianlin when one sentence stopped me: Besides the KV cache, decoding now has another variable — MTP, or speculative decoding, whose idea is to trade compute for speed. But MLA behaves during decoding like an MQA with head_dims=512+, and has already consumed most of the compute up front, so “MLA+MTP” tends to lose out. My first reaction was: what does that mean? Why would MLA “consume compute up front” during decode? And why should it conflict with MTP that its decode FLOPs come out equivalent to a head-dim-512+ MHA? ...

TL;DR: Agent execution comes down to one question — how do you sensibly parallelize a single task across multiple agents? That’s the Bitter Lesson playing out at the collaboration layer, and it’s what 24 hours at a hackathon made obvious to me. I recently spent 24 hours at a hackathon, and the more I think about it, the more convinced I am that the collaboration model in the agentic era is genuinely different from the pre-agent one. This post is less a conclusion than a list of directions I think are worth working on next. ...

TL;DR In LLM RL post-training, every optimizer step on the trainer has to push fresh weights into the inference engine doing rollouts. Under a disaggregated training/inference architecture, this has long been treated as a problem that demands high-bandwidth RDMA — sync cost scales linearly with model size and tends to dominate the sync phase. In the first half of 2026, three independent threads — academic, industrial, and open-source — converged on the same observation: after one RL step, only a small fraction of the weights actually change; in BF16, more than 99% of elements are byte-identical to the previous step. Sending only the changes (the delta) cuts wire traffic by roughly two orders of magnitude, losslessly and bit-identically. The bottleneck shifts from hardware (a high-bandwidth interconnect) to software (how to encode the sparse delta) — RL training can now run over plain Ethernet, and even across data centers backed by shared storage. ...

I recently started looking into LoRA + RL, and as a side project I dug into how SGLang implements LoRA serving. This post is a concise walkthrough. TL;DR At a high level, SGLang’s LoRA support looks like this (source: https://arxiv.org/abs/2311.03285): it separates base-model computation from LoRA-adapter computation, computes them independently, and adds them together. This enables one base model to serve multiple adapters. To batch requests that use different adapters, SGLang uses SGMV (Segmented Gather Matrix-Vector Multiplication) (source: https://arxiv.org/pdf/2310.18547), so a mixed-adapter batch can still be handled efficiently with shared kernels. ...

When you spin up an LLM with SGLang, you get this satisfying sequence: progress bars flying, safetensors loading, and then — boom — the model starts generating tokens. But have you ever wondered what’s actually happening under the hood? From allocating GPU memory, to reading weights off disk, to wiring everything together so the model can actually run inference — there’s a lot going on. Specifically, questions like: How does SGLang know which module to use to load a given model? How does it know which weight in a safetensors file maps to which module? And even if it knows the module, how does it know which parameter inside that module a weight belongs to? A single module (like an MLP layer) gets reused across many layers — how does each instance know which weights are “its own”? I recently had to dig into all of this while adding support for a new model in SGLang, so let’s walk through it together. ...

Ref: https://www.bilibili.com/video/BV1darmBcE4A/ I listened to the n+e podcast and wrote down the key points and my takeaways. The conversation ranges from childhood learning habits to building RL infrastructure, open source philosophy, and how to think about research vs. industry. Impressions I first read n+e’s blog in my senior year. I already felt he was exceptional and very original in his thinking. The podcast confirmed it: he is genuinely talented and I learned a lot. What also stood out is that even for someone like him, life is not a smooth path. He faced setbacks in informatics competitions and graduate applications. It reminded me to look at my own situation from a higher perspective and not get trapped in local optima. ...

Ref: Ilya Sutskever’s talk on AI development and future directions I recently listened to a fascinating talk by Ilya Sutskever, co-founder of OpenAI, discussing the current state and future directions of AI development. His insights on scaling, reinforcement learning, and AGI are particularly thought-provoking. Disclaimer: Personally, I found this talk quite scattered without a clear mainline. Some issues were discussed in a very vague and high-level way without providing concrete answers. I’ve organized what I heard based on my notes—feel free to skim for some general sense, but I strongly encourage you to watch the original talk yourself for the most direct information. ...

Ref: https://www.bilibili.com/video/BV1s6pgzLE3y/?spm_id_from=333.337.search-card.all.click&vd_source=a360bf9c247675ec6bfc3395495c3873 I recently had the privilege of listening to a talk by Chen Tianqi, the creator of well-known open-source projects like TVM, XGBoost, and MXNet. His story is not just a chronicle of a tech leader’s growth, but a profound exploration of choice, perseverance, and courage. The Early Years: A Self-Taught ACMer Chen’s journey into research began in high school, where he taught himself ACM (International Collegiate Programming Contest) and even wrote his own compiler. This experience taught him the immense power of learning and collaboration through online forums. At Shanghai Jiao Tong University, he joined the ACM class, where he learned to “do things low-key, but be a person high-key” and to balance ambition with a cooperative spirit. He also credits the class’s mandatory student presentations for significantly improving his learning and public speaking skills. ...

Ref: https://www.xiaoyuzhoufm.com/episode/688cc1cc8e06fe8de7d920cd?s=eyJ1IjoiNjE4M2U4NGJlMGY1ZTcyM2JiNjgzZDViIn0%3D The world of Artificial Intelligence often highlights groundbreaking models and dazzling applications. But beneath the surface, a critical, often-overlooked component is driving this revolution: AI Infrastructure (AI Infra). Just as a powerful engine is essential for a high-performance car, robust AI Infra is the backbone of cutting-edge AI. Understanding AI Infra: A Cloud Analogy At its core, AI Infra can be understood through an analogy with cloud computing: Hardware: Think of AI chips (like GPUs) as the fundamental building blocks. Software: This mirrors cloud services, encompassing: Infra as a Service (IaaS): Providing raw compute, storage, and networking. Platform as a Service (PaaS): Offering development platforms and tools. Software as a Service (SaaS): This includes training and inference frameworks, which are crucial for developing and deploying AI models. Similar to the early days of search engine infrastructure, building AI Infra requires immense initial computational power and highly specialized capabilities. To achieve excellence in AI, core infrastructure is indispensable. We are currently in a pivotal moment for AI Infra development. ...

References: Interview on AI Infrastructure Career Path and Insights from AWS/Meta Job Transition and Interview Preparation for AI Infra in North America AI infrastructure has emerged as a key enabler in the acceleration of artificial intelligence technologies, supporting both training and inference processes. From scaling large AI models to optimizing hardware for inference, AI infra professionals are crucial to building the foundational platforms that power next-gen AI applications. Here’s an in-depth look at the latest developments in the field and how they are shaping career paths in AI infrastructure. ...