Posts

I recently started looking into LoRA + RL, and as a side project I dug into how SGLang implements LoRA serving. This post is a concise walkthrough. TL;DR At a high level, SGLang’s LoRA support looks like this (source: https://arxiv.org/abs/2311.03285): it separates base-model computation from LoRA-adapter computation, computes them independently, and adds them together. This enables one base model to serve multiple adapters. To batch requests that use different adapters, SGLang uses SGMV (Segmented Gather Matrix-Vector Multiplication) (source: https://arxiv.org/pdf/2310.18547), so a mixed-adapter batch can still be handled efficiently with shared kernels. ...

When you spin up an LLM with SGLang, you get this satisfying sequence: progress bars flying, safetensors loading, and then — boom — the model starts generating tokens. But have you ever wondered what’s actually happening under the hood? From allocating GPU memory, to reading weights off disk, to wiring everything together so the model can actually run inference — there’s a lot going on. Specifically, questions like: How does SGLang know which module to use to load a given model? How does it know which weight in a safetensors file maps to which module? And even if it knows the module, how does it know which parameter inside that module a weight belongs to? A single module (like an MLP layer) gets reused across many layers — how does each instance know which weights are “its own”? I recently had to dig into all of this while adding support for a new model in SGLang, so let’s walk through it together. ...

Ref: https://www.bilibili.com/video/BV1darmBcE4A/ I listened to the n+e podcast and wrote down the key points and my takeaways. The conversation ranges from childhood learning habits to building RL infrastructure, open source philosophy, and how to think about research vs. industry. Impressions I first read n+e’s blog in my senior year. I already felt he was exceptional and very original in his thinking. The podcast confirmed it: he is genuinely talented and I learned a lot. What also stood out is that even for someone like him, life is not a smooth path. He faced setbacks in informatics competitions and graduate applications. It reminded me to look at my own situation from a higher perspective and not get trapped in local optima. ...

Ref: Ilya Sutskever’s talk on AI development and future directions I recently listened to a fascinating talk by Ilya Sutskever, co-founder of OpenAI, discussing the current state and future directions of AI development. His insights on scaling, reinforcement learning, and AGI are particularly thought-provoking. Disclaimer: Personally, I found this talk quite scattered without a clear mainline. Some issues were discussed in a very vague and high-level way without providing concrete answers. I’ve organized what I heard based on my notes—feel free to skim for some general sense, but I strongly encourage you to watch the original talk yourself for the most direct information. ...

Ref: https://www.bilibili.com/video/BV1s6pgzLE3y/?spm_id_from=333.337.search-card.all.click&vd_source=a360bf9c247675ec6bfc3395495c3873 I recently had the privilege of listening to a talk by Chen Tianqi, the creator of well-known open-source projects like TVM, XGBoost, and MXNet. His story is not just a chronicle of a tech leader’s growth, but a profound exploration of choice, perseverance, and courage. The Early Years: A Self-Taught ACMer Chen’s journey into research began in high school, where he taught himself ACM (International Collegiate Programming Contest) and even wrote his own compiler. This experience taught him the immense power of learning and collaboration through online forums. At Shanghai Jiao Tong University, he joined the ACM class, where he learned to “do things low-key, but be a person high-key” and to balance ambition with a cooperative spirit. He also credits the class’s mandatory student presentations for significantly improving his learning and public speaking skills. ...

Ref: https://www.xiaoyuzhoufm.com/episode/688cc1cc8e06fe8de7d920cd?s=eyJ1IjoiNjE4M2U4NGJlMGY1ZTcyM2JiNjgzZDViIn0%3D The world of Artificial Intelligence often highlights groundbreaking models and dazzling applications. But beneath the surface, a critical, often-overlooked component is driving this revolution: AI Infrastructure (AI Infra). Just as a powerful engine is essential for a high-performance car, robust AI Infra is the backbone of cutting-edge AI. Understanding AI Infra: A Cloud Analogy At its core, AI Infra can be understood through an analogy with cloud computing: Hardware: Think of AI chips (like GPUs) as the fundamental building blocks. Software: This mirrors cloud services, encompassing: Infra as a Service (IaaS): Providing raw compute, storage, and networking. Platform as a Service (PaaS): Offering development platforms and tools. Software as a Service (SaaS): This includes training and inference frameworks, which are crucial for developing and deploying AI models. Similar to the early days of search engine infrastructure, building AI Infra requires immense initial computational power and highly specialized capabilities. To achieve excellence in AI, core infrastructure is indispensable. We are currently in a pivotal moment for AI Infra development. ...

References: Interview on AI Infrastructure Career Path and Insights from AWS/Meta Job Transition and Interview Preparation for AI Infra in North America AI infrastructure has emerged as a key enabler in the acceleration of artificial intelligence technologies, supporting both training and inference processes. From scaling large AI models to optimizing hardware for inference, AI infra professionals are crucial to building the foundational platforms that power next-gen AI applications. Here’s an in-depth look at the latest developments in the field and how they are shaping career paths in AI infrastructure. ...

From AIEngineer: RAG Agents in Prod: 10 Lessons We Learned In the rush toward AI adoption, everyone is chasing cutting-edge models, fine-tuning pipelines, and building impressive prototypes. But the looming question remains: Where’s the ROI? Many companies pour massive resources into AI initiatives only to end up with sleek demos and very little actual business impact. This disconnect runs deeper than tooling—it’s rooted in what we expect AI to be good at. Much like Moravec’s Paradox in robotics (where high-level reasoning is easier for machines than basic perception or motor skills), modern AI excels at things we assume are hard—like coding, summarization, or legal analysis—but struggles with what we think should be easy: understanding context. ...

References: YouTube Video by Jordan Has No Life Kafka Summary Notes This article summarizes my personal understanding after watching the aforementioned video and reading the Kafka paper. I am still a student, so any inaccuracies or misunderstandings are unintentional—corrections are welcome. Kafka is a widely adopted distributed streaming platform. In this article, I will explore the key trade-offs Kafka makes in order to achieve its design goal of efficient log processing, and how these decisions contribute to Kafka’s architecture and performance. ...

The idea of building a personal site started back in September last year. I got a student-discounted instance from Alibaba Cloud (2 cores, 2GB RAM), and even went through the hassle of registering a domain name: changyi.fun. At first, I used WordPress. It worked — until it didn’t. I remember in one of my security classes, the professor specifically called out WordPress for being notoriously insecure. Around the same time, I tried to set up an SSL certificate (in the most complicated way possible via DNS challenge), and ended up completely breaking the site. I didn’t know back then that there were simpler ways. ...