AI Infrastructure
SGLang
Open-source contributor · JUL 2026 — PRESENTMaking multimodal inference faster and more reliable in SGLang-Omni, from speech generation to multi-GPU deployment.
SGLang-Omni ↗Field notes
- CUDA graphs. Profiled text-to-speech inference and added CUDA graph support, cutting first-audio latency by 54–59% and raising request throughput by 15–21% in paired H200 benchmarks at concurrency 16.
- GPU placement. Standardizing GPU assignment across SGLang-Omni stage factories so each stage can be replicated across GPUs, covering 32 model topologies.
- FLOPs accounting. Corrected prefill FLOPs accounting for cached prefixes and batched requests. Long-context workloads had been underestimated by 10–100×; GPU throughput reports are meaningful again.
- Health probes. Separated readiness from liveness so a server still warming up no longer receives traffic, and documented the Kubernetes health-check configuration.
AI-MLFQ
Scheduling simulator · EARLY EXPLORATIONI built a simulator that lets quick chats interrupt long AI jobs. In the recorded experiment, chats started sooner and long jobs finished later; total processing time stayed the same.
Code & experiment results ↗Field notes
- Two ways to queue. I compared serving requests in arrival order with giving long jobs shorter turns. Pausing a long generation lets a quick conversation get a turn sooner.
- Who gains, who waits?. Quick conversations started sooner, while long jobs finished later and the full batch took the same total time. Earlier first replies did not mean smoother streaming: the gaps between pieces of each reply also grew.
- Pausing has a cost. A paused conversation still needs its saved working memory. The simulator accounts for limited space and the delay of moving that memory out and back in.
- Inspect the experiment. I built a visual replay of the queues and memory use to inspect where delays came from. This was my entry into AI infrastructure; the results come from a simulation, not a deployed AI service.
Inference Playground
Interactive teaching website · PERSONAL PROJECTAn interactive website for seeing how people share one AI server. Add users, lengthen their messages, or change who gets a turn to see why replies slow down—all in a browser simulation.
Open interactive demo ↗Field notes
- Watch a reply appear. Follow the demo from reading a prompt to producing an answer a piece at a time. It separates the wait before the first word from the speed of the reply that follows.
- Fill up the memory. Add conversations with a slider and watch the memory meter fill. Longer conversations take more space, so the same computer can handle fewer of them at once.
- Compare two queues. Run the mixed-traffic scenario, then compare first-come, first-served (FCFS) with multi-level feedback queues (MLFQ), which give long jobs shorter turns. Watch which requests wait and how memory fills up.
BAIR
Undergraduate Researcher · SEP — DEC 2024Contributed data-loading and benchmarking software to Robo-DM, an open-source robotics data management toolkit, and co-authored its CoRL 2024 Workshop paper.
Field notes
- Faster loading. Built multiprocessing data-loading pipelines with chunked decoding, cutting dataset load times for ML training.
- Trajectory sampling. Developed frame-level random slicing for efficient trajectory sampling, reducing average data-retrieval latency.
- CoRL workshop paper. Co-authored the Robo-DM paper at the CoRL 2024 Workshop.
Tethys
Scientific experiment · HACKATHON PROJECTI built an experiment that asks an AI to recover known physics equations from data. In the rocket test, removing scientific clues meant about 12 times as many attempts to reach the same accuracy.
Read the results ↗View project ↗Field notes
- Recover a known law. I asked the AI to reconstruct known equations from their data, including the rocket equation, which describes how fuel use changes a rocket’s speed. Knowing the original equations let me check its answers.
- Change only the clues. Both versions saw the same data and had the same mathematical tools. One received labels such as mass and velocity, plus scientific hints; the other saw anonymous names.
- The rocket result. With clues, all three runs recovered the rocket equation; without clues, reaching the same accuracy took 8.7–14.5 times as many candidate programs (11.9 times as a geometric mean). One no-clue run found a close approximation rather than the exact equation.
- Gravity: mixed results. For Newton’s law of gravity, the geometric mean was about 20 times as many candidate programs without clues. The three runs ranged from 1.7 to 100 times, so this result was much less consistent than the rocket test.
- Kepler: no difference. For Kepler’s orbital law, clues made almost no difference: both versions needed about the same number of attempts. The experiments showed that the value of context depends on the problem.