🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

RIP to my friend, colleague, and AI visionary Nils Nilsson. Your work on the A* algorithm has improved countless lives (this is how we find the shortest path from A to B). I will always remember your work, but even more importantly your kindness. ai.stanford.edu/~nilsson/

Thanks!

Our paper on model parallel framework #EasyParallellLibrary (#EPL) was accepted by #ATC’22! Many thanks to the reviewer and co-authors. Stay tuned for exciting updates! GitHub: github.com/alibaba/EasyPa…

Saddened that due to visa disapproval, I won’t be able to attend the #ATC conference to present our work. That said, I am very thankful to USENIX Association and Noa Zilberman for helping me on this matter. Hope to see you next year!

Very insightful!

Great explanation! I also had the same question regarding tensor core. btw, TPU also uses systolic array, but some argue that systolic array is not ideal due to the lack of flexibility. Maybe the future belongs to RISC-V arch😄

Some notes that I took while attending the ACM Summer School on HPC at Barcelona: zijishi.xyz/post/optimizat…

Reminder to enroll DLSysCourse to build your own deep learning framework from scratch!

Just received my travel grant for the HPC summer school from Association for Computing Machinery. Much appreciated! The notes can be found here: zijishi.xyz/post/optimizat…

Eagerly expecting a break after the ASPLOS deadline...

TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation. Ziji Shi, Le Jiang, Ang Wang, Jie Zhang, Xianyan Jia, Yong Li, Chencan Wu, Jialin Li, and Wei Lin arxiv.org/abs/2302.00247

#ChatGPT has been phenomenal, but have you ever wondered how it was trained? In fact, finding the optimal parallel strategy for such LLM is very challenging, as the candidate space grows exponentially w.r.t size. (1/2)

We recently uploaded our work with Alibaba on *quickly* and *automatically* finding the optimal #tensorparallel strategy for #LLM. Compared to SoTA approaches, we are ~20-160x faster. Comments are welcomed! Arxiv: arxiv.org/abs/2302.00247

🧵🧠 We're witnessing incredible scientific progress in image & text reconstruction from fMRI nowadays. But what about reconstructing video from fMRI? Allow me to introduce our recent preprint: Mind-Video arxiv.org/abs/2305.11675 mind-video.com drive.google.com/drive/folders/…

(1/2) As large-scale models continue to evolve, the need for associated foundational systems is also growing. We've set up an MLSys discussion group (mlsys-sg.org), planning to host bi-weekly discussions on academic papers or updates on cutting-edge advancements.

(2/2) We warmly welcome all professionals in the field to join us, engage in enriching conversations, and contribute to our vibrant community! #LLM #Singapore #MLSys #AI #CommunityBuilding 🤝

I’m attending #ISCA co-located with #FCRC 🎉 We will present two papers at the MlArchSys and ASSYST workshop on #LLM and #GAN at Canary 2. Feel free to drop by and say hi! ISCA

1/ Announcing the development of OpenMoE project! 🚀 Open Mixture-of-Experts Language Models! MoE + UL2 objective + umT5 tokenizer + 50% code data mix. GitHub: github.com/XueFuzhao/Open… Blog: xuefuzhao.notion.site/Aug-2023-OpenM…

ParaGAN: A Scalable Distributed Training Framework for Generative Adversarial Networks by Ziji Shi et al. arxiv.org/abs/2411.03999…

Introducing Domino: a novel zero-cost communication tensor parallelism (TP) training engine for both single node and multi-node settings. - Near-complete communication hiding - Novel multi-node scalable TP solution Blog: github.com/microsoft/Deep…