In this highly visual guide, we explore the architecture of a Mixture of Experts in Large Language Models (LLM) and Vision Language Models. Timeline 0:00 Introduction 0:34 A Simplified Perspective 2:14 The Architecture of Experts 3:05 The Router 4:08 Dense vs. Sparse Layers 4:33 Going through a MoE Layer 5:35 Load Balancing 6:05 KeepTopK 7:27 Token Choice and Top-K Routing 7:48 Auxiliary Loss 9:23 Expert Capacity 10:40 Counting Parameters with Mixtral 7x8B 13:42 MoE in Vision Language Models 13:57 Vision Transformer 14:45 Vision-MoE 15:50 Soft-MoE 19:11 Bonus Content! 🛠️ Written version of this visual guide Support to my newsletter for more visual guides: ✉️ Newsletter I wrote a book! 📚 Hands-On Large Language Models #datascience #machinelearning #ai