Join my group for people learning Azure :) 👉 Get the resources (Github): Let's make an End-to-End Data Pipeline in Databricks. -------------------- 📊 Learn Data & AI (50% OFF): 🤖 Early access to the Applied AI Community: 📮 Join the Newsletter: 🔗 Follow me on Linkedin: -------------------- 🏅 RECOMMENDED COURSES: (All under one subscription) *Top AI Courses:* AI Engineer for Developers Course - AI Engineer for Data Scientists Course - Developing AI Applications Course - *Top Data Courses:* Data Engineer in Python - Associate Data Scientist in Python - Data Analyst with Python - Associate Data Analyst in SQL - -------------------- 0:00 - Introduction 3:47 - What is Databricks and its Architecture 5:33 - Medallion Architecture Explained 10:47 - Setting up Databricks in Azure 19:04 - Access Control and Security 29:04 - Connecting Databricks to Storage 36:07 - Introduction to Databricks Notebooks 44:24 - Setting up Compute Resources 45:54 - Configuring Compute Resources 48:02 - Installing Required Libraries 50:34 - Setting Up Notebooks 53:05 - Bronze Notebook: Fetching Raw Data 1:00:00 - Verifying Data in Bronze Layer 1:02:14 - Silver Notebook: Data Transformation 1:10:09 - Gold Notebook: Advanced Transformations 1:18:01 - Writing to Gold Layer 1:23:19 - Creating a Workflow in Databricks 1:27:25 - Configuring Task Dependencies 1:30:34 - Testing the Workflow 1:33:27 - Conclusion and Next Steps -------------------- Business inquiries: hello@