The Astronomer Blog: Data Orchestration Insights & Guides

December 10, 2024

Airflow in Action: DataOps Insights from 2,200 Pipelines at Instacart

Discover how Instacart scaled Apache Airflow to handle 2,200 pipelines and 16 million tasks monthly with custom tools, IaC, and centralized management.

December 5, 2024

Airflow in Action: ML and Data Engineering Insights from the Analytics-Obsessed World of MLB

Explore how the Texas Rangers and Philadelphia Phillies use Airflow to revolutionize data orchestration and ML, driving faster insights and a competitive edge.

December 4, 2024

Data Observability 101: An Introduction to the Most Critical Features of Modern Data Observability

Explore core data observability features: analytics, monitoring, and alerts. Gain insights into pipeline health, prevent failures, and ensure data reliability.

December 3, 2024

Airflow in Action: How Panasonic Energy is Accelerating the EV Transition With Data Engineering

Learn how Panasonic leverages Apache Airflow to streamline battery production, scale workflows, and plan for streaming and Kubernetes.

December 2, 2024

ELT for Beginners: Extract from S3, Load to Databricks and Run Transformations

Learn how to build an ELT pipeline extracting data from S3, loading it into Databricks, and transforming it with notebooks using Apache Airflow.

November 27, 2024

Airflow in Action: ML and LLM Ops Insights from ASAPP — Reducing Workflow Runtimes by 85%

Discover how ASAPP turbocharges MLOps with Airflow and Spark, scaling LLM workflows to boost efficiency & achieve lightning-fast processing in hours vs days.

November 26, 2024

Airflow in Action: Infrastructure Management Insights from 1 Million Monthly Deploys at LinkedIn

Discover how LinkedIn uses Airflow to orchestrate 12k pipelines managing 1m deploys for 7k services. And Airflow 3.0 will make provisioning even easier!

November 21, 2024

Airflow in Action: Data Engineering Insights from Processing PBs of Data Every Day at Stripe

Learn how Stripe processes petabytes of data daily with Apache Airflow, ensuring compliance while accelerating developer workflows with its dev/test tooling

November 19, 2024

Airflow in Action: Scaling Insights from Bosch and 1.2 Million Pipeline Runs Per Day

Go on an Airflow journey with Bosch as they scale from 1k to 50k DAG runs per hour and 1.2 million pipeline runs per day with an average 1 second latency.

Astronomer Blog

Build, run, & observe your data workflows. All in one place.

Build, run, & observe
your data workflows.
All in one place.