Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

airflow

airflow• Pinned

airflow: A platform to programmatically author, schedule, and monitor workflows.

What problem it solves

Teams need a maintainable way to define workflows as code, schedule dependent tasks, and monitor running pipelines. Airflow addresses this with code-defined Dags, task execution across workers, command-line tools, and a user interface for visualizing progress and troubleshooting issues.

Key features

  • Code-defined workflows — Define pipelines as Python code with dynamic Dag generation and parameterization. Workflows can be versioned, tested, and collaboratively maintained.

    Source: README

  • Dependency-aware task execution — Author workflows as Dags that orchestrate tasks and execute them on workers according to specified dependencies. Tasks can pass metadata through XCom while data-intensive work can be delegated to external services.

    Source: README

  • Extensible and templated — Extend the framework with built-in operators and customizations supported by Jinja templating. This supports workflows tailored to different systems and processing needs.

    Source: README

  • Multiple workflow domains — Orchestrate data processing, machine learning, and agentic or LLM-based workloads. Airflow can coordinate stages such as data preparation, tool calls, model invocation, and evaluation.

    Source: README

Quick start

Copied verbatim from the project's README.

pip install 'apache-airflow==3.4.0' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.4.0/constraints-3.10.txt"

Who it may suit

This may suit developers and system administrators who need to define, schedule, and monitor workflows in Python.

Topics to discuss

  • How should Airflow workflows be structured for static and slowly changing pipelines?

  • Which workflow patterns should the community improve for machine learning and LLM-based workloads?

  • What guidance should be added for installing Airflow with constraint files and optional providers?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/apache/airflow#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/apache/airflow

Comments

…