Jessica laughlin this is the clearest thinking around the usage of airflow i’ve read in a long time…
- Select a language for the TTS:
- UK English Female
- UK English Male
- US English Female
- US English Male
- Australian Female
- Australian Male
- Language selected: (auto detect) - EN
Play all audios:
Jessica Laughlin this is the clearest thinking around the usage of Airflow I’ve read in a long time and the approach makes so much sense. The decision of Airflow to package a huge number of
built-in operators along withs its core installation seems very convenient at first, but causes a *huge* swath of problems. I’ve been working on a project called Dagster
(https://github.com/dagster-io/dagster) which has a nascent Airflow integration which very much aligns with your approach. Effectively you encode your business logic, data dependencies, etc
in a higher level abstraction and then compile/create your Airflow programmatically. These Airflow DAGs are comprised of a single operator type, and it could be a k8s one like you describe.
Would love to chat and get your feedback if you are interested!