Progress can be tracked in Jira ( AIRFLOW-1314).ĭevelopment is being done in a fork of Airflow at bloomberg/airflow. The wiki contains a discussion about what this will look like, though the pages haven't been updated in a while. Work is in progress that should lead to native support by Airflow for scheduling jobs on Kubernetes. The Helm chart mentioned below does this. However, you can also deploy your Celery workers on Kubernetes. The simplest way to achieve this right now, is by using the kubectl commandline utility (in a BashOperator) or the python sdk. The reason that I make this distinction is that you typically need to perform some different steps for each scenario. And of course you can run them in Kubernetes and deploy to Kubernetes as well. Or you can host them on Kubernetes, but deploy somewhere else, like on a VM. you can use Jenkins or Gitlab (buildservers) on a VM, but use them to deploy on Kubernetes. You can actually replace Airflow with X, and you will see this pattern all the time.
Using Airflow to schedule jobs on Kubernetes.There are some related, but different scenarios:
Please do not hesitate to provide updates, suggestions, fixes etc. Because things move quickly, I've decided to put this on Github rather than in a blog post, so it can be easily updated. Here I write down what I've found, in the hope that it is helpful to others. Also, there are many forks and abandoned scripts and repositories. While there are reports of people using them together, I could not find any comprehensive guide or tutorial. Recently I spend quite some time diving into Airflow and Kubernetes.
0 kommentar(er)
