K = KubernetesPodOperator(namespace='default',

Volume = Volume(name='test-volume', configs=volume_config) Volume_mount = VolumeMount('test-volume',Ĭonfigmaps = Secret_all_keys = Secret('env', None, 'airflow-secrets-2') Secret_env = Secret('env', 'SQL_CONN', 'airflow-secrets', 'sql_alchemy_conn') Since the full pod k8s API is supported we can supply anything our pod might need as arguments to the Operator: secret_file = Secret('volume', '/etc/sql_conn', 'airflow-secrets', 'sql_alchemy_conn') We are also specifying to look for the in_cluster authentication configuration (which uses our service account token) and to keep completed pods with is_delete_operator_pod. The KubernetesPodOperator has some required parameters like image, namespace, cmds, name, and task_id but the full Kubernetes pod API is supported. Here we are using the KubernetesPodOperator to run a container as a pod from our DAG. When these two prerequisites are met we can start running containerized workloads on Kubernetes from our DAG’s using the KubernetesPodOperator.Īs an example take this dag below: import loggingįrom _operator import DummyOperatorįrom _pod_operator import KubernetesPodOperator We also need a service account and service account token available in our cluster to ensure the Operator can authenticate and is allowed to run Pods on the namespace we provide. The init container is responsible for bootstrapping the database.Ī simple PostgreSQL database setup is required, with internal service to enable airflow to connect to it: I stripped out some of the configuration here like livenessProbe and readinessProbe to make it a bit more verbose but the full code sample can be found in the repository.Īirflow requires at least a web server and scheduler component. Image: eu.gcr.io/fullstaq-st-tim/st-airflow:latest For this, we are using a simple deployment consisting of the Airflow webserver, scheduler/executor, and a separate PostgreSQL database deployment for the Airflow metadata DB. We want to ensure that we have Airflow running on our cluster. This is by far the easiest way to get started running container workloads from Airflow on Kubernetes. It is perfect for when you want to use/re-use some existing containers in your ecosystem but want to schedule them from airflow and incorporate them into your workflow. The KubernetesPodOperator enables you to run containerized workloads as pods on Kubernetes from your DAG. From ones that can run your own python code to MySQL, azure, spark, cloud storage or serverless operators. There are many different operators available. This DAG then gets scheduled by the Airflow scheduler and executed by the Executor.

In your DAG you specify your pipeline steps as tasks using operators and define their flow (upstream and downstream dependecies). The KubernetesPodOperator is an airflow builtin operator that you can use as a building block within your DAG’s.Ī DAG stands for Acyclic Directed Graph and is basically your pipeline defitinion / workflow written in pure python. Some prior knowledge of Airflow and Kubernetes is required. If you’d rather watch the accompanying video to this post, I did a Show ‘n Tell webinar on this topic previously which you can rewatch here. KEDA enables elastic from and to zero scaling of Airflow CeleryWorkers on KubernetesĬode samples and configuration can be found here. The KubernetesExecutor natively runs any task in your DAG as a pod on Kubernetes. The KubernetesPodOperator enables you to run containerized workloads from within your DAG’s. Horizontally scalable dynamic data pipelines, who doesn’t want that? If you want to get started running Airflow on Kubernetes, containerizing your workloads, and using most out of both platforms then this post will show you how to do that in three different ways. We love leveraging the power of Airflow with Kubernetes.

Airflow kubernetes secret