How to start with Apache Airflow in Docker (Windows)

-

 


In general terms, Apache Airflow is an open-source tool that allow us to manage, monitor, plan and schedule workflows that is normally used as a workflow (services) orchestrator.

Prerequisites:

    1. Install Docker Desktop in your computer.

Get Started

Follow this steps in order to start with your testing Airflow environment in docker:

    1. The first step is to download Docker Desktop from the official website. For this article, I installed 4.4.4 version.

    2. After installing Docker Desktop, we need to download a docker-compose.yaml file that you can also find here.

    3. Now that we have both files, we need to create our airflow directory. Go to the following path: C:/Users/<your_user>/. Inside of that directory, create a folder called docker and inside of docker create another folder called airflow.

    4. Now that we have our airflow folder, we must do the following: a) Create three folders called dags, plugins and logs respectively; b) Move our YAML to that directory.

Your directory will look like this

    5. Now we are ready to start our instance of Airflow in docker. We need to open a PowerShell window and go to the directory above.

    6. Then, we need to run the following commands:

    docker-compose up airflow-init

    docker-compose up

    Note: After second command, a window will keep running some code. We can close this window without any problem.

    7. After that, we can go to our Docker Desktop app and we will see that a container named airflow was created inside section Container/Apps. This container will have 7 sub-containers inside.

Your container will look like this

    8. With this, we have our Apache Airflow instance completely ready to start developing our DAGS.

    9. Go to localhost:8080, login with user "airflow" and password "airflow" and start coding.

    Note: You can put all your DAGs in the dags folder and all your plugins in plugins folder both created in step 4.


References

[1] Airflow. Running Airflow in Docker.
https://airflow.apache.org/docs/apache-airflow/2.0.2/start/docker.html


Comments

Post a Comment

Popular posts from this blog

Sending Emails using Apache Airflow Email Operator

-
Image
  Automation reduces manual work and plays a key role in improving productivity in various industries. It is one of the fastest and less time-consuming techniques that most businesses practice to gain higher production rates and improve work efficiency. But, unsure how to perform, some of them fail to automate tasks and end up performing functions manually. Each IT expert has a different job or workflow to perform, right from collecting data from other sources to processing it, uploading, and creating reports. There are many tasks that experts need to perform manually on a daily basis. Thus, to trigger automatic workflow and reduce the time and effort of experts, we recommend using Apache Airflow. Apache Airflow is an open-source tool that assists in managing complex workflows. The powerful workflow management platform helps resolve issues and aids in programmatically authoring, scheduling, and monitoring daily tasks. Data Scientists or Data Engineers often find it helpful for thei...
Read more

Built a working Hadoop-Spark-Hive-Superset cluster on Docker

-
Image
Let’s get this thing started Now you can download my repo  hadoop-spark-hive-superset and from the directory where you placed it all it takes is this command to get the multi-container environment running: And you can break it all down again by going to that same directory and running this: All the containers will then be stopped and removed. But: the images and volumes stay! So don’t be surprised that the csv file you uploaded to HDFS will still be there. Quick starts Quick start HDFS Find the Container ID of the namenode. Copy  supermart_grocery_sales.csv  to the namenode. Go to the bash shell on the namenode with that same Container ID of the namenode. Create a HDFS directory /data//sales/ supermart_grocery_sales . Copy  supermart_grocery_sales.csv  to HDFS: Quick start Spark Go to http://<dockerhadoop_IP_address>:8080 or  http://localhost:8080/  on your Docker host (laptop). Here you find the spark:// master address: Go to the command line o...
Read more

ETL Process Using Airflow and Docker

-
Image
  Introduction In this article, I will create an ETL process for extracting Forex data and wrap the whole thing as a data pipeline using Airflow and Docker. The ETL process will extract data from  fixer.io  API, transform it, and load it to a PostgreSQL database. This project aims to have an automated process that constantly feeds the PostgreSQL database with data. Every 2 minutes, the ETL process will load an updated batch of Forex data. Note that this article assumes some knowledge of Airflow, Docker, Python, and SQL. I won't go into too many details to keep this article short. Project Steps     1. Setting up Airflow Architecture     2. 1st Dag - Check if API is available     3. 2nd Dag - Create a table     4. 3rd Dag - Extract     5. 4th Dag - Transform     6. 5th Dag - Load     7. Query data in the pgAdmin UI Step #1 - Setting up Airflow Architecture The first thing we should do is set up the basic ...
Read more