Real time analytics: Airflow + Kafka + Druid + Superset

-

Real-time analytics has become a necessity for large companies around the world. When your data has been analyzed in a streaming fashion that allows you to continuously analyze customer behavior and act on it. I also want to test Druid's real-time capability, I am looking for a real-time analytics solution. These blogs give an introduction to setting up streaming analytics using open-source technologies. 

Airflow

In this blog use Airflow (a task scheduling platform) that allows you to create, orchestrate and monitor data workflows. In this case, using like a producer sent data to the Kafka topic.

Kafka

Kafka is a distributed messaging platform that allows you to sequentially log streaming data into topic-specific feeds, which other applications can tap into. In this setup, Kafka collects and buffers the events, which are then ingested by Druid.

Druid

Apache Druid provides low-latency real-time data ingestion from Kafka, flexible data exploration, and rapid data aggregation. Druid is not considered a data lake but rather a data river. Since the data is generated by the user, sensor, or whatever, it will work in the foreground application, as with the Hive/Presto setup, data is typically available hourly or daily, but with Druid, the data is available to query upon accessing the database. Druid rates it as a 90%-98% speed improvement over Apache Hive (untested).

Superset

Apache Superset is an Open Source data visualization tool that can represent data graphically. Superset was initially created by AirBnB and later released to the Apache community. Apache Superset is developed in Python language and uses Flask Framework for all web interactions. Superset supports the majority of RDBMS through SQL Alchemy.

How to it works?

Let's set up an example of a real-time coin price analysis system based on Airflow-Kafka-Druid-Superset. Using Docker, it's easy to set up a local instance and make it easier to try and explore your ideas.

    1. To set the system, start by cloning the repo git.

    2. Next, we need to build the local images.

         ∎    Infos of services:

       ∎    Note that the Airflow user is admin and the password will be automatically generated at the a-airflow directory path /app/standalone_admin_password.txt after the server runs. As for the Superset, it is necessary to go to the running container and execute the command unit to create the user with the tag:    


        ∎    Airflow Scheduler app_airflow/app/dags/demo.py is configured to run once a minute executing a message to the Kafka demo topic with list and price coin data ['BTC', 'ETH', 'BTT', 'DOT']. The structure of the message data is as below, I just randomize the price for simplicity. To start streaming, sign in to airflow and enable demo dags.



        ∎    Note: you also can using python to run producer.py like alternative airflow demo dags.


    3. Configure Druid to receive streaming

From Druid Service http://localhost:8888/ select load data > Kafka, enter information Kafka server kafka:9092 and topic demo and config output.

    4. Setup Superset connect to Druid like data source

Login into Superset http://localhost:8088/ create new database Data > Databases > + Database connection on Druid using sqlalchemy uri: druid://broker:8082/druid/v2/sql/ for detail can read at Superset-Database-Connect

    5. Create dashboard

To create dashboards with superset. Go to SQL Lab > SQL Editor select the database as druid, schema druid, table demo, and execute the query you need. Once done, click on Explore select your chart, and publicizes it as a dashboard.


Finally enjoy it! 💥💥

Comments

Post a Comment

Popular posts from this blog

Sending Emails using Apache Airflow Email Operator

-
Image
  Automation reduces manual work and plays a key role in improving productivity in various industries. It is one of the fastest and less time-consuming techniques that most businesses practice to gain higher production rates and improve work efficiency. But, unsure how to perform, some of them fail to automate tasks and end up performing functions manually. Each IT expert has a different job or workflow to perform, right from collecting data from other sources to processing it, uploading, and creating reports. There are many tasks that experts need to perform manually on a daily basis. Thus, to trigger automatic workflow and reduce the time and effort of experts, we recommend using Apache Airflow. Apache Airflow is an open-source tool that assists in managing complex workflows. The powerful workflow management platform helps resolve issues and aids in programmatically authoring, scheduling, and monitoring daily tasks. Data Scientists or Data Engineers often find it helpful for thei...
Read more

Built a working Hadoop-Spark-Hive-Superset cluster on Docker

-
Image
Let’s get this thing started Now you can download my repo  hadoop-spark-hive-superset and from the directory where you placed it all it takes is this command to get the multi-container environment running: And you can break it all down again by going to that same directory and running this: All the containers will then be stopped and removed. But: the images and volumes stay! So don’t be surprised that the csv file you uploaded to HDFS will still be there. Quick starts Quick start HDFS Find the Container ID of the namenode. Copy  supermart_grocery_sales.csv  to the namenode. Go to the bash shell on the namenode with that same Container ID of the namenode. Create a HDFS directory /data//sales/ supermart_grocery_sales . Copy  supermart_grocery_sales.csv  to HDFS: Quick start Spark Go to http://<dockerhadoop_IP_address>:8080 or  http://localhost:8080/  on your Docker host (laptop). Here you find the spark:// master address: Go to the command line o...
Read more

ETL Process Using Airflow and Docker

-
Image
  Introduction In this article, I will create an ETL process for extracting Forex data and wrap the whole thing as a data pipeline using Airflow and Docker. The ETL process will extract data from  fixer.io  API, transform it, and load it to a PostgreSQL database. This project aims to have an automated process that constantly feeds the PostgreSQL database with data. Every 2 minutes, the ETL process will load an updated batch of Forex data. Note that this article assumes some knowledge of Airflow, Docker, Python, and SQL. I won't go into too many details to keep this article short. Project Steps     1. Setting up Airflow Architecture     2. 1st Dag - Check if API is available     3. 2nd Dag - Create a table     4. 3rd Dag - Extract     5. 4th Dag - Transform     6. 5th Dag - Load     7. Query data in the pgAdmin UI Step #1 - Setting up Airflow Architecture The first thing we should do is set up the basic ...
Read more