Built a working Hadoop-Spark-Hive-Superset cluster on Docker

-

Let’s get this thing started

Now you can download my repo hadoop-spark-hive-superset and from the directory where you placed it all it takes is this command to get the multi-container environment running:

And you can break it all down again by going to that same directory and running this:

All the containers will then be stopped and removed. But: the images and volumes stay! So don’t be surprised that the csv file you uploaded to HDFS will still be there.

Quick starts

Quick start HDFS

Find the Container ID of the namenode.


Copy supermart_grocery_sales.csv to the namenode.


Go to the bash shell on the namenode with that same Container ID of the namenode.


Create a HDFS directory /data//sales/supermart_grocery_sales.


Copy supermart_grocery_sales.csv to HDFS:


Quick start Spark

Go to http://<dockerhadoop_IP_address>:8080 or http://localhost:8080/ on your Docker host (laptop). Here you find the spark:// master address:


Go to the command line of the Spark master and start spark-shell.


Load supermart_grocery_sales.csv from HDFS.


Quick start Hive

Find the Container ID of the Hive Server.


Go to the command line of the Hive server and start hiveserver2

Maybe a little check that something is listening on port 10000 now

Okay. Beeline is the command line interface with Hive. Let’s connect to hiveserver2 now.

Show databases

Create new database

And let's create a table

And have a little select statement going.







Comments

Post a Comment

Popular posts from this blog

Sending Emails using Apache Airflow Email Operator

-
Image
  Automation reduces manual work and plays a key role in improving productivity in various industries. It is one of the fastest and less time-consuming techniques that most businesses practice to gain higher production rates and improve work efficiency. But, unsure how to perform, some of them fail to automate tasks and end up performing functions manually. Each IT expert has a different job or workflow to perform, right from collecting data from other sources to processing it, uploading, and creating reports. There are many tasks that experts need to perform manually on a daily basis. Thus, to trigger automatic workflow and reduce the time and effort of experts, we recommend using Apache Airflow. Apache Airflow is an open-source tool that assists in managing complex workflows. The powerful workflow management platform helps resolve issues and aids in programmatically authoring, scheduling, and monitoring daily tasks. Data Scientists or Data Engineers often find it helpful for thei...
Read more

ETL Process Using Airflow and Docker

-
Image
  Introduction In this article, I will create an ETL process for extracting Forex data and wrap the whole thing as a data pipeline using Airflow and Docker. The ETL process will extract data from  fixer.io  API, transform it, and load it to a PostgreSQL database. This project aims to have an automated process that constantly feeds the PostgreSQL database with data. Every 2 minutes, the ETL process will load an updated batch of Forex data. Note that this article assumes some knowledge of Airflow, Docker, Python, and SQL. I won't go into too many details to keep this article short. Project Steps     1. Setting up Airflow Architecture     2. 1st Dag - Check if API is available     3. 2nd Dag - Create a table     4. 3rd Dag - Extract     5. 4th Dag - Transform     6. 5th Dag - Load     7. Query data in the pgAdmin UI Step #1 - Setting up Airflow Architecture The first thing we should do is set up the basic ...
Read more