Data Engineering Project — Cửa hàng bán lẻ phần 4 — Analyzing the Data

-

 



Comments

Post a Comment

Popular posts from this blog

How to automate web scraping using Selenium in Python: IMDB case

-
Image
There are some ways to scrape websites in python. Just mention famous libraries such as BeautifulSoup or Scrapy that have been widely utilized for crawling the web. But, in this article, I will share my experience using Selenium library for web-scraping. Selenium library work to automate the manual activity of the user interacting with the website. Imagine you wanted to get some information on some website, what would you do? Of course, you'll initiate with opening the browser, put the address of the website on the browser, go to the site, take a look at the information and then finally get the information. Selenium does the same step as the manual process, but it is done automatically. Let's get started to see how it works.                                                                   ...
Read more

ETL Process Using Airflow and Docker

-
Image
  Introduction In this article, I will create an ETL process for extracting Forex data and wrap the whole thing as a data pipeline using Airflow and Docker. The ETL process will extract data from  fixer.io  API, transform it, and load it to a PostgreSQL database. This project aims to have an automated process that constantly feeds the PostgreSQL database with data. Every 2 minutes, the ETL process will load an updated batch of Forex data. Note that this article assumes some knowledge of Airflow, Docker, Python, and SQL. I won't go into too many details to keep this article short. Project Steps     1. Setting up Airflow Architecture     2. 1st Dag - Check if API is available     3. 2nd Dag - Create a table     4. 3rd Dag - Extract     5. 4th Dag - Transform     6. 5th Dag - Load     7. Query data in the pgAdmin UI Step #1 - Setting up Airflow Architecture The first thing we should do is set up the basic ...
Read more

Sending Emails using Apache Airflow Email Operator

-
Image
  Automation reduces manual work and plays a key role in improving productivity in various industries. It is one of the fastest and less time-consuming techniques that most businesses practice to gain higher production rates and improve work efficiency. But, unsure how to perform, some of them fail to automate tasks and end up performing functions manually. Each IT expert has a different job or workflow to perform, right from collecting data from other sources to processing it, uploading, and creating reports. There are many tasks that experts need to perform manually on a daily basis. Thus, to trigger automatic workflow and reduce the time and effort of experts, we recommend using Apache Airflow. Apache Airflow is an open-source tool that assists in managing complex workflows. The powerful workflow management platform helps resolve issues and aids in programmatically authoring, scheduling, and monitoring daily tasks. Data Scientists or Data Engineers often find it helpful for thei...
Read more