Twitter Data Analysis using Hadoop

Computer Science &Engineering

In today‘s highly developed world, every minute, people around the globe express themselves via various platforms on the Web. And in each minute, a huge amount of unstructured data is generated. Such data is termed as big data. Twitter, one of the largest social media site receives millions of tweets every day on variety of important issues. This huge amount of raw data can be used for industrial, social, economic, government policies or business purpose by organizing according to our requirement and processing. Hadoop is one of the best tool options for twitter data analysis as it works for distributed big data, streaming data, time stamped data, text data etc. Hence, Flume is used to extract real time twitter data into HDFS. Hive and Pig which is SQL like query language is used for some extraction and analysis. The goal of this project is to compare the results of Hive and Pig and analyse the retrieval time for a query. Finally we will conclude which framework will work fast and scalable for our dataset.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.