Sqoop vs Flume vs HDFS in Hadoop

-
Sqoop Flume HDFS
Sqoop is used for importing data from structured data sources such as RDBMS.Flume is used for moving bulk streaming data into HDFS. HDFS is a distributed file system used by Hadoop ecosystem to store data.
Sqoop has a connector based architecture. Connectors know how to connect to the respective data source and fetch the data.Flume has an agent based architecture. Here, code is written (which is called as 'agent') which takes care of fetching data. HDFS has a distributed architecture where data is distributed across multiple data nodes.
HDFS is a destination for data import using Sqoop.Data flows to HDFS through zero or more channels. HDFS is an ultimate destination for data storage.
Sqoop data load is not event driven. Flume data load can be driven by event. HDFS just stores data provided to it by whatsoever means.
In order to import data from structured data sources, one has to use Sqoop only, because its connectors know how to interact with structured data sources and fetch data from them. In order to load streaming data such as tweets generated on Twitter or log files of a web server, Flume should be used. Flume agents are built for fetching streaming data.HDFS has its own built-in shell commands to store data into it. HDFS can not import streaming data

Comments

Post a Comment

Popular posts from this blog

Introduction to BIG DATA: Types, Characteristics & Benefits

-
Image
In order to understand 'Big Data', we first need to know what  'data'  is. Oxford dictionary defines  'data'  as - "The quantities, characters, or symbols on which operations are performed by a computer, which may be stored and transmitted in the form of electrical signals and recorded on magnetic, optical, or mechanical recording media. " In this tutorial we will learn, Examples Of 'Big Data' Categories Of 'Big Data' Characteristics Of 'Big Data' Advantages Of Big Data Processing So, 'Big Data' is also a  data  but with a  huge size . 'Big Data' is a term used to describe collection of data that is huge in size and yet growing exponentially with time.In short,  s uch a data is so large and complex that none of the traditional data management tools are able to store it or process it efficiently. Examples Of 'Big Data' Following are some the examples of 'Big Data'- ...
Read more

Learn Python Main Function with Examples: Understand __main__

-
Image
Before we jump more into Python coding, we get familiarize with Python Main function and its importance. Consider the following code def main() print "hello world!" print "Guru99" Here we got two pieces of print one is defined within a main function that is "Hello World" and the other is independent which is "Guru99". When you run the function def main ():  Only "Guru99" prints out and not the code "Hello World." It is because we did not declare the call  function "if__name__== "__main__". When Python interpreter reads a source file, it will execute all the code found in it. When Python runs the "source file" as the main program, it sets the special variable (__name__) to have a value ("__main__"). When you execute the main function, it will then read the "if" statement and checks whether __name__ does equal to __main__. In Python  "if__name__== "__...
Read more

Hadoop Tutorial: Features, Components, Cluster & Topology

-
Image
Apache HADOOP is a framework used to develop data processing applications which are executed in a distributed computing environment. In this tutorial we will learn, Components of Hadoop Features Of 'Hadoop' Network Topology In Hadoop Similar to data residing in a local file system of personal computer system, in Hadoop, data resides in a distributed file system which is called as a  Hadoop Distributed File system . Processing model is based on  'Data Locality'  concept wherein computational logic is sent to cluster nodes(server) containing data. This computational logic is nothing but a compiled version of a program written in a high level language such as Java. Such a program, processes data stored in Hadoop HDFS. HADOOP is an open source software framework. Applications built using HADOOP are run on large data sets distributed across clusters of commodity computers. Commodity computers are cheap and widely available. These are mainly useful for ...
Read more