Sqoop vs Flume vs HDFS in Hadoop

-
Sqoop Flume HDFS
Sqoop is used for importing data from structured data sources such as RDBMS.Flume is used for moving bulk streaming data into HDFS. HDFS is a distributed file system used by Hadoop ecosystem to store data.
Sqoop has a connector based architecture. Connectors know how to connect to the respective data source and fetch the data.Flume has an agent based architecture. Here, code is written (which is called as 'agent') which takes care of fetching data. HDFS has a distributed architecture where data is distributed across multiple data nodes.
HDFS is a destination for data import using Sqoop.Data flows to HDFS through zero or more channels. HDFS is an ultimate destination for data storage.
Sqoop data load is not event driven. Flume data load can be driven by event. HDFS just stores data provided to it by whatsoever means.
In order to import data from structured data sources, one has to use Sqoop only, because its connectors know how to interact with structured data sources and fetch data from them. In order to load streaming data such as tweets generated on Twitter or log files of a web server, Flume should be used. Flume agents are built for fetching streaming data.HDFS has its own built-in shell commands to store data into it. HDFS can not import streaming data

Comments

Post a Comment

Popular posts from this blog

Learn Python For & While Loop with Examples

-
Image
Loops can execute a block of code number of times until a certain condition is met. Their usage is fairly common in programming. Unlike other programming language that have For Loop, whileloop, dowhile, etc. Python has only two loop functions 1) While Loop 2) For Loop   In this tutorial, we will learn How to use "While Loop" How to use "For Loop" How to use For Loop for set of other things besides numbers Break statements in For Loop Continue statement in For Loop Enumerate function for For Loop Pratical Example How to use for loop to repeat the same statement over and again How to use "While Loop" Whileloop does the exactly same thing what "if statement" does, but instead of running the code block once, they jump back to the point where it began the code and repeats the whole process again. Syntax While expression: Statement: Example : Code Line 4: Variable x is set to 0 Code Line 7: Whi...
Read more

Variables in Python

-
Image
What is Variable? A variable is nothing but a reserved memory location to store values. In other words a variable in a program gives data to the computer to work on. Every value in Python has a datatype. Different data types in Python are Numbers, List, Tuple, Strings, Dictionary, etc. Variables can be declared by any name or even alphabets like a, aa, abc etc. In this tutorial we will learn, How to declare and use a variable Re-declare a Variable Concatenate Variables Local & Global Variables Delete a variable How to declare and use a variable Let see an example. We will declare variable "a" and print it. a=100 print a The output of variable should be "100" Re-declare a Variable You can re-declare the variable even after you have declared it once. Here we have variable initialized to f=0. Later, we re-assign the variable f to value "guru99" Concatenate Variables Let's see whether you can concatenate di...
Read more

Learn Python Main Function with Examples: Understand __main__

-
Image
Before we jump more into Python coding, we get familiarize with Python Main function and its importance. Consider the following code def main() print "hello world!" print "Guru99" Here we got two pieces of print one is defined within a main function that is "Hello World" and the other is independent which is "Guru99". When you run the function def main ():  Only "Guru99" prints out and not the code "Hello World." It is because we did not declare the call  function "if__name__== "__main__". When Python interpreter reads a source file, it will execute all the code found in it. When Python runs the "source file" as the main program, it sets the special variable (__name__) to have a value ("__main__"). When you execute the main function, it will then read the "if" statement and checks whether __name__ does equal to __main__. In Python  "if__name__== "__...
Read more