Sqoop vs Flume vs HDFS in Hadoop

-
Sqoop Flume HDFS
Sqoop is used for importing data from structured data sources such as RDBMS.Flume is used for moving bulk streaming data into HDFS. HDFS is a distributed file system used by Hadoop ecosystem to store data.
Sqoop has a connector based architecture. Connectors know how to connect to the respective data source and fetch the data.Flume has an agent based architecture. Here, code is written (which is called as 'agent') which takes care of fetching data. HDFS has a distributed architecture where data is distributed across multiple data nodes.
HDFS is a destination for data import using Sqoop.Data flows to HDFS through zero or more channels. HDFS is an ultimate destination for data storage.
Sqoop data load is not event driven. Flume data load can be driven by event. HDFS just stores data provided to it by whatsoever means.
In order to import data from structured data sources, one has to use Sqoop only, because its connectors know how to interact with structured data sources and fetch data from them. In order to load streaming data such as tweets generated on Twitter or log files of a web server, Flume should be used. Flume agents are built for fetching streaming data.HDFS has its own built-in shell commands to store data into it. HDFS can not import streaming data

Comments

Post a Comment

Popular posts from this blog

Python vs RUBY vs PHP vs TCL vs PERL vs JAVA

-
Programming Languages Python versus other languages Java   vs. Python Python takes less time to develop although it is expected to run slower than Java programs Due to Python high-level data types and its dynamic typing programs are shorter than Java programs Perl vs. Python Although both these languages are considered similar in many ways,   Perl   supports common application-oriented task like report generating, file scanning, etc.while Python supports for common programming methodologies like data structure design and OOPs programming for programmers making language to readable code Tcl vs. Python Tcl is a standalone programming language it is considered weak on data structures It is slower in executing code compared to Python PHP Python has easy to read code while   PHP   has more syntax from C/C++ and Perl In Python, classes are used extensively in the standard library while PHP has SPL which is fully class based Python su...
Read more

What is Normalization? 1NF, 2NF, 3NF & BCNF

-
Image
What is Normalization? Normalization is a database design technique which organizes tables in a manner that reduces redundancy and dependency of data. It divides larger tables to smaller tables and links them using relationships. In this tutorial, you will learn- Database Normal Forms 1NF Rules What is a KEY? What is Composite Key 2NF Rules Database - Foreign Key What are transitive functional dependencies? 3NF Rules Boyce-Codd Normal Form (BCNF) The inventor of the relational model Edgar Codd proposed the theory of normalization with the introduction of First Normal Form, and he continued to extend theory with Second and Third Normal Form. Later he joined with Raymond F. Boyce to develop the theory of Boyce-Codd Normal Form.  Theory of Data Normalization in SQL is still being developed further. For example, there are discussions even on 6 th  Normal Form.  However, in most practical applications, normalization achieves its best in 3 rd  Normal ...
Read more

Learn Python Main Function with Examples: Understand __main__

-
Image
Before we jump more into Python coding, we get familiarize with Python Main function and its importance. Consider the following code def main() print "hello world!" print "Guru99" Here we got two pieces of print one is defined within a main function that is "Hello World" and the other is independent which is "Guru99". When you run the function def main ():  Only "Guru99" prints out and not the code "Hello World." It is because we did not declare the call  function "if__name__== "__main__". When Python interpreter reads a source file, it will execute all the code found in it. When Python runs the "source file" as the main program, it sets the special variable (__name__) to have a value ("__main__"). When you execute the main function, it will then read the "if" statement and checks whether __name__ does equal to __main__. In Python  "if__name__== "__...
Read more