Working with Spark – Big Data Hadoop MapReduce

Introduction


Before moving to Spark RDD concept, which is the base line of Spark, we need to understand the concept of Hadoop Mapredue. RDD is the improvement of Hadoop Mapreduce to get 100 outputs in memory. In this blog post we are going to discuss about Hadoop Mapredue to understand the concept only.

Hope it will be interesting.

 

What is MapReduce


MapReduce is a software framework and programming module to handle huge data. Hadoop is capable to run MapReduce programs which are written in different language like Java, Ruby, Python and C++.


How MapReduce Works


As the name specified, MapReduce is a combination of Mapping and Reducing.

 

We can divide the MapReduc into following Sections.


·         Input Splits

·         Mapping

·         Shuffling

·         Reducing

 

Before examining those, let’s take an example.

 

Please consider the following input data as Map Reduce program.

 

Mayuree how are you

We welcome Mayuree in India

Mayaure is a best developer

 

Please follow the bellow diagram.

 


 

 

Final Output:

 



 

Input Splits

An input to a MapReduce job is divided into fixed-size pieces called input splits Input split is a chunk of the input that is consumed by a single map

Mapping

This is the very first phase in the execution of map-reduce program. In this phase data in each split is passed to a mapping function to produce output values.

Shuffling

This phase consumes the output of Mapping phase. Its task is to consolidate the relevant records from Mapping phase output.

Reducing

In this phase, output values from the Shuffling phase are aggregated. This phase combines values from Shuffling phase and returns a single output value. In short, this phase summarizes the complete datasets.

 

 

 

 

 

Hope you like it.

 


Comments

Post a Comment

Popular Posts

Working with Python -- File Handling

Image
Introduction File Handling is a very important portion for any application. In Python, we have several function to Create, Read, Update and Delete file. In this blog post, we are trying to discuss about Python File handling. Hope it will be interesting. Open Function This is the key function to working with file. The Open function takes two parameters. One is File name and second is mode. "r"   -  Read   - Default value. Opens a file for reading, error if the file does not exist "a"  -  Append  - Opens a file for appending, creates the file if it does not exist "w"  -  Write  - Opens a file for writing, creates the file if it does not exist "x"  -  Create  - Creates the specified file, returns an error if the file exists In addition of that, we can specify that the file can be treated as Binary or as Text file. "t"  -  Text  - Default value. Text mode "b"  -  Binary  - Binary mode (e.g. images) Exa...
Read more

Incremental Load by using ADF

Image
Introduction lot of my friend from different group is asking to provide a blog post related to incremental load. If you search Google related to this tropics, you can find bundle of blog post. All are the same using water mark technique. But in my view point it is not a practical approach for incremental load. When we talk about incremental load, we assume that we have billions of records in source which is ever changing and we need to move on destination table. Here in this demonstration we used ADF control flow but it can be done with any ETL or ELT tools like SSIS. One last thing that I would like to mention if we try to load huge records from source to designation, it definitely gives us performance problem or stuck in between. So, we have to load the data in a smaller chunk or batch. We have to keep in track that data that we already loaded in the destination is going to updated in source or not. If it is going too updated, we have to update the destination accordingly. Yo...
Read more

Copying Multiple File from Blob Storage to Single SQL Table – Part-2

Image
Introduction This is the extension of my proves blog post. If you not go with previous, I strongly recommended you to read my previous blog post named “Copying Multiple File from Blob Storage to Single SQL Table”. Case Scenario Here in this project, we are going to retrieve the CSV file from Azure Blob Storage, but we must retrieve the current date file only and store it into a Azure SQL DB Table. For SSIS developer it is work of using Expression, here I am also using the expression but in different ways that supports by Azure ADF. Source Files in Azure Blob Storage We have 3 file in our Azure Blob Storage. But only one file have Modified Date 30/5/2020. So, we need to retrieve that file only which matches the current system date.   ADF Solution   Step-1 We are using Get MetaData   Activity which retrieve all file information from Blob Storage and send it to the ForEach Look container Activity. Step-2 Within the ForEach Loop Activity we used anothe...
Read more