Working with Spark – Spark RDD

Introduction

In my previous blog post, we are discussing about Hadoop Map Reduce. Now in this blog post we are going to discuss about Spark Fundamentals concept like RDD.

RDD stands for Resilient Distributed Dateset, these are the elements that run and operate on multiple nodes to do parallel processing on a cluster. 

Hope it will be interesting.

What is RDD


According to Apache Spark documentation - 

"Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. There are two ways to create RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat".

RDDs are immutable elements, which means once we create an RDD we cannot change it. RDDs are fault tolerant as well, hence in case of any failure, they recover automatically. We can apply multiple operations on these RDDs to achieve a certain task.

We can apply operation in RDD in two ways.

1.       Transformation

2.       Action

 

Transformation


These are the operations, which are applied on a RDD to create a new RDD. Filter, groupBy and map are the examples of transformations.

Spark Transformation is a function that produces new RDD from the existing RDDs. It takes RDD as input and produces one or more RDD as output. Each time it creates new RDD when we apply any transformation. Thus, the so input RDDs, cannot be changed since RDD are immutable in nature.


Applying transformation built an RDD lineage, with the entire parent RDDs of the final RDD(s). RDD lineage, also known as RDD operator graph or RDD dependency graph. It is a logical execution plan i.e., it is Directed Acyclic Graph (DAG) of the entire parent RDDs of RDD.

 

Transformations are lazy in nature i.e., they get execute when we call an action. They are not executed immediately.


After the transformation, the resultant RDD is always different from its parent RDD. It can be smaller (filter, count, distinct, sample),  bigger (flatMap(), union(), Cartesian()) or the same size (map)

 

There are two type of Transformation

Narrow Transformation


In Narrow transformation, all the elements that are required to compute the records in single partition live in the single partition of parent RDD. A limited subset of partition is used to calculate the result. Narrow transformations are the result of map(), filter().

 

 


Wide transformation


In wide transformation, all the elements that are required to compute the records in the single partition may live in many partitions of parent RDD. The partition may live in many partitions of parent RDD. Wide transformations are the result of groupbyKey() and reducebyKey().

 

 

 

Summarized the Basic facts about Spark RDD


  • Resilient Distributed Datasets (RDDs) are basically an immutable collection of elements which is used as fundamental data structure in Apache Spark.
  • We can create RDDs by two methods - Parallelize collection & referencing external datasets.
  • RDDs are immutable i.e. read only data structures so you can't change original RDD. But we can always create a new one.
  • RDD supports two types of Spark operations - Transformations & Actions.

 

 

Hope you like it.


Comments

Post a Comment

Popular Posts

Activity Dependencies with Logical AND in ADF

Image
Introduction In this article we are going to discuss about Activity Dependencies at ADF Control flow. As a SSIS developer we all have Experience about Precedence Constrains and how useful it is when building SSIS control flow. In ADF we have the similar concept with some limitation. Depends on this, we have to design our control flow in ADF. Hope it will be interesting.   Precedence Constraints in SSIS Before moving further, we have to review our SSIS Precedence Constraints.   Here we have: 1.       Constraint – Success, Failure, Completion 2.       Expression 3.       Logical AND / OR   It is really flexible.   Activity Dependency in ADF     In ADF Activity Dependencies provides only Logical AND not any Logical OR. It has Add Activity on: 1.       Success 2.       Failure 3.    ...
Read more

Working with Python – Module

Image
Introduction In this blog post, we are going to discuss about Python Module. Before that, we have to understand what the module is. Module is a code Library. Remember, when we are at C or C++, we are using #Include statement at beginning of the code to call specific code library. How to Create a Module If we need to create a module, we just save our code with  .py   extension. def  greeting(name):    print ( "Hello, "  + name)   Suppose, we have the above code and we are just saving the code in the name of  mymodule.py Calling Module   import   mymodule mymodule.greeting( "Mayuree" )   Hello, Mayuree   If we are looking at the calling procedure. mymodule . greeting ( "Mayuree" ) mymodule  – Module Name greeting  – Function Name   Can we assign Variable in Module   Module can have function that we already demonstrate. However, it can also have Variables. Example: ...
Read more