Azure DataBricks Architecture Part-2 (DataBricks Cluster)

 

Introduction:

In this Blog post we are going to discuss about DataBricks cluster type and creation options that need to be selected when creating DataBricks cluster.

 


Type of DataBricks Cluster:

Mainly we can divide it into two types.

·         All-purpose Cluster

·         Job Cluster

Difference between Them:

All-purpose Cluster

Job Cluster

Created Manually

Created By JOB

Persisted (Can Terminated Manually)

Non Persisted (Terminated when Job Ends)

Suitable for Interactive Workload

Suitable for Automated Work Load

Shared among many users

Isolated Just for JOB

Expensive to run

Cheaper to run

 

Note: We cannot Create Job cluster. It is automatically created when Job runs.

Cluster Configuration option Details:

We need to select those options when creating cluster.

 



 

·         Single/Multi Node:

In Single node there is only one node. There is no worker node present. It is not suitable for large ETL. In multi node, there are one master node and multiple worker nodes. It is generally used to heavy load balancing.

 

·         Access Mode:

a. Single User:
Only one user can access it. It is supported by the Python, SQL, Scala and R.

b. Shared:
Multiple users can access. Only available in the Premium and supported by Python and SQL. It provides process isolation. One process cannot see the other process, data and credentials.

c. No Isolation Shared:
Multiuser access. Support Python, SQL, Scala and R.  It is supported by Standard and Premium version. It is not providing any process isolation. Failing of a process can affect others. It is less secure. One process may use all the resources.

 

DataBricks Runtime:

It is the library that runs on DataBricks cluster. There are 4 library mentioned bellow.

·         DataBricks Runtime:
Support optimized version of spark. Support scala, java, Python and R. It supports ubuntu libraries, GPU libraries, Delta Lake and other DataBricks services.

·         DtaBricks Runtime ML:
All the libraries form DataBricks runtime with popular ML libraries such as PyTorch, Keras. TensorFlow, XG Boost etc.

·         Photon Runtime:
Support everything from DataBricks runtime and Photon engine.

·         DataBricks runtime light:
Runtime option for only Jobs not requiring advance feature.


Auto Termination:
It reduce unnecessary cost of ideal cluster. We can terminate the cluster when it is not use after specified minutes. Default value for single node and standard cluster is 120 minutes. We can specify the value from 10 to 10000 minutes as duration.

 

Auto Scaling:
Can automatically add and remove cluster. We can specify minimum and maximum nodes. Auto scaling between min and max node based on workload. It is not recommended for streaming workload.

 

Cluster VM Type / Size:

·         Memory optimized.

·         Compute optimized

·         Storage optimized

·         General purpose

·         CPU Accelerated

 

Cluster Policy:
Admin user can create cluster policy with restriction and assigned it to user and groups.

 

Comments

Post a Comment

Popular Posts

Working with Spark – Spark RDD

Image
Introduction In my previous blog post, we are discussing about Hadoop Map Reduce. Now in this blog post we are going to discuss about Spark Fundamentals concept like RDD. RDD stands for  Resilient Distributed  Dateset , these are the elements that run and operate on multiple nodes to do parallel processing on a cluster.  Hope it will be interesting. What is RDD According to Apache Spark documentation -  " Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. There are two ways to create RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat ". RDDs are immutable elements, which means once we create an RDD we cannot change it. RDDs are fault tolerant as well, hence in case of any failure, they ...
Read more

Activity Dependencies with Logical AND in ADF

Image
Introduction In this article we are going to discuss about Activity Dependencies at ADF Control flow. As a SSIS developer we all have Experience about Precedence Constrains and how useful it is when building SSIS control flow. In ADF we have the similar concept with some limitation. Depends on this, we have to design our control flow in ADF. Hope it will be interesting.   Precedence Constraints in SSIS Before moving further, we have to review our SSIS Precedence Constraints.   Here we have: 1.       Constraint – Success, Failure, Completion 2.       Expression 3.       Logical AND / OR   It is really flexible.   Activity Dependency in ADF     In ADF Activity Dependencies provides only Logical AND not any Logical OR. It has Add Activity on: 1.       Success 2.       Failure 3.    ...
Read more

Working with Python – Module

Image
Introduction In this blog post, we are going to discuss about Python Module. Before that, we have to understand what the module is. Module is a code Library. Remember, when we are at C or C++, we are using #Include statement at beginning of the code to call specific code library. How to Create a Module If we need to create a module, we just save our code with  .py   extension. def  greeting(name):    print ( "Hello, "  + name)   Suppose, we have the above code and we are just saving the code in the name of  mymodule.py Calling Module   import   mymodule mymodule.greeting( "Mayuree" )   Hello, Mayuree   If we are looking at the calling procedure. mymodule . greeting ( "Mayuree" ) mymodule  – Module Name greeting  – Function Name   Can we assign Variable in Module   Module can have function that we already demonstrate. However, it can also have Variables. Example: ...
Read more