Azure DataBricks Architecture Part-2 (DataBricks Cluster)

 

Introduction:

In this Blog post we are going to discuss about DataBricks cluster type and creation options that need to be selected when creating DataBricks cluster.

 


Type of DataBricks Cluster:

Mainly we can divide it into two types.

·         All-purpose Cluster

·         Job Cluster

Difference between Them:

All-purpose Cluster

Job Cluster

Created Manually

Created By JOB

Persisted (Can Terminated Manually)

Non Persisted (Terminated when Job Ends)

Suitable for Interactive Workload

Suitable for Automated Work Load

Shared among many users

Isolated Just for JOB

Expensive to run

Cheaper to run

 

Note: We cannot Create Job cluster. It is automatically created when Job runs.

Cluster Configuration option Details:

We need to select those options when creating cluster.

 



 

·         Single/Multi Node:

In Single node there is only one node. There is no worker node present. It is not suitable for large ETL. In multi node, there are one master node and multiple worker nodes. It is generally used to heavy load balancing.

 

·         Access Mode:

a. Single User:
Only one user can access it. It is supported by the Python, SQL, Scala and R.

b. Shared:
Multiple users can access. Only available in the Premium and supported by Python and SQL. It provides process isolation. One process cannot see the other process, data and credentials.

c. No Isolation Shared:
Multiuser access. Support Python, SQL, Scala and R.  It is supported by Standard and Premium version. It is not providing any process isolation. Failing of a process can affect others. It is less secure. One process may use all the resources.

 

DataBricks Runtime:

It is the library that runs on DataBricks cluster. There are 4 library mentioned bellow.

·         DataBricks Runtime:
Support optimized version of spark. Support scala, java, Python and R. It supports ubuntu libraries, GPU libraries, Delta Lake and other DataBricks services.

·         DtaBricks Runtime ML:
All the libraries form DataBricks runtime with popular ML libraries such as PyTorch, Keras. TensorFlow, XG Boost etc.

·         Photon Runtime:
Support everything from DataBricks runtime and Photon engine.

·         DataBricks runtime light:
Runtime option for only Jobs not requiring advance feature.


Auto Termination:
It reduce unnecessary cost of ideal cluster. We can terminate the cluster when it is not use after specified minutes. Default value for single node and standard cluster is 120 minutes. We can specify the value from 10 to 10000 minutes as duration.

 

Auto Scaling:
Can automatically add and remove cluster. We can specify minimum and maximum nodes. Auto scaling between min and max node based on workload. It is not recommended for streaming workload.

 

Cluster VM Type / Size:

·         Memory optimized.

·         Compute optimized

·         Storage optimized

·         General purpose

·         CPU Accelerated

 

Cluster Policy:
Admin user can create cluster policy with restriction and assigned it to user and groups.

 

Comments

Post a Comment

Popular Posts

Vertipaq Engine – Data Compression

Image
Introduction Microsoft introduces a very powerful engine for fast query named Vertipaq . Which is used in the Power BI and SSAS Tabular model? Here we are not going to discuss about Power BI or SSIS Tabular model. Our main concern is to identify how the Vertipaq Engine works and how it provide Fastest Query. The Vertipaq Engine stores the table in a columnar format and established relation between columns. In this blog post we are going to discuss about Data Compression mechanism and try to explain it.   Case Study We have a table which is imported into Vertipaq engine. Table Name:  Order_Details     How Vertipaq works on this Table Vertipaq Compress Text in a Table Text takes lot of space in data base. So Vertipaq need to compress it. It maintain another table with distinct Text value and Id number like this. It is also called Dictionary/ Hash compression.   Product_ID_Dictionary              Value Encoding Here Vertipaq tries to convert Value into One Digit where as possible. TO u
Read more

Azure Data Bricks Architecture Part-1

Image
  Introduction:   We decide to provide several blog posts for learning Azure DataBricks step by step. This blog post is useful for beginners who want to start Azure DataBricks.   Azure DataBricks High Level Diagram:   Azure DataBricks is an analytics platform optimized for the Microsoft Azure cloud services platform. It provides a collaborative environment with Apache Spark-based analytics that enables big data processing, real-time analytics, and machine learning tasks. Here are some key features of Azure DataBricks: ·          Optimized Apache Spark Environment: It allows you to set up an Apache Spark environment quickly, with autoscaling and auto-termination to optimize costs. ·          Collaborative Workspace: It supports multiple languages like Python, Scala, R, Java, and SQL, and integrates with tools like GitHub and Azure DevOps for version control. ·          Machine Learning Capabilities: Azure DataBricks integrates with Azure Machine Learning to provide au
Read more

Azure DataBricks Accessing Data Lake (Using Access Key)

Image
  Introduction: In this Blog post we are going to discuss about accessing Azure Data Lake Gen2.   How can we Access the Azure Data Lake Gen2: Access can be done by Ø   Using Storage Access key Ø   Using Shared access signature (SAS token) Ø   Using Service Principal Ø   Using Azure Active directory authentication pass-through Ø   Using unity catalog   In this post we are going to discuss about accessing Azure Data Lake Gent 2 by using Access key. Authenticate Data Lake with Access Key: ·          Each storage account comes with 2 access keys ·          Each access key is 512 bits ·          Access key gives full access of storage account ·          Conceder it as super user ·          Key can be rotated (re-generated)   Access Key Spark configuration: Here we take myschool as Azure Data Lake gen2 and there is a container within this data lake named bronze . The bronze containers have a csv file named school.csv   spark.conf.set(     "
Read more