Azure Data Bricks Architecture Part-1

 

Introduction:
 We decide to provide several blog posts for learning Azure DataBricks step by step. This blog post is useful for beginners who want to start Azure DataBricks.

 

Azure DataBricks High Level Diagram:

 




Azure DataBricks is an analytics platform optimized for the Microsoft Azure cloud services platform. It provides a collaborative environment with Apache Spark-based analytics that enables big data processing, real-time analytics, and machine learning tasks.

Here are some key features of Azure DataBricks:

·         Optimized Apache Spark Environment: It allows you to set up an Apache Spark environment quickly, with autoscaling and auto-termination to optimize costs.

·         Collaborative Workspace: It supports multiple languages like Python, Scala, R, Java, and SQL, and integrates with tools like GitHub and Azure DevOps for version control.

·         Machine Learning Capabilities: Azure DataBricks integrates with Azure Machine Learning to provide automated machine learning capabilities, making it easier to identify suitable algorithms and manage machine learning models.

·         Modern Data Warehousing: It enables you to combine data at any scale, providing insights through analytical dashboards and operational reports.

·         It’s designed to simplify and accelerate data science on large datasets and is rooted in open source with a commitment to integrating with open-source tools.

 

What Spark Provides:

·         100% open source under Apache license

·         Simple and easy to use API

·         In memory processing engine

·         Distributed computing platform

·         Unified engine which support SQL, Streaming, ML and Graph Processing

High Level Architecture of SPARK

 



 

Note: Catalyst Optimizer is used for High Optimized Query and Tungsten is used for Memory management.

What DataBricks Provide to Spark:

·         Cluster

·         Work Space/Note Book

·         Administration and Controls

·         Optimized Spark (5x Faster)

·         Database / Table

·         Delta Lake

·         SQL Analytics

·         ML Flow

 

What Azure Provide with DataBricks:

·         Azure Active Directory

·         Unified Azure Portal and Billing

·        Data Services Like
Data Lake, Blob Storage, Azure Cosmos DB, SQL DB, Synapse

·         Messaging Service
Azure IOT Hub, Azure Event Hub

·         Power BI

·         Azure ML

·         Azure Data Factory

·         Azure DevOps

Comments

Post a Comment

Popular Posts

Working with Spark – Spark RDD

Image
Introduction In my previous blog post, we are discussing about Hadoop Map Reduce. Now in this blog post we are going to discuss about Spark Fundamentals concept like RDD. RDD stands for  Resilient Distributed  Dateset , these are the elements that run and operate on multiple nodes to do parallel processing on a cluster.  Hope it will be interesting. What is RDD According to Apache Spark documentation -  " Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. There are two ways to create RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat ". RDDs are immutable elements, which means once we create an RDD we cannot change it. RDDs are fault tolerant as well, hence in case of any failure, they ...
Read more

Activity Dependencies with Logical AND in ADF

Image
Introduction In this article we are going to discuss about Activity Dependencies at ADF Control flow. As a SSIS developer we all have Experience about Precedence Constrains and how useful it is when building SSIS control flow. In ADF we have the similar concept with some limitation. Depends on this, we have to design our control flow in ADF. Hope it will be interesting.   Precedence Constraints in SSIS Before moving further, we have to review our SSIS Precedence Constraints.   Here we have: 1.       Constraint – Success, Failure, Completion 2.       Expression 3.       Logical AND / OR   It is really flexible.   Activity Dependency in ADF     In ADF Activity Dependencies provides only Logical AND not any Logical OR. It has Add Activity on: 1.       Success 2.       Failure 3.    ...
Read more

Working with Python – Module

Image
Introduction In this blog post, we are going to discuss about Python Module. Before that, we have to understand what the module is. Module is a code Library. Remember, when we are at C or C++, we are using #Include statement at beginning of the code to call specific code library. How to Create a Module If we need to create a module, we just save our code with  .py   extension. def  greeting(name):    print ( "Hello, "  + name)   Suppose, we have the above code and we are just saving the code in the name of  mymodule.py Calling Module   import   mymodule mymodule.greeting( "Mayuree" )   Hello, Mayuree   If we are looking at the calling procedure. mymodule . greeting ( "Mayuree" ) mymodule  – Module Name greeting  – Function Name   Can we assign Variable in Module   Module can have function that we already demonstrate. However, it can also have Variables. Example: ...
Read more