Azure DataBricks Accessing Data Lake (Using Access Key)

 

Introduction:
In this Blog post we are going to discuss about accessing Azure Data Lake Gen2.

 

How can we Access the Azure Data Lake Gen2:

Access can be done by

Ø  Using Storage Access key

Ø  Using Shared access signature (SAS token)

Ø  Using Service Principal

Ø  Using Azure Active directory authentication pass-through

Ø  Using unity catalog

 

In this post we are going to discuss about accessing Azure Data Lake Gent 2 by using Access key.





Authenticate Data Lake with Access Key:

·         Each storage account comes with 2 access keys

·         Each access key is 512 bits

·         Access key gives full access of storage account

·         Conceder it as super user

·         Key can be rotated (re-generated)

 

Access Key Spark configuration:

Here we take myschool as Azure Data Lake gen2 and there is a container within this data lake named bronze. The bronze containers have a csv file named school.csv

 

spark.conf.set(

    "fs.azure.account.key.myschool.dfs.core.windows.net",

    "<512 bit access key in Azure data Lake>")

 

 

Microsoft recommended abfs (azure blob file system) driver protocol to access

abfss://bronze@myschool.dfs.core.windows.net

Note book command:

spark.conf.set(

    "fs.azure.account.key.myschool.dfs.core.windows.net",

    "<512 bit access key in Azure data Lake>")

dbutils.fs.ls("abfss://bronze@myschool.dfs.core.windows.net")

è  It gives us the list of files within bronze containers

display(Spark.read.csv("abfss://bronze@myschool.dfs.core.windows.net/school.csv"))

è  It is going to read and display the school.csv file

Comments

Post a Comment

Popular Posts

Copying Multiple File from Blob Storage to Single SQL Table

Image
Introduction This blog is for SSIS people, who want to move to ADF v2. Here we are using a simple case scenario like SSIS and continuing to extend the solution in my coming blog post. Here we assume that the readers has basic understanding with ADF. Hope it will be interesting. Case Scenario The case scenario is most common for SSIS developer. I have a Blob Storage and there are two txt file with same metadata.   I need to load it an Azure SQL Server Table (Both the file).   Source Files in Azure Blob Storage ADF Solution Lookup Activity Configuration Here I am using a lookup to extract data from a table that have table name that need to be load from Azure Blob storage to Azure SQL Table object. SELECT FileNames FROM    LookupTab WHERE   NeedLoad = 'Y'   There is a property in Lookup named First Row Only We need to unchecked it as we need all the records that is return by our SQL statement. ForEach Activity Configura...
Read more

ADF using Parameterized Stored Procedure

Image
Introduction Here in this article, we are trying to demonstrate the Stored Procedure Activity of ADF and how we pass the parameters in it. Hope it will be interesting. Case Scenario We have a table named [ ProductInfo ]. It contains all the product name.   CREATE TABLE ProductInto (       ProductID    INT ,       ProductName VARCHAR ( 50 ) ) GO INSERT INTO ProductInto VALUES ( 1 , 'Tooth Past' ),( 2 , 'Tooth Brush' ) Now, we have another Table named [ OrderInfo ], contains all the order information. CREATE TABLE OrderInfo (    OrderID               INT ,    ProductID             INT ,    Qty                   INT ,    Rate   ...
Read more

Copying Multiple File from Blob Storage to Single SQL Table – Part-2

Image
Introduction This is the extension of my proves blog post. If you not go with previous, I strongly recommended you to read my previous blog post named “Copying Multiple File from Blob Storage to Single SQL Table”. Case Scenario Here in this project, we are going to retrieve the CSV file from Azure Blob Storage, but we must retrieve the current date file only and store it into a Azure SQL DB Table. For SSIS developer it is work of using Expression, here I am also using the expression but in different ways that supports by Azure ADF. Source Files in Azure Blob Storage We have 3 file in our Azure Blob Storage. But only one file have Modified Date 30/5/2020. So, we need to retrieve that file only which matches the current system date.   ADF Solution   Step-1 We are using Get MetaData   Activity which retrieve all file information from Blob Storage and send it to the ForEach Look container Activity. Step-2 Within the ForEach Loop Activity we used anothe...
Read more