Flattening the JSON file by using ADF Data Flow

Introduction


In this Blog post, we are going to Flattening the JSON file by using ADF Data Flow.

Hope it will be interesting.

 

JSON File Structure

 


 

If you look at the JSON structure carefully, you find some Array also present in the nesting.

 

ADF Flow

 


 

In our Pipeline we have a Mapping Data Flow Task only. The entire structure is build in Data Flow Task.

 

Source JSON Data Sets

 


 

Add Source Settings

 


 


 

Flatten 1 Settings

 


 

Flatten 2 Settings

 


 

Sink Settings

 


 

Output

 


 

Hope you like it.


Comments

Post a Comment

Popular Posts

Vertipaq Engine – Column & Segment Elimination

Image
  Introduction As we are discussing about the power of Vertipaq Engine, in this post we are going to discuss about another beautiful technique that Vertipaq Engine maintain to make our query faster and that is Column Elimination and Segment Elimination.   Hope it will be interesting. Case Study We have a table which is imported into Vertipaq engine. Table Name: Order_Details   Column Elimination As Vertipaq Engine stores data as a columner fashion so it’s looks like.   Now we need to fetch data for Customer ‘ABC Company’ and want to know the Total Quntity the specified company purchased. For that we need only columns named Customer and Qty. All others columns are simply eliminated. It actually limit the data by eleminating the columns and called columns elemenation. Segment Elimination This is done by horizontal Partitioning.   For Power Pivot it takes 1 million per partition and for SSAS Tabular it takes 8 million per Partit...
Read more

Azure Data Bricks Architecture Part-1

Image
  Introduction:   We decide to provide several blog posts for learning Azure DataBricks step by step. This blog post is useful for beginners who want to start Azure DataBricks.   Azure DataBricks High Level Diagram:   Azure DataBricks is an analytics platform optimized for the Microsoft Azure cloud services platform. It provides a collaborative environment with Apache Spark-based analytics that enables big data processing, real-time analytics, and machine learning tasks. Here are some key features of Azure DataBricks: ·          Optimized Apache Spark Environment: It allows you to set up an Apache Spark environment quickly, with autoscaling and auto-termination to optimize costs. ·          Collaborative Workspace: It supports multiple languages like Python, Scala, R, Java, and SQL, and integrates with tools like GitHub and Azure DevOps for version control. ·  ...
Read more

Azure DataBricks Architecture Part-2 (DataBricks Cluster)

Image
  Introduction: In this Blog post we are going to discuss about DataBricks cluster type and creation options that need to be selected when creating DataBricks cluster.   Type of DataBricks Cluster: Mainly we can divide it into two types. ·          All-purpose Cluster ·          Job Cluster Difference between Them: All-purpose Cluster Job Cluster Created Manually Created By JOB Persisted (Can Terminated Manually) Non Persisted (Terminated when Job Ends) Suitable for Interactive Workload Suitable for Automated Work Load Shared among many users Isolated Just for JOB Expensive to run Cheaper to run   Note: We cannot Create Job cluster. It is automatically created when Job runs. Cluster Configuration option Details: We need to select ...
Read more