Joining multi Flat files - JSON File and Table in ADF Data Flow

Introduction

In this blog post, we are just going to Join and Join but in logical manners to understand ADF Data Flow. Here in ADF Data flow we are using Flatten, Select and Join Transform.

Hope it will be interesting.

 

Case scenario

We have some Flat file which contains Employee Code, Salary and Bonus Information. There are multiple Flat files in our Azure BLOB Container.

Now we have a JSON file which contains information about Employee Code, Employee Name and Employee Email. We also have table object named Employee State. Which contain Employee Code and State name.

 


 

Now we have to join all using ADF data flow. First we used Get Meta Data activity to collect all file information. By using ForEach we are moving one by one file. Within the for Each we have Mapping Data Flow activity which actually call our data low. Rest are done by Data Flow.

 

ADF Solution

 

Control Flow Structure

 


 

Data Flows Structure

 

 

 

Here we are not going to describe related to Control Flow and Parameterized Data sets. You can easily capture it from my previous blog post.

 

From Control Flow, we are going to show only Mapping Data Flow.

 

Mapping Data Flow of Control Flow

 


 

Data Flow

 

Source 1

 


 

Source 2

 


 

Source 3

 


 

Flatten Transform

 


 

JOIN Transform

 


 

 

Sink and Output

 


 

 

 

Hope you like it.


Comments

Post a Comment

Popular Posts

Implementing LIKE Operator in ADF

Image
Introduction SQL WHERE clause we often use LIKE operator with wild card character (% or _). But in ADF we can’t use it directly.  We can use heard coded LIKE and wildcard character in Control flow Lookup Activity, but we need to automate the process. Hope it will be interesting.   Case Study We have a Flat file named DeveloperList.txt and a AZURE SQL Table object named [dbo].[WorkCommesta]     Now we need to search each [ DevName ] from DeveloperList.txt to our [dbo].[WorkComments].[comments] table using wild card and LIKE operator. It just likes SQL command:   Select * From Workcommensts WHERE comments LIKE '%Joydeep%'   Stored Procedure     ADF Control Flow     Outer Lookup Activity     ForEach Activity     Inner Lookup Activity (Under ForEach)     Set Variable Activity (Under ForEach)   Stored Procedure Activity (Under Fo...
Read more

Using Lookup in ADF Data Flow

Image
Introduction SSIS developer, we all know the importance of Lookup Transform . Here in ADF, we have two types of Lookup. One is for control flow and other is for data flow. The behavior of both the lookup is not same. Control flow lookup provide only record sets but in Data flow we are using it as for looking up some records from other data sets. In ADF data flow Lookup feature is more than the SSIS Lookup Transform. So, in this article we are thinking to demonstrate all the functionality of ADF Data Flow lookup with a simple example.   Hope it will be interesting.   Case scenario We have a Flat file which contains Employee Code, Salary and Bonus Information. There are multiple Flat files in our Azure BLOB Container. Now we have a JSON file which contains information about Employee Code, Employee Name and Employee Email. Now we have to use the Lookup Transform to find the Employee Name and Employee Email from our JSON file.   About Data Flow Lookup Be...
Read more

Working with Python – Class and Object Concept

Image
Introduction Python is an Object oriented Language. An object contains Propertied and Method and the Class is the blueprint of creating Object. In this Blog post, we are going to discuss about Class and Objects of Python. How to Create a Class To create a class we must use the keyword  Class . Example: class  ExampleClass:       x =  5 In this example we are just create a class named ExampleClass .   How to Create an Object After creating class, we have to create the Object. class  ExampleClass:       x =  5 a = ExampleClass() print (a.x)   In the above example, the class is as a simplest form. To understand the class, we have to understand the  __Init__()   function     Understanding the __init__() Function Hope, we all have the knowledge of  constructor  in C++ Language. __init__() function do the same thing as Constructer does. This function is called au...
Read more