Parameters and Variable is ADF

Introduction
In this Blog post, we are going to explore related to pipeline Parameters and Variable.

Hope it will be interesting.

 

Little Theoretical portion that we need to know

 

If we think about SSIS, we have package level parameters and we can define variables within the package. We have to define the scope of the variable, where we can access it or not, such as entire package or a definite container.

 

In ADF also, we have both Pipeline level Parameters and Variables.

 

A Parameter has data type like String, Int, Float, Bool, Array, Object and Secure String. But one thing we have to remember that, when we define a parameters and its value we are unable to over write parameters value within any activity of pipeline. We can just read the value of parameters.

 

Note : SecureString is a string type that provides a measure of security. It tries to avoid storing potentially sensitive strings in process memory as plain text. This type implements the IDisposable interface. When you have finished using an instance of the type, you should dispose of it either directly or indirectly.

 

In other side, a Variable has only there type of Data Type i.e. String, Array, Boolean. We can Access the Variable in any activity within the pipeline and also assigned the value.  Assignment of value within variable is done by two activities named Set Variables and Appends Variables.

 

Set variable is used to assign the value of all kind of variables. In other hand the Append Variable is used to assign the value of Array Data Type.

 

 

Now proceed to a simple Example to understand

 

ADF Flow

 


 

Pipeline Parameter and Variable Declaration

 


Set Variable Activity

Here the name is “Set Files”

 


 

Here the variable named “Files” is an Array Type, we can assign multiple value by using coma separator. Please remember space is not allowed. In our example we used

 

colors,inventories,inventory_parts,part_categories,parts,sets,themes

 

If Connection Activity

 

Remember we used a parameters in our pipe line named “LoadAllFiles” type Boolean. By using true or false we can decide the If Condition. Please remember the boolen true or false is in small letter only.

 


 

Append Variables Activity

 


 

Output

 


 


 


 


 

 

Hope you like it.


Comments

Post a Comment

Popular Posts

Triggering Pipeline in ADF

Image
Introduction Pipeline is ready and we already publish it, now we need to trigger our pipeline in a schedule interval. For SSIS Developer, we have SQL Server Agent job to schedule and run the SSIS Project or Package.   Scheduling Jobs in SQL agent have some limitation. We have to logically check the SSIS Package execution is completed or not before proceeding to next run of same or dependent project. It also has some flexibility by using Steps. Suppose we need to run a SQL Script on Step-1, then SSIS package on Step-2. We are trying to say that dependent activity can run step by step. In Azure ADF triggering, we have some options that we not find in SQL Agent Job, so both have some advantage and Limitation also. In this blog post we are going to discuss ADF Add Trigger options. Hope it will be interesting.   Where we get the Add Trigger     When we open the Pipeline we can get the Add Trigger button beside the Debug button. After clicking this butt...
Read more

Working with Python – Class and Object Concept

Image
Introduction Python is an Object oriented Language. An object contains Propertied and Method and the Class is the blueprint of creating Object. In this Blog post, we are going to discuss about Class and Objects of Python. How to Create a Class To create a class we must use the keyword  Class . Example: class  ExampleClass:       x =  5 In this example we are just create a class named ExampleClass .   How to Create an Object After creating class, we have to create the Object. class  ExampleClass:       x =  5 a = ExampleClass() print (a.x)   In the above example, the class is as a simplest form. To understand the class, we have to understand the  __Init__()   function     Understanding the __init__() Function Hope, we all have the knowledge of  constructor  in C++ Language. __init__() function do the same thing as Constructer does. This function is called au...
Read more

Working with Spark – Spark RDD

Image
Introduction In my previous blog post, we are discussing about Hadoop Map Reduce. Now in this blog post we are going to discuss about Spark Fundamentals concept like RDD. RDD stands for  Resilient Distributed  Dateset , these are the elements that run and operate on multiple nodes to do parallel processing on a cluster.  Hope it will be interesting. What is RDD According to Apache Spark documentation -  " Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. There are two ways to create RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat ". RDDs are immutable elements, which means once we create an RDD we cannot change it. RDDs are fault tolerant as well, hence in case of any failure, they ...
Read more