Triggering Pipeline in ADF

Introduction

Pipeline is ready and we already publish it, now we need to trigger our pipeline in a schedule interval. For SSIS Developer, we have SQL Server Agent job to schedule and run the SSIS Project or Package.  

Scheduling Jobs in SQL agent have some limitation. We have to logically check the SSIS Package execution is completed or not before proceeding to next run of same or dependent project. It also has some flexibility by using Steps. Suppose we need to run a SQL Script on Step-1, then SSIS package on Step-2. We are trying to say that dependent activity can run step by step.

In Azure ADF triggering, we have some options that we not find in SQL Agent Job, so both have some advantage and Limitation also.

In this blog post we are going to discuss ADF Add Trigger options.

Hope it will be interesting.

 

Where we get the Add Trigger

 


 

When we open the Pipeline we can get the Add Trigger button beside the Debug button. After clicking this button we have two options.

1.      Trigger Now

2.      New/Edit

Trigger Now option, immediately run the Pipeline activity. For Settings of trigger, we need to click on New/Edit option.

 

New Trigger

 

There are three type of trigger we can set

1.      Schedule

2.      Tumbling Window

3.      Event

 

Schedule

As the name mentioned, this one is used to execute the Pipeline within a specified Date time.

 


 

Tumbling Window

Tumbling window trigger is a type of trigger that fires at periodic time intervals from a specified start time. Tumbling windows are a series of fixed-sized, non-overlapping and contiguous time intervals. It is important to remember that tumbling window triggers can be configured to initiate past and future dated loads. A tumbling window trigger has a one-to-one relationship with a pipeline and can only reference a single pipeline.

As mentioned in the previous section, a tumbling window trigger allows loading data for past and future periods. As the name suggests, the time slices handled by a tumbling window trigger must be, fixed, non-overlapping, and contiguous, see the below figure as an example for a trigger with hourly windows:

 

 

 

Event

 

An event-based trigger runs pipelines in response to an event, such as the arrival of a file, or the deletion of a file, in Azure Blob Storage.

 


 

 

Hope you like it.


Comments

Post a Comment

Popular Posts

Working with Python – Class and Object Concept

Image
Introduction Python is an Object oriented Language. An object contains Propertied and Method and the Class is the blueprint of creating Object. In this Blog post, we are going to discuss about Class and Objects of Python. How to Create a Class To create a class we must use the keyword  Class . Example: class  ExampleClass:       x =  5 In this example we are just create a class named ExampleClass .   How to Create an Object After creating class, we have to create the Object. class  ExampleClass:       x =  5 a = ExampleClass() print (a.x)   In the above example, the class is as a simplest form. To understand the class, we have to understand the  __Init__()   function     Understanding the __init__() Function Hope, we all have the knowledge of  constructor  in C++ Language. __init__() function do the same thing as Constructer does. This function is called au...
Read more

Working with Spark – Spark RDD

Image
Introduction In my previous blog post, we are discussing about Hadoop Map Reduce. Now in this blog post we are going to discuss about Spark Fundamentals concept like RDD. RDD stands for  Resilient Distributed  Dateset , these are the elements that run and operate on multiple nodes to do parallel processing on a cluster.  Hope it will be interesting. What is RDD According to Apache Spark documentation -  " Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. There are two ways to create RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat ". RDDs are immutable elements, which means once we create an RDD we cannot change it. RDDs are fault tolerant as well, hence in case of any failure, they ...
Read more