Vertipaq Engine – Column & Segment Elimination

 Introduction

As we are discussing about the power of Vertipaq Engine, in this post we are going to discuss about another beautiful technique that Vertipaq Engine maintain to make our query faster and that is Column Elimination and Segment Elimination.  

Hope it will be interesting.

Case Study

We have a table which is imported into Vertipaq engine.

Table Name: Order_Details


 

Column Elimination

As Vertipaq Engine stores data as a columner fashion so it’s looks like.


 

Now we need to fetch data for Customer ‘ABC Company’ and want to know the Total Quntity the specified company purchased. For that we need only columns named Customer and Qty. All others columns are simply eliminated.


It actually limit the data by eleminating the columns and called columns elemenation.

Segment Elimination

This is done by horizontal Partitioning.


 

For Power Pivot it takes 1 million per partition and for SSAS Tabular it takes 8 million per Partition.

Now how it works.

 

Suppose we have query to fetch number of Quantity sold for customer ‘ABC Company’ within the period of 1/10/219 to 8/12/2019. In this situation it just simply eliminated one segment that have higher range of data.


 

Hope you like that.

Comments

  1. SunTec Business Solutions23 December 2021 at 11:25

    pricing management software for banking
    Personalize products, offers, pricing and loyalty programs; prevent revenue leakage and ensure regulatory compliance with a billing solution.

    ReplyDelete

Post a Comment

Popular Posts

Incremental Load by using ADF

Image
Introduction lot of my friend from different group is asking to provide a blog post related to incremental load. If you search Google related to this tropics, you can find bundle of blog post. All are the same using water mark technique. But in my view point it is not a practical approach for incremental load. When we talk about incremental load, we assume that we have billions of records in source which is ever changing and we need to move on destination table. Here in this demonstration we used ADF control flow but it can be done with any ETL or ELT tools like SSIS. One last thing that I would like to mention if we try to load huge records from source to designation, it definitely gives us performance problem or stuck in between. So, we have to load the data in a smaller chunk or batch. We have to keep in track that data that we already loaded in the destination is going to updated in source or not. If it is going too updated, we have to update the destination accordingly. Yo...
Read more

Working with Spark – Spark RDD

Image
Introduction In my previous blog post, we are discussing about Hadoop Map Reduce. Now in this blog post we are going to discuss about Spark Fundamentals concept like RDD. RDD stands for  Resilient Distributed  Dateset , these are the elements that run and operate on multiple nodes to do parallel processing on a cluster.  Hope it will be interesting. What is RDD According to Apache Spark documentation -  " Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. There are two ways to create RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat ". RDDs are immutable elements, which means once we create an RDD we cannot change it. RDDs are fault tolerant as well, hence in case of any failure, they ...
Read more

Activity Dependencies with Logical AND in ADF

Image
Introduction In this article we are going to discuss about Activity Dependencies at ADF Control flow. As a SSIS developer we all have Experience about Precedence Constrains and how useful it is when building SSIS control flow. In ADF we have the similar concept with some limitation. Depends on this, we have to design our control flow in ADF. Hope it will be interesting.   Precedence Constraints in SSIS Before moving further, we have to review our SSIS Precedence Constraints.   Here we have: 1.       Constraint – Success, Failure, Completion 2.       Expression 3.       Logical AND / OR   It is really flexible.   Activity Dependency in ADF     In ADF Activity Dependencies provides only Logical AND not any Logical OR. It has Add Activity on: 1.       Success 2.       Failure 3.    ...
Read more