Vertipaq Engine – Columnar Database

 Introduction

 In our previous post, we are discussing about Vertipaq Data compression one of the best feature of Vertipaq engine by Microsoft. In this post we are trying to discuss about Vertipaq Columnar Database. The ultimate feature of Vertipaq engine is to speed up the query performance.

Hope it will be interesting.

 

What the Special in Vertipaq Engine

Vertipaq is an in-memory columnar database. Being in-memory means that all of the data handled by a model reside in RAM.

 

How Database works with a Table

To understand the columnar database we have to understand, how data is retrieved from a table in our traditional database like MS SQL Server.

Let’s take an example of a table.

Table Name: tbl_OrderDetails


The data in a Database table stores in 8 KB page named data page. The 8 data page combined together to from an Extinct.

When we search for a specified product to understand how many Quantity has been sold the entire data page with all the columns comes into memory and search the desired Product one by one. So the load of the memory (RAM) is huge for long table. Please remind that we need only columns named ProductName and Qty only.  Rest of the columns is not needed to get the answer of how many quantity is sold for a specified product. It is just unnecessary down load into the memory and holding the space and hence for long table the performance is slow.


How the Vertipaq Engine works

The In memory storage of Vertipaq is different than the traditional DB storage. It stores the data in to columnar fission.


To complete the above query it takes only two columns named ProductName and Qty. It search the desired product from ProductName columns and sum the value of Quantity columns.



 

 Hope you like that.

Comments

Post a Comment

Popular Posts

Working with Spark – Spark RDD

Image
Introduction In my previous blog post, we are discussing about Hadoop Map Reduce. Now in this blog post we are going to discuss about Spark Fundamentals concept like RDD. RDD stands for  Resilient Distributed  Dateset , these are the elements that run and operate on multiple nodes to do parallel processing on a cluster.  Hope it will be interesting. What is RDD According to Apache Spark documentation -  " Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. There are two ways to create RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat ". RDDs are immutable elements, which means once we create an RDD we cannot change it. RDDs are fault tolerant as well, hence in case of any failure, they ...
Read more

Activity Dependencies with Logical AND in ADF

Image
Introduction In this article we are going to discuss about Activity Dependencies at ADF Control flow. As a SSIS developer we all have Experience about Precedence Constrains and how useful it is when building SSIS control flow. In ADF we have the similar concept with some limitation. Depends on this, we have to design our control flow in ADF. Hope it will be interesting.   Precedence Constraints in SSIS Before moving further, we have to review our SSIS Precedence Constraints.   Here we have: 1.       Constraint – Success, Failure, Completion 2.       Expression 3.       Logical AND / OR   It is really flexible.   Activity Dependency in ADF     In ADF Activity Dependencies provides only Logical AND not any Logical OR. It has Add Activity on: 1.       Success 2.       Failure 3.    ...
Read more

Working with Python – Module

Image
Introduction In this blog post, we are going to discuss about Python Module. Before that, we have to understand what the module is. Module is a code Library. Remember, when we are at C or C++, we are using #Include statement at beginning of the code to call specific code library. How to Create a Module If we need to create a module, we just save our code with  .py   extension. def  greeting(name):    print ( "Hello, "  + name)   Suppose, we have the above code and we are just saving the code in the name of  mymodule.py Calling Module   import   mymodule mymodule.greeting( "Mayuree" )   Hello, Mayuree   If we are looking at the calling procedure. mymodule . greeting ( "Mayuree" ) mymodule  – Module Name greeting  – Function Name   Can we assign Variable in Module   Module can have function that we already demonstrate. However, it can also have Variables. Example: ...
Read more