Illustration of Incremental Load in Real Time

The practice of loading only new or modified records from a database into an existing QVD is known as an incremental load. As compared to complete loads, incremental loads are more effective, which is especially useful for large data sets. The incremental load can be applied in various ways, with the following being the most common:

• Insert only (Do not validate for duplicate records).
• Insert and update.
• Insert, update and delete.

1. Insert Only: 

Let’s assume we have sales raw data (in Excel) updated with necessary details about the transaction by modified date if a new sale is registered. We already had a QVD produced before yesterday because we are working on QVDs (25-Aug-14 in this case). Now you can load incremental data (Highlighted in yellow below).

To begin, build a QVD for data up until August 25, 2014. We need to know the date on which QVD was last changed to find new incremental data. The maximum Modified_date in the available QVD file will be used to determine this. As previously stated, It is concluded that “Sales. qvd” is up to date with data until August 25, 2014. The following code will be used to determine the last updated date of “Sales. qvd”:

We have loaded the most recent QVD into memory and then identified the most recent modified date by storing the maximum number of “Modified_Date” values. We then save this date in a variable called “Last_Updated_Date” and delete the “Sales” table. I used the Peek() function to store the maximum number of changed dates in the above code. The syntax is as follows:

Peek( FieldName, Row Number, TableName)

lets’s get started with QlikView Tutorial! 

This function retrieves the contents of a given field from an internal table row. FieldName and TableName must be string values, while Row must be an integer value. The first record is indicated by a 0, the second by a 1, and so on. Negative numbers indicate the order of the table from the top. The last record is indicated by a -1.

We can load incremental records of the data set (Where clause in Load statement) and merge them with available QVD because we know when the records will be considered new records after that date (Look at the snapshot below).

Now, load the most recent QVD (Sales), which will have incremental records.

As you can see, two records from August 26, 2014, have been added. However, we’ve also added a duplicate record. Since we haven’t accessed the available records, we may tell that an INSERT is the only approach that will not validate duplicate records.

Furthermore, we are unable to update the value of existing records using this method.

To recap, the steps to load only incremental records to QVD using the INSERT only method are as follows:

1. Recognize and load new records.
2. Combine this data with the QVD file.
3. Replace the old concatenated table with the new QVD file.

2. Insert and Update method:

We can’t search for duplicate records or update existing records, as seen in the previous case. The Insert and Update approach comes in handy here:

Assume ID is the primary key, and we should be able to define and distinguish new or updated records based on change date and ID.

To use this process, repeat the steps for identifying new records as in the INSERT the only method. Then, apply the search for duplicated records or change old records’ value when concatenating incremental data with existing records.

We’ve only loaded records where the Primary Key(ID) is new. The Exists() feature prevents the QVD from loading old records because the Latest version is already in memory, so expired record values are immediately updated.

Both specific records are now available in QVD, along with an updated sales value for ID (PRD858).

Subscribe to our YouTube channel to get new updates..!

3. INSERT, UPDATE, and DELETE method:

This method’s script is somewhat similar to the INSERT & UPDATE method, except there is an additional step to remove deleted records.

We’ll use an inner join with a concatenated data set (Old+Incremental) to load primary keys for all records in the new data set. Only common records shall be maintained, and unnecessary records will be deleted due to the inner join. Assume that in the previous case, we want to remove a record with the ID PRD1058.

We have a data set of one record added (ID PRD1458), one record modified (ID PRD158), and one record deleted (ID PRD1058).

Advantages of Incremental Load

The following are the benefits of the incremental load.

• By removing the maximum load of data, it provides a productive load at any time.
• As opposed to the standard model, it lowers the time it takes to get complete data by 100 times.
• Incremental load reduces the database’s traffic load.
• It reduces the workload for data source drivers.
• The Incremental load minimizes the load on RAM.
• It functions as a JIT (Just-In-Time) engine in the Data Extraction layer, fetching data in real-time.
• It makes use of QVD file formatted tables, which significantly compresses the results.

Data Localization

The incremental load uses newly added data and attaches it to the recently incremented table, resulting in data access that is still local to the BI application.

Conclusion

This blog has addressed how incremental loads are faster and more effective than FULL loads for loading data. You should make regular backups of your data as the best idea, and if there are problems with your database server or network, your data can be affected or lost. It would be best to choose which approach is best for you based on your business and application needs. Insert and Update is used in the majority of BFSI applications. In most cases, records are not deleted.

Other Related Articles: