Archive

Posts Tagged ‘SSIS’

Handling and Logging Sproc errors with SSIS

May 12th, 2009 No comments

I frequently use SSIS to schedule some maintence tasks using stored procedures. I want to use transactions to control my data and want to use SSIS to control my work flow. To accomplish this I use try..catch and Raiserror in the stored procedure and SSIS logging to keep track of the errors.

For example I use this sproc :

CREATE PROCEDURE MoveData
AS
INSERT INTO History.ReportTable
SELECT * FROM Reporting.ReportTable
WHERE reportdate < cast(dateadd(DAY,-2,GETDATE())as date);
 
 DELETE FROM Reporting.ReportTable
 WHERE reportdate < cast(dateadd(DAY,-2,GETDATE())as date);

But I want to control the data so all or nothing is commited to the data, since SQL Server 2005 you can use the try..catch function and use transactions with that, so adding that maken:

CREATE PROCEDURE MoveData
AS
BEGIN TRANSACTION;
BEGIN TRY

INSERT INTO History.ReportTable
SELECT * FROM Reporting.ReportTable
WHERE reportdate < cast(dateadd(DAY,-2,GETDATE())as date);
 
 DELETE FROM Reporting.ReportTable
 WHERE reportdate < cast(dateadd(DAY,-2,GETDATE())as date);

END TRY
BEGIN CATCH

 IF @@TRANCOUNT > 0
        ROLLBACK TRANSACTION;
END CATCH;
IF @@TRANCOUNT > 0
    COMMIT TRANSACTION;

This stored procedure makes sure that when the insert of delete fails for any reason it will be rollbacked. But my sproc will just run and not return a error, my SSIS execute SQL task won’t know anything went wrong. That’s were RAISERROR comes in. Adding that to the sproc makes:

CREATE PROCEDURE MoveData
AS
BEGIN TRANSACTION;
BEGIN TRY

INSERT INTO History.ReportTable
SELECT * FROM Reporting.ReportTable
WHERE reportdate < cast(dateadd(DAY,-2,GETDATE())as date);
 
 DELETE FROM Reporting.ReportTable
 WHERE reportdate < cast(dateadd(DAY,-2,GETDATE())as date);

END TRY
BEGIN CATCH
  DECLARE @ErrorMessage NVARCHAR(4000);
    DECLARE @ErrorSeverity INT;
    DECLARE @ErrorState INT;

– Get the current error
SELECT @ErrorMessage = ERROR_MESSAGE(),
           @ErrorSeverity = ERROR_SEVERITY(),
           @ErrorState = ERROR_STATE();

    — Use RAISERROR inside the CATCH block to return
    — error information about the original error that
    — caused execution to jump to the CATCH block.
    RAISERROR (@ErrorMessage, — Message text.
               @ErrorSeverity, — Severity.
               @ErrorState — State.
               );
 IF @@TRANCOUNT > 0
        ROLLBACK TRANSACTION;
END CATCH;
IF @@TRANCOUNT > 0
    COMMIT TRANSACTION;

You now can create your own error handling, whether in a workflow or use the SSIS logging to monitor and log your errors, like I do.

Categories: SQL Server, SSIS Tags:

SSIS Logging when callig sub packages

April 17th, 2009 No comments

I have one parent package which calls one sub package with the execute package task. I put on OnError logging to a database on both packages but it seems the parent package logs alle events in the sub package as well, so disable the logging on the sub package or you get the message twice which will give a rather strange view.

Categories: BI Technical, SSIS Tags: ,

Using the Merge statement in SSIS

April 15th, 2009 No comments

Today i had to create a package where i had to create a flat aggregation table with facts and measures for a operational system, the problem is that the data keeps refreshing every minute whereas the data is aggregate over an hour. That means data has to be updated or inserted.

My initial thought was to use SSIS to retrieve all of the rows intended for the dimension table in a data flow via a source component. Then the rows would be run through a lookup component to determine if they existed in the target table. Rows to be updated (those found in the lookup) would be run through an OLE DB Command component. Rows to be inserted would be sent to an OLE DB Destination.

But then I remembered the new SQL 2008 Merge statement which will give you the ability to write one single statement to update the row if it already exists otherwise it will insert a new row. Too bad there isn’t a Merge destination component in SSIS.

To use the merge statement I created a simple package with 2 SQL Execute tasks, one that inserts the new data in a temporary table and one that merges the data from the temporary table to the new aggregation table. I used the measure values as JOIN condition for the merge. Realy simple and not much work.

The merge statement is a great tool to keep the flow of update / insert really simple and without different timelines.

Sites i got my info from:
http://agilebi.com/cs/blogs/jwelch/archive/2007/07/05/sql-server-2008-using-merge-from-ssis.aspx
http://technet.microsoft.com/en-gb/library/bb522522.aspx
http://sqltips.wordpress.com/2007/08/27/merge-statement-of-sql-seerver-2008/

Categories: BI Technical, SQL Server, SSIS Tags: ,

SQL Server 2008 BI and Kerberos

April 14th, 2009 No comments

Chris web  just found  a very detailed white paper from Boyan Penev and Mark Dasco about setting up Kerberos for Microsoft BI which he and I thought was worth linking to for future reference:
http://bp-msbi.blogspot.com/2009/04/enhanced-security-and-integration-of.html

SSIS Continue the For Loop while having errors in package

April 14th, 2009 No comments

I have created a package with a for loop that runs for a few hours and process incoming files , but when a file is wrong and the processing gives an Error I want the for loop to continue with the next package but still log the error. To do this you have to play around with the Propagate variable. The Propagate variable controls whether the event is raised to the next container.

I have created 2 separate packages, one for the file watching and one for processing, the process package is executed from a execute package task, on this task I have created a OnError eventhandler and in this eventhandler set the propagate error to false.

This means when the package throws an error it will follow the Error constraint to enable you to for example send a mail or move the file but it won’t bubble the Error to the superseding task allowing the package to keep running.

Read more at: http://social.msdn.microsoft.com/Forums/pt-BR/sqlintegrationservices/thread/a8c841bf-8899-415f-a91f-1a4b17db6917 and http://agilebi.com/cs/blogs/jwelch/archive/2008/01/15/handling-multiple-errors-in-ssis-revisited.aspx

Categories: BI Technical, Coding, SSIS Tags:

Converting String values to a DATETIME datatype in SSIS expressions

April 8th, 2009 No comments

While transforming a flat file to a strongtyped SSIS dataset I needed a DATETIME value to use in my query later on. Too bad i only have a date and a time string value in my derived column component. I was expecting a solid date time format function to create a DATETIME datatype (like VB’s formatdatetime of C# DateTime.Parse ), too bad none of that is available. You have to do it with straightforward substring, this is the monster is eventualy created:

(DT_DBTIMESTAMP)(SUBSTRING(datum,1,4) + “-” + SUBSTRING(date,5,2) + “-” + SUBSTRING(date,7,2) + ” ” + SUBSTRING(time,1,2) + “:” + SUBSTRING(time,3,2) + “:” + SUBSTRING(time,5,2))

I hope they will create some more solid conversion functions in the future.

Categories: BI Technical, SSIS Tags: ,

SSIS: We Loaded 1TB in 30 Minutes with SSIS, and So Can You

March 4th, 2009 No comments

Microsoft has released a whitepaper about high performance ETL:

In February 2008, Microsoft announced a record-breaking data load using Microsoft® SQL Server® Integration Services (SSIS): 1 TB of data in less than 30 minutes. That data load, using SQL Server Integration Services, was 30% faster than the previous best time using a commercial ETL tool. This paper outlines what it took: the software, hardware, and configuration used. We will describe what we did to achieve that result, and offer suggestions for how to relate these techniques to typical scenarios. Even for customers who don’t have needs quite like this benchmark, such efforts can teach a lot about getting optimal performance.

Read the entire post from MS here.

Categories: BI Technical Tags: ,