Tuesday, May 31, 2022

Log T-SQL processing events

I frequently see questions about the best way to log events in T-SQL processing, including success and failure messages.  This method has the following advantages:

  • Minimal code needs to be added to your procedures.
  • Can log both informational status messages and error messages.
  • Events go into a log table, and are displayed in the SSMS messages pane, and if it has an error it “throws” the error to the calling program with error detail info.
  • Error messages include the following:
    • Date and time
    • Application that is running the procedure
    • User that is running the procedure
    • Server name
    • Database name
    • Schema name
    • Procedure name (this is the calling procedure where the even happened)
    • Message to record, e.g. “Start of MyProc”, “Insert sales”
    • Line number of the error
    • Error Message
    • Error number
    • Error severity
    • Error state
    • (optionally) the SQL command causing the error

Start by creating the following table to log events:

CREATE TABLE DW.EventLog (
id            int IDENTITY(1,1) NOT NULL,
DateLogged    datetime NOT NULL DEFAULT (GETDATE()),
AppName       nvarchar(240),
UserName      nvarchar(128) DEFAULT (USER_NAME()),
ServerName    nvarchar(240),
DBName        nvarchar(240),
SchemaName    nvarchar(128),
ProcName      nvarchar(128),
[Message]     varchar(240),
ErrorLine     int,
ErrorMessage  nvarchar(4000),
ErrorNumber   int,
ErrorSeverity int,
ErrorState    int
)

Create the following procedure to log the event:

--Procedure to log error and success information.
--Call this using: EXEC DW.LogEvent 'Description of next step', @@PROCID
ALTER PROCEDURE [DW].[LogEvent] @Message VARCHAR(240) = '', @ProcID int = NULL
AS  
BEGIN

DECLARE 
  @AppName  		NVARCHAR(128)
, @UserName 		NVARCHAR(128)
, @ServerName		NVARCHAR(128)
, @DBName   		NVARCHAR(128)
, @SchemaName		NVARCHAR(128)
, @ProcName 		NVARCHAR(128)
, @ErrorLine		INT
, @ErrorMessage		NVARCHAR(4000)
, @ErrorNumber		INT
, @ErrorSeverity	INT
, @ErrorState		INT
, @FullMessage		NVARCHAR(max);

SET @AppName		= APP_NAME()
SET @UserName		= USER_NAME()
SET @ServerName		= @@SERVERNAME
SET @DBName 		= DB_NAME()
SET @SchemaName		= COALESCE(LEFT(ERROR_PROCEDURE(),CHARINDEX('.',ERROR_PROCEDURE())-1), OBJECT_SCHEMA_NAME(@ProcID), OBJECT_SCHEMA_NAME(@@PROCID))
SET @ProcName		= COALESCE(SUBSTRING(ERROR_PROCEDURE(),CHARINDEX('.',ERROR_PROCEDURE())+1,128), OBJECT_NAME(@ProcID), OBJECT_NAME(@@PROCID))
SET @ErrorLine		= ERROR_LINE()
SET @ErrorMessage	= ERROR_MESSAGE()
SET @ErrorNumber	= ERROR_NUMBER()
SET @ErrorSeverity	= ERROR_SEVERITY()
SET @ErrorState		= ERROR_STATE()

--Log the error.
INSERT INTO [DW].[EventLog](
AppName,
ServerName,
DBName,
SchemaName,
ProcName,
[Message],
ErrorLine, 
ErrorMessage, 
ErrorNumber,  
ErrorSeverity, 
ErrorState
) 
SELECT
@AppName,
@ServerName,
@DBName,
@SchemaName,
@ProcName,
@Message,
@ErrorLine,  
@ErrorMessage,  
@ErrorNumber,
@ErrorSeverity,  
@ErrorState;

SET @FullMessage = CONCAT
( '['	, @ServerName
, '].['	, @DBName
, '].['	, @SchemaName
, '].['	, @ProcName
, ']: '	, @Message
);

IF @ErrorNumber IS NOT NULL
SET @FullMessage = LEFT(CONCAT
( @FullMessage
, '. ErrorLine: '      , TRIM(STR(@ErrorLine))
, '. ErrorMessage: '   , @ErrorMessage
, '. ErrorNumber: '    , TRIM(STR(@ErrorNumber))
, '. ErrorSeverity: '  , TRIM(STR(@ErrorSeverity))
, '. ErrorState: '     , TRIM(STR(@ErrorState))
, '. Application: '    , @AppName
, '. User: '           , @UserName
), 2048);

--Display the message.
PRINT @FullMessage;

--If this is an error then throw the error to the calling application, otherwise return to calling application.
IF @ErrorNumber IS NOT NULL
	THROW 50000, @FullMessage, @ErrorState;
ELSE
	RETURN

END

Here is an example of a procedure that is modified to log these events.  The changes you need to make to your code are:

  • Add “DECLARE @step VARCHAR(240);
  • Add the TRY/CATCH.  Note: the @@PROCID is needed to accurately record the name of the current procedure.
  • Add the “EXEC DW.LogEvent…” lines to announce information that you want recorded.
  • Add “SET @step = 'what is next'” prior to each SQL command to describe where it failed.

CREATE PROCEDURE [DW].[test_error]
AS
BEGIN TRY
	DECLARE @step VARCHAR(240);

	EXEC DW.LogEvent 'Starting error test', @@PROCID;	--announce start of run


	--Body of the procedure follows
	SET @step = 'second step'	--describe each coming step
	SELECT 1/0


	EXEC DW.LogEvent 'Error test completed', @@PROCID;	--announce end of run

	RETURN

END TRY
BEGIN CATCH
	EXEC DW.LogEvent @step, @@PROCID;
END CATCH

The only thing missing is the actual text of the SQL command that caused the error. If that is needed you can execute your commands using the following procedure:

CREATE PROCEDURE [DW].[ExecuteSQL] @SQL NVARCHAR(max), @ProcID int = NULL
AS
--Procedure to execute a sql command and log the error and the offending sql text.
--Call this using: EXEC DW.ExecuteSQL 'SQL Command here', @@PROCID
BEGIN
	BEGIN TRY  
		DECLARE @SqlCommand	NVARCHAR(240)
		SET @SqlCommand = LEFT(CONCAT('SQL: ', @SQL), 240)
		--execute the sql command
		EXECUTE sp_executesql @SQL
	END TRY  
	BEGIN CATCH  
		PRINT 'ERROR WITH THE FOLLOWING COMMAND:'
		PRINT @SQL
		-- Log the error.  
		EXECUTE DW.LogEvent @SqlCommand, @ProcID;
	END CATCH;
END;

Wednesday, May 11, 2022

Introducing the Unified Star Schema

A few months after my prior blog post on the OneBigTableapproach I learned of a new way to design Data Warehouses that is somewhat similar.  It is a new way to design star schemas and is known as the Unified Star Schema (USS).  It is recommended by the grandfather of Data Warehousing Bill Inmon in a book that just came out in the past year.  Once I heard about it I immediately went out and bought The Unified Star Schema: An Agile and Resilient Approach to Data Warehouse and Analytics Design, by Bill Inmon and Francesco Puppini and read & underlined all 288 pages.  I agree this new approach is better than the Kimball multi-star approach.  USS is the best approach for building a ROLAP data warehouse using row-oriented tables.  My OneBigTable approach is the best way to build a data warehouse for a columnar share-nothing hyper-converged DBMS. (The book portrays the OBT in passing in Figure 14.15.)

The core of the approach is the construction of a single “Puppini Bridge” table that works like a single super-fact table of only keys and transforms all other tables to fact/dimensions that users can blend as they see fit.  Actually, the distinction between fact and dimension is gone.  The Puppini Bridge table has no attributes and no metrics, only primary & foreign keys.  It joins to all other tables like a star schema with no snowflaking.  It is built quickly by performing a UNION ALL of the primary key (PK) & foreign keys (FK) of every table. This transforms an OLTP normalized database into a performant OLAP star data warehouse by the addition of just one table and no other ETL.

USS works well with BI tools that use “associations” in relationships to solve the “Fan Trap”, such as Microsoft’s Power BI, QlikView, and Tableau (starting with version 2020.2), but not so well with BI tools that use SQL JOIN, like Tibco Spotfire and Business Objects.  (A Fan trap is when metrics in a parent table gets duplicated against the child tables causing incorrect aggregations.)

The classic star schema approach causes a problem known as the “chasm trap” when two or more fact tables are used in a BI tool.  The display of data from multiple fact tables results in a cartesian join duplication.  The Puppini Bridge  & OneBigTable overcomes this situation, letting the details show in a logical list of one fact followed by the other fact.

The book left out one point that I think is essential to building the Puppini Bridge:  One should include not only each record’s PK & FKs but also the FKs to all ancestor tables (parent of parent, etc.).

I also learned an important point when building the OneBigTable:  attributes should be duplicated down but metrics should only show on the record that provided the PK.

Most exciting, I believe USS allows the rapid generation of code that creates the solution if the referential constraints are all defined in the database.  This also makes the solution more maintainable and robust.  I plan to share this technique in a future post.

BTW, a company named Incorta pitches a data warehouse that they say is the “Death of the Star Schema.”  The USS book mentions it in Figure 14.6 and it appears to have nothing in common with USS or OBT.

Here are some other reviews:

CONCLUSION:  USS & OBT is better than classic star design in usability, maintenance and development time.  Muti-star solutions should be replaced with USS in row-oriented databases and OBT should be considered for MPP columnar databases like AWS Redshift & Azure Synapse.

At this point no one has a valid reason to stick with classic star schemas.