Friday, December 16, 2016

Converting UTC to local time

Problem

Values in the database are stored as UTC date and time in a simple datetime format. You want to see data time values in the local time.

Solution


Use the following code to convert UTC date/time to local:

CREATE TABLE #table (date_UTC datetime)

/* Populate with UTC date time */
Insert #table Select GETUTCDATE()

Select
/* UTC value */
date_UTC
/* convert UTC to local time */
,DATEADD(MILLISECOND,DATEDIFF(MILLISECOND,getutcdate(),GETDATE()),date_UTC) as date_In_LocalTime
From
table

Thursday, December 15, 2016

Use sp_replrestart to fix “The process could not execute ‘sp_repldone/sp_replcounters’” error

This error is generally raised by the log reader agent while running transactional replications. The error indicates that distributor and subscribe databases has data that may be more recent than data in the publisher database (the agent compares latest LSN in transaction logs of Publisher database and latest LSN recorded in the distributer).

This problem usually occurs after publisher database has been restored from a backup. 
To resolve the issue, run sp_replrestart stored procedure in the publisher database and then reinitialize the subscribers. Also, consider running DBCC CHECKDB on publisher database to validate consistency.

Monday, November 21, 2016

How to renew “MCSE:Business Intelligence” credentials

The other day a colleague of mine asked if I am preparing for a recertification exam for my “MCSE: Business Intelligence” certificate. His certificate was close to the expiration date and he wanted to see what resources I will be using.

That caught me by complete surprise! I totally forgot about the 3 year status of that certification. I quickly checking my profile at http://mcp.microsoft.com and there it was - my MCSE certificate was no longer active.

It turns out that Microsoft has tried to warn me, but I didn’t get those emails as they were going to an email account I don’t actively use.

At first I thought that since I missed the deadline my status has been cancelled and I would need to re-take all original exams in order to get it back. However, I decided to email Microsoft Support (certquest@microsoft.com) first and ask them about my options. Next day I received a response from them saying that my status hasn’t actually been cancelled, but was rather deactivated and that it will be re-activated as soon as I reconfirm my status by completing one of the two possible options:
  • Option 1: Take re-certification exam 470
  • Option 2: Recertify through Microsoft Virtual Academy
The second option required that I review and complete assessments for the following 9 MS Virtual Academy courses:
  1. Database Fundamentals
  2. Design and Implement Cloud Data Platform Solutions
  3. Data Storage and Processing in the Cloud Demystified
  4. Querying with Transact-SQL
  5. Faster Insights to Data with Power BI Jump Start
  6. Implementing a Data Warehouse with SQL Server Jump Start
  7. Updating Your Database Management Skills to SQL Server 2014
  8. Implementing Data Models & Reports with Microsoft SQL Server
  9. Designing BI Solutions with Microsoft SQL Server

I chose the second option because I thought it would be a great refresher and also offered convenient free online access. It took me about 1 week to watch the videos and complete quick assessment. I really enjoyed the presentation and material (yes, you can skip forward on the videos). 

Once finished I emailed my certificate of completion to certquest@microsoft.com with a request to reactivate my status which was done 2 days after.

For details on Microsoft Virtual Academy see the following link: https://www.microsoft.com/en-us/learning/recertification-virtual-academy.aspx

Wednesday, January 22, 2014

Data-Driven Subscriptions in SQL Server

I recently had a project that used data-driven subscriptions for bulk processing of SSRS reports. Subscriptions is a very powerful tool, but it lacks centralized monitoring tools, so I had to dig deep into the content database where SSRS service maintains all processing data. By default, this database is called ReportServer.

List of reports published to SSRS service is stored in Catalog table where each report is assigned an ItemID. Report can be associated with multiple subscriptions. Information about each individual subscription is stored in Subscriptions table. Here is a query that retrieves a list of reports with their corresponding subscriptions:

Select
       c.ItemID,
       c.Name,
       s.*
From
       [dbo].[Catalog] c
       Inner Join [dbo].[Subscriptions] s On s.Report_OID = c.ItemID

Subscription has many parameters defined by the user when subscription is created and stored in various text columns of Subscription table in the form of XML.

For example, ExtensionsSettings column stores information about rendering format, destination and file name settings, etc. It can be accessed by converting XML data into a record set and running XML query functions on it such as:

;WITH x AS (Select SubscriptionID, CAST(ExtensionSettings as xml) as ExtensionSettingsXML From [dbo].[Subscriptions])
Select
       s.SubscriptionID
       ,ExtensionSettingsXML
       ,ExtensionSettingsXML.v.value ('Name[1]', 'varchar(100)') as ParamValue
       ,ExtensionSettingsXML.v.value ('Field[1]', 'varchar(100)') as ParamValue
       ,ExtensionSettingsXML.v.value ('Value[1]', 'varchar(100)') as ParamValue
From
       [dbo].[Subscriptions] s
       Inner Join x On x.SubscriptionID = s.SubscriptionID
       CROSS APPLY ExtensionSettingsXML.nodes ('/ParameterValues/ParameterValue') as ExtensionSettingsXML(v)

When Subscription is initiated, it creates a record in ActiveSubscriptions table for each executing instance. Once processed, this record is removed.

Select * From [dbo].[ActiveSubscriptions]

Data-Driven Subscription works by generating a dataset (by executing a query) that provides values for report input, mapping those values to report parameters and queuing report execution. You can also use input data set attributes to specify file name for report rendering, control location where files are saved, and other parameters.
Table Events serves a role of execution queue where a record is created for each actual instance of the report with set parameters. For example, if your input data set contains 5000 records, Events table will have 5000 records one for each instance of report execution. Records are deleted from Events table as soon as report execution is complete.

Select * From [dbo].[Event]

With all that information in mind, you can create a SSRS report that queries ReportServer database and reports status of all active subscriptions. For example, you can use the following query to get this information:

Select
       s.SubscriptionID,
       'ReportName' = c.Name,
       'ReportPath' = c.Path,
       'SubscriptionDesc' = s.Description,
       'SubscriptionOwner' = us.UserName,
       'LatestStatus' = s.LastStatus,
       'LastRun' = s.LastRunTime,
       asub.ActiveID,
       asub.TotalNotifications,
       asub.TotalSuccesses,
       asub.TotalFailures,
       n.RemaininReports
From
       Subscriptions s
       join Catalog c on c.ItemID = s.Report_OID
       join ReportSchedule rs on rs.SubscriptionID = s.SubscriptionID
       join Users uc on uc.UserID = c.ModifiedByID
       join Users us on us.UserID = s.OwnerId
       Left Join ActiveSubscriptions asub On asub.SubscriptionID = s.SubscriptionID
       Left Join
              (
                     Select
                           SubscriptionID,
                           ActivationID,
                           count(*) as RemaininReports
                     From
                           Notifications n
                     Group by
                           SubscriptionID, ActivationID
              ) n On asub.ActiveID = n.ActivationID

Base on this users will be able to monitor existing subscriptions and see their last status as well as monitor active subscriptions and their progress.

Friday, March 8, 2013

Capturing user activity in the database using SQL Server Auditing functionality


SQL Server comes with a build in audit functionality that saves a lot of development effort when one of the business requirements states is to keep a record of user activity in the database.

For each query executed in the database, SQL Server Audit captures the names of the affected tables, SQL statement used for query, whether query execution was successful and whether or not user had permission to access particular table.

The deployment is very easy and straight forward. It starts from defining new SERVER AUDIT object:

USE [master]
GO
CREATE SERVER AUDIT [ServerAuditTest]
TO FILE
(      FILEPATH = N'C:\MSSQL\Data\'
,MAXSIZE = 10MB
,MAX_ROLLOVER_FILES = 10
,RESERVE_DISK_SPACE = OFF
)
WITH
 (     QUEUE_DELAY = 1000
,ON_FAILURE = CONTINUE
)

Script defines storage mode for the log data (file or application log), max size of audit files, max number of files and whether SQL should function if audit fails to start. Once created, audit object must be enabled by executing the following script:


ALTER SERVER AUDIT [ServerAuditTest]
WITH (STATE = ON);

Database audit specification object is attached to the audit object created earlier. It defines a combination of actions (events), securables (database objects) and principles (users or roles) that should be audited.

The following script defines audit for all users that belong to “DatabaseRoleWithAudit” database role executing Delete, Insert, Select, Update or Execute commands against any objects owned by dbo schema:

USE [DatabaseName]
GO

CREATE DATABASE AUDIT SPECIFICATION [DatabaseAuditTest]
FOR SERVER AUDIT [ServerAuditTest]
ADD (DELETE ON SCHEMA::[dbo] BY [DatabaseRoleWithAudit]),
       ADD (EXECUTE ON SCHEMA::[dbo] BY [DatabaseRoleWithAudit]),
       ADD (INSERT ON SCHEMA::[dbo] BY [DatabaseRoleWithAudit]),
       ADD (SELECT ON SCHEMA::[dbo] BY [DatabaseRoleWithAudit]),
       ADD (UPDATE ON SCHEMA::[dbo] BY [DatabaseRoleWithAudit])
WITH (STATE = ON)

If created with State=ON, audit starts working right away.

Querying data collected by audit is very simple. You can do it through SQL Server Management Studio by navigating to “Security \ Audit \ Audit Name”, clicking right mouse and selecting “View log” or by query audit logs in Query Analyser using the following command:

SELECT
      *
FROM
      sys.fn_get_audit_file(N'C:\MSSQL\Data\ServerAuditTest*.sqlaudit', null, null)

In the script above I defined that audit logs should be stored as rollover files, so it makes sense to add a SQL job that would query audit logs on a scheduled basis and move new records from log into a permanent table where it can be indexed for better query performance and made available to the users for analysis.

Tuesday, December 18, 2012

Vertical and horizontal lines or strips on Line Chart in SSRS


How do you show a vertical or horizontal line on a line chart in SSRS? After hours or research it seems that approach involving StripLines works the best.
Horizontal or vertical likes can be used on the chart to indicate baselines or highlight particular sections as on the sample below:


So, how do you build a chart like that? Start with the data set.
My sample dataset contains a simple query that returns: month (as date) for X axis, value for Y axis, baseline value for Y axis, baseline value for X axis (date). In order to properly plot the vertical baseline the date used as a baseline value for X axis must be converted to its integer representation. 

Here is the query:

Declare @table Table (id int identity(1,1), event_date datetime, event_value int)

Insert @table 
Select 'Jan 1, 2010', 200
Union Select 'Feb 1, 2010', 250
Union Select 'Mar 1, 2010', 300
Union Select 'Apr 1, 2010', 200
Union Select 'May 1, 2010', 150
Union Select 'Jun 1, 2010', 50
Union Select 'Jul 1, 2010', 300
Union Select 'Aug 1, 2010', 400
Union Select 'Sep 1, 2010', 200
Union Select 'Oct 1, 2010', 150
Union Select 'Nov 1, 2010', 100
Union Select 'Dec 1, 2010', 100

Declare @baseline_value int,
@baseline_date datetime

Set @baseline_value  = 250
Set @baseline_date = 'June 1, 2010'

Select 
event_date,
event_value,
@baseline_value as baseline_value,
@baseline_date as baseline_date,
CAST(@baseline_date as int) as baseline_date_int
From 
@table

The result of the query looks like this:


Create a new report and use above query as a data source. Add new Line Chart object, associate it with the data source. Select “event_value” column to be used for Values and “event_date” column will be used for Category Groups.

Modify Horizontal Axis properties as highlighted below:


Format axis label as "MMM yyyy".

Click on X axis and in the properties window find "StipLines" attribute. Click on "Collection" to open collection editor.

Modify attributes as highlighted below:


IntervalOffset value is set using formula "=CInt(Max(Fields!baseline_date_int.Value))".

Do the same steps for Y axis. IntervalOffset value is set using formula "=CInt(Max(Fields!baseline_value.Value))"

Preview the report and enjoy the result!

You can use the same technique to highlight areas on X or Y axis as below. Click here to download sample "Vertical and Horizontal Lines on line graph.rdl" for more details.






Wednesday, November 14, 2012

Scripting table data


Every now and then I need to create a SQL script to move table data between environments or to simply load table during database deployment process.

For a number of years I am using the stored procedure created by Narayana Vyas Kondreddi (http://vyaskn.tripod.com). This is a great tool for scripting data without any problems!
Attached script creates “sp_generate_inserts” stored procedure in master database (so it is accessable from any database on the server).

The description portion in the script contains the complete description of the parameters and usage examples.

You can get the script here:


This is what the output looks like: