# Thursday, October 01, 2009

SQL Server - Insert to table with ALL default values

If you're trying to do an insert to a table where every column has a default value (or is an identity column), then SQL Server will give you an error if you don't specify 'something' in the field list/values clause... Or so I thought...

This is so simple it's funny, but not so simple to find in the documentation:

INSERT INTO TableName DEFAULT VALUES



posted on Thursday, October 01, 2009 9:01:02 AM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]
# Monday, September 28, 2009

SQL Server Stored Procedure to disable / enable all foreign keys and constraints in a database

I've been doing a bit of batch archiving work, and needed a nice and quick way to disable foreign keys and other constraints without actually removing them.  I found a nice article on disabling all constraints in a database, and thought I'd just take it one step further by making it into a Stored Procedure, and adding a parameter to toggle whether the constraints are enabled or disabled.

Nice and easy.  Here's the script.


USE [YOURDBNAME]
GO

IF  EXISTS 
(SELECT FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[usp_SetDBTableConstraints]'AND type in (N'P'N'PC'))
DROP PROCEDURE dbo.usp_SetDBTableConstraints
GO

CREATE PROCEDURE 
dbo.usp_SetDBTableConstraints(@Enable BIT)

AS

/*** DISABLE/ENABLE ALL TABLE CONSTRAINTS ************************

This script will disable all constraints on all tables within the database
that it is run in.

************************************************************/

SET NOCOUNT ON
SET ROWCOUNT 


DECLARE @Count INT
DECLARE 
@String NVARCHAR (1000)
DECLARE @ConstraintName VARCHAR(128)
DECLARE @TableName VARCHAR(128)

--Find all constraints and their respective tables from the sysobjects table and place into a temp table.
--Primary Key and Unique Constraints via Unique Indexes are not disabled through this command
--You should use the ALTER INDEX...DISABLE command in SQL Server 2005
SELECT 
        name                     AS 
constraintname,
        object_name(parent_obj)  
AS tablename 
INTO #Const_Table
FROM sysobjects s 
where xtype in ('F')

SELECT @Count Count(*) FROM #Const_Table

--Setting the rowcount to one allows for one row from the temp table to be picked off at a time.
--Used as an alternative to a cursor.
SET ROWCOUNT 1

--Loop until all rows in temp table have been processed.
WHILE @Count > 0
BEGIN
    
--The rowcount of one ensures that only one tablename and constraint name is picked.
    
SELECT @TableName TableName, @ConstraintName ConstraintName
    
FROM #Const_Table

    
--Build execution string to disable constraint.
    
IF @Enable 
        
SET @String 'ALTER TABLE ['+ @tablename + '] CHECK CONSTRAINT [' + @constraintname +']'
    
ELSE
        SET 
@String 'ALTER TABLE ['+ @tablename + '] NOCHECK CONSTRAINT [' + @constraintname +']'

    
--Execute the SQL
    
EXEC sp_executesql @string

    
--Remove this row from the temp table, since it has now been processed.
    
DELETE FROM #Const_Table WHERE ConstraintName @ConstraintName and TableName @TableName

    
SET @Count @Count - 1
END -- Loop

SET ROWCOUNT 0

DROP TABLE #Const_Table
 

GO
posted on Monday, September 28, 2009 12:34:45 PM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]
# Wednesday, May 27, 2009

HOW TO Use SQL Server DTS Transform Data Task to export to dynamic Excel spreadsheet file

Many people (including me) are often still using SQL 2000 - which means that you might be using DTS (Data Transformation Services) to perform utility jobs on your databases. 
I've had a requirement to output data to Excel (not my choice - Excel usually isn't!) - which is subtly different to text output as CSV*.  Another requirement was to have a 'generated' file name in a known folder.  This is as simple as a date/time driven file name.  The 'job' runs once (or more) a day and so creates a new file each time.  Easy right?

Well - it turns out that if you go through the normal motions of creating an Excel connection and using a Transform Data Task it's likely to work first time and leave you thinking 'great'.  Unfortunately when you run it again you're likely to get a message similar to the following:

Table 'New_Table$' not found. 

After much digging (hence the reason for writing this), I discovered that in order to map the columns successfully you basically have to create the table in the spreadsheet before running the transformation (the 'create table' part of creating the transformation).  If you've got a dynamic file name then the next time you'll  effectively have a blank spreadsheet without the required receiving 'table', and it will fail, unless you've recreated the sheet before the transformation.

OK so assuming you want to output some data to a spreadsheet with a dynamic filename with a Transform Data task then here's how you do it.

  1. I put all files in the same folder so I use 2 global variables to build the filename - NetworkLocation and FinalLocation
    1. First, Right-click on a blank part of your package screen and choose Package Properties, then Global Variables tab.
    2. Create the two global variables NetworkLocation and FinalLocation
    3. Edit the NetworkLocation value to be the folder (including a final '\') of where you want to store the files.
    4. Leave FinalLocation blank (we'll create that in the next step.
  2. Add an ActiveX Script Task with the following code (which you can obviously tailor the name to your needs).  It basically creates a date/time based file name, and stores a concatenation of the network location and file name in the FinalLocation global variable.

    '**********************************************************************
    '  Visual Basic ActiveX Script
    '************************************************************************
    Function Main()

        Dim fileName
        fileName = "MyOutput_" & DatePart("yyyy",Date) & Right("0" & DatePart("m",Date), 2) & Right("0" & DatePart("d",Date), 2) & "_" & Right("0" & DatePart("h",Now), 2) & Right("0" & DatePart("n",Now), 2) & Right("0" & DatePart("s",Now), 2)
        DTSGlobalVariables("FinalLocation").Value = DTSGlobalVariables("NetworkLocation").Value + fileName + ".xls"

        Main = DTSTaskExecResult_Success

    End Function
  3. Add your data connections
    1. Create your source connection (presumably SQL Server or wherever your data's coming from)
    2. Create your destination connection (A Microsoft Excel 97-2000 connection) and set an initial file name (best to be in the folder specified in NetworkLocation)
  4. Create a Data Transformation Task connecting your source and destination connections as per normal
    1. Add in your Table/View or SQL Query to define the source data.
    2. In the Destination tab click 'Create...' next to 'Table name:' (a dialog will pop up 'Create Destination Table').  Change the table name (or column definitions) if you like, then COPY THE ENTIRE TEXT - and paste into a text editor for safe keeping.
    3. Click OK (This will create a new sheet in the spreadsheet you've specified in your connection, ready to receive your data).  Your fields should then also be populated in the table below.
    4. Check the transformations tab - you should get a nice one to one mapping between source and destination columns.
  5. Create a Dynamic Properties Task (Here's where it all comes together)
    1. Click 'Add...' to create a new property.  Choose your Excel Connection and pick the 'DataSource' property, then click 'Set...'
    2. Choose 'Global Variable' as the source and FinalLocation as the variable - Click OK.
    3. Click 'Add...' to create another property.  Choose the DTSTask_DTSDataPumpTask_? in the Tasks node (your data transformation task) and pick the 'DestinationObjectName' property.
    4. Choose 'Query' as the source and paste in your 'CREATE TABLE' statement from earlier - Parse to check, then click OK.
  6. Create the Workflow for the task
    1. Select the Dynamic Properties Task, right-click and select Workflow --> Workflow Properties.
    2. Click 'New' and set the ActiveX Script Task as 'Source Step'.
    3. Select the Data Transformation Task, right-click and choose Workflow Properties.
    4. Click 'New' and set the Dynamic Properties Task as 'Source Step'.
That's basically it - the package should look something like this:



It should also now do the following (in order):
  1. Set filenames (stored in global variables)
  2. Set filename of the excel connection, create the destination 'sheet' from the global variables and the query in the transformation
  3. Do the transformation and store the results in a date/time named Excel spreadsheet in your configured folder

If you ever need to change the query (columns) then you'll need to regenerate the 'CREATE TABLE' statement and reflect it in the Dynamic Properties Task (5.c)


*Typically an Excel export covers your back with datatypes where a CSV export won't - so when you open in Excel you'll have to format columns specifically yourself - like converting dates, and also long numbers from scientific format to text format.
posted on Wednesday, May 27, 2009 9:34:47 AM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]
# Wednesday, December 31, 2008

Using local SQL 2005 Express databases, |DataDirectory| setting and SQLBulkCopy with SemiColon delimited text files

Had some fun with this today as I was utterly confused why I wasn't able to save data to my SQL2005 express DB (linked within a Windows Forms VS 2005 project), even though I wasn't getting any errors from a SQLBulkCopy operation.  The connection string was set (from the wizard) to use '|DataDirectory|\dbname.mdf.

It turned out of course that the reason there wasn't an error was because the data was being written fine, but just not to the database I was looking at.  By default any database linked in your project file will be copied to the output folder (bin\debug) when you build.  It does this 'every time'.  I thought I was going mad as I wasn't getting primary key violations trying to insert the same data over and over again!

The useful info is in this MSDN blog entry.  It talks about different options for working around the local databases, but I plumped for a subfolder (called data) in my project, then setting 'Copy if newer' on the MDF file rather than 'do not copy'.  This just means that if you make a schema change then it will blow everything away (probably what I normally want in debug-land).

The reason I was using the SQLBulkCopy class was so I could take in a dataset picked from a semicolon-delimited text file. 
The first thing to do was read the text file into a dataset - and after some mucking about with the Jet OleDb provider and avoiding changing registry entries (to get it to recognise ';' as the delimiter) I found that a schema.ini file is what's required.

Here's some quick code I then used to get a DataSet from a semicolon-delimited Text File:

       
       /// <summary>
        /// Import from a given file to database
        /// </summary>
        /// <param name="fileName">File to import from (CSV)</param>
        
public void Import(string fileName)
        {
            
if (File.Exists(fileName))
            {
                SqlConnection connection 
= new SqlConnection(Properties.Settings.Default.RBStatsConnectionString);
                
//Get Dataset from file
                
DataSet importData GetDataSetFromFile(fileName);

                
//Import using SQLBulkCopy
                
SqlBulkCopy copyManager = new SqlBulkCopy(connection.ConnectionString, SqlBulkCopyOptions.UseInternalTransaction);
                
copyManager.DestinationTableName "TShirtStatistic";
                
copyManager.BatchSize importData.Tables[0].Rows.Count;
                try
                
{
                    copyManager.WriteToServer(importData.Tables[
0]);
                
}
                
catch (Exception exception)
                {
                    Console.WriteLine(exception)
;
                
}
                copyManager.Close()
;
                
connection.Close();
            
}


        }

        
public DataSet GetDataSetFromFile(string fileName)
        {
            
//Ensure we've got a schema file to fudge the semicolon delimited text
            
string schemaFile Path.Combine(Path.GetDirectoryName(fileName), "schema.ini");
            
File.Delete(schemaFile);
            string 
schemaContents "[" + Path.GetFileName(fileName) + "]\nColNameHeader=True\nCharacterSet=ANSI\nFormat=Delimited(;)";
            
StreamWriter writer File.CreateText(schemaFile);
            
writer.Write(schemaContents);
            
writer.Close();

            string 
connectionString "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + Path.GetDirectoryName(fileName) + "\\;Extended Properties='text;HDR=Yes;FMT=Delimited'";
            string 
query "SELECT * FROM " + Path.GetFileName(fileName);
            
OleDbConnection connection = new OleDbConnection(connectionString);
            
OleDbCommand command = new OleDbCommand(query, connection);

            
connection.Open();

            
OleDbDataAdapter adapter = new OleDbDataAdapter(query, connection);
            
DataSet ds = new DataSet();
            
adapter.Fill(ds);

            return 
ds;

        
}
posted on Wednesday, December 31, 2008 11:40:27 AM (AUS Eastern Daylight Time, UTC+11:00)  #    Comments [0]
# Wednesday, November 26, 2008

Create SQL Server User and Role Script to execute Stored Procedures Only

I really thought I'd posted this before, but here's a script I use quite often when I need to create a SQL user that needs to just have 'execute' access to Stored Procs. 

This is usually used for web apps where the server is going through as a single user.  I'd encourage anyone to ensure that you always start off with granting the minimum access required and only add more when actually needed.  If you've got different roles in the application (viewer, administrator etc) then create different roles, with a user for each.

The intention is to remove the need to give direct table access (as we only call Stored Procs right?).  The script doesn't actually do the 'grants' for you, but will generate another batch that you can execute if you like.  I prefer to put the grants in the script for each proc/function anyway, just so it's a little easier to maintain, and you can specifically grant access to the 'admin' procs to the 'admin' role, whilst limiting any 'normal' users to the read-only/everyday ones. 

There's satisfaction to be had from an app that connects to the database via an appropriate user, which is in a role that can only do what is necessary for the role, and can't access tables directly.  Not only is it neat, it's also (relatively) secure.

Here's the script...




-- MSIMNER: This script will generate a role and user for SQL Access to a single database
-- The role will have no specific access to read or write tables but will be granted access
-- through execution of stored procedures ONLY

USE <databasename, sysname, MyDB>

DECLARE @UserName VARCHAR(20)
DECLARE @Password VARCHAR(20)
DECLARE @DBName   VARCHAR(20)
DECLARE @Role     VARCHAR(20)

--SET PASSWORD HERE
SET @Password 'ixFg2tfk'

SET @UserName '<databasename, sysname, MyDB>User'
SET @Role     'db_exec_<databasename, sysname, MyDB>'
SET @DBName   '<databasename, sysname, MyDB>'

IF EXISTS(SELECT FROM master.dbo.syslogins WHERE name = @UserName)
BEGIN
    exec 
sp_revokedbaccess @name_in_db @UserName
    
exec sp_droplogin @loginame @UserName
    
exec sp_droprole @rolename @Role

END

--Create User
exec sp_addlogin @loginame @UserName, @passwd @Password
exec sp_grantdbaccess @loginame @UserName
exec sp_defaultdb @loginame @UserName, @defdb @DBName
exec sp_addrole @rolename @Role, @ownername 'dbo'
exec sp_addrolemember @rolename @Role, @membername @UserName


-- The output of this command will need to be run separately - i.e. paste the output into a new window and run... 

--SELECT 'GRANT EXECUTE ON ' + name + ' TO ' + @Role , *
--FROM sysobjects
--WHERE type = 'P'

Colorized by: CarlosAg.CodeColorizer
posted on Wednesday, November 26, 2008 10:10:59 AM (AUS Eastern Daylight Time, UTC+11:00)  #    Comments [0]
# Tuesday, July 22, 2008

Refactoring the inefficient data loop

OK. Last one for today.  I'm not going to go into too much detail except to say that the offending piece of code loads all items in a 'matrix' database table and builds a business object collection hierarchy.  This table has 4 fields, one or more of which may be NULL - for different meanings.  I really don't particularly like this type of approach but changing it right now would involve quite a bit of work.

The table looks something like

Level1ID
Level2aID
Level2bID
Level2cID

This ultimately translates to a business object hierarchy of:

 

  • Rows with Level1ID (only) - all other fields null - are Level1 objects
  • Rows with Level1ID and Level2a ID (only) are Level2a objects

etc...

 

The matrix table just contains the primary keys for each object so the code loads each one in the style below...

 

Load matrix
for each row in matrix
{
 if row type == Level1
     Add to Level1Collection
 else if row type == Level2a
 {
  Load Level2a object
  Add Level2a object to Level1[Level1ID].Level2aCollection
 }
 else if row type == Level2b
 {
  Load Level2b object
  Add Level2b object to Level1[Level1ID].Level2bCollection
 }
 else if row type == Level2c
 {
  Load Level2c object
  Add Level2c object to Level1[Level1ID].Level2cCollection
 }
}

 

This seems reasonable enough (logical anyway) given the way the data's being retrieved.

This does however load another several hundred rows from more stored proc calls that load each new object into the child collections.  This whole thing consistently takes around 8 seconds.

 

The Refactoring

 

A wise man once told me that if you can be sure the bulk of your operation is data access then lump it all together if you can, and do the loop or other grunt processing on the client. 

  1. I created a new Stored Procedure to return 4 result sets.  These come back as 4 tables in the target DataSet.  Each resultset is qualified by the same master criteria, and just uses joins to get a set of data that can be loaded directly into the collections.  The original stored proc is no longer required and this is now the only data access call.
  2. I changed my collection classes slightly to allow a LoadData method which takes in a dataset, a tablename and a parent key.  This means I can add Level2a objects to the appropriate Level1 collection.  The pseudo code now looks like...

Load multiple results
for each row in matrix
{
 if Level1 Results present 
   LoadData on Level1Collection
    
 if Level2a Results present
  for each Level1 row
   LoadData on Level1[Level1ID].Level2aCollection
   
 if Level2b Results present
  for each Level1 row
   LoadData on Level1[Level1ID].Level2bCollection

 if Level2c Results present
  for each Level1 row
   LoadData on Level1[Level1ID].LevelcbCollection
 
}

As I said at the beginning, there are some definite improvements to be made from changing the data structure, and a lot of this code could look a lot nicer by using Typed Datasets with relationships defined.

 

The new approach actually completes in less than 100ms.  I couldn't actually believe it myself, and made sure I killed connections, cache etc to make sure the database was coming in cold.  Still the same.

 

This basically proves that for data-heavy operations, things really start to hurt when you're making repeated client round-trips, however small the call.  This is basically a 99% saving in load time for this business object. 

 

The resulting page is also really snappy now and I'm sure the customer won't even notice :-)

 

posted on Tuesday, July 22, 2008 5:15:02 PM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]
# Tuesday, July 08, 2008

Search SQL Server Stored Procedures for any text string

Here's the SQL script I probably use most when I'm trying to work out dependencies and scope of development changes.  Query Analyzer allows you to search objects (hit F4 - it's cool!) and find objects by 'name', but there's no obvious way to search stored procs for a particular string.

SQL Server hides Stored Procedure code in the sysComments table (in each database), and so it's possible to search.  This is another script I found from not sure where (so can't really take credit).  It does a great job of scanning and even gives you line numbers.  It would be great to be able to just shove this into the master database, but you need to be in the context of the database you're searching in (or it would have to be amended somewhat to run some more 'dynamic' SQL). 

I have previously front-ended this for development databases with an ASP.NET page that simply runs the query substituting a parameter for the search text and connecting to the correct database.  The results show nicely in a grid.

Here's the code - just paste into a new query in your chosen database, change @SearchTerm and away you go...

 

SET NOCOUNT ON
GO

declare 
@dbname sysname
,@SearchTerm    varchar(100)
,@BlankSpaceAdded   
int
,@BasePos       int
,@CurrentPos    int
,@TextLength    int
,@LineId        int
,@AddOnLen      int
,@LFCR          int --lengths of line feed carriage return
,@DefinedLength int
,@obj_name    varchar(255)
,@prev_obj_name 
varchar(255)


/* NOTE: Length of @SyscomText is 4000 to replace the length of
** text column in syscomments.
** lengths on @Line, #CommentText Text column and
** value for @DefinedLength are all 255. These need to all have
** the same values. 255 was selected in order for the max length
** display using down level clients
*/
,@SyscomText    nvarchar(4000)
,@Line          
nvarchar(255)

set @SearchTerm 'MyTable'
set @dbname db_name()


set @SearchTerm '%' + @SearchTerm + '%'

Select @DefinedLength 255
SELECT @BlankSpaceAdded /*Keeps track of blank spaces at end of lines. Note Len function ignores
                             trailing blank spaces*/
CREATE TABLE #CommentText
(obj_name 
varchar(255)
 ,LineId    
int
 
,Text  nvarchar(255collate database_default)

/*
**  Make sure the @objname is local to the current database.
*/
if @dbname is not null and @dbname <> db_name()
        
begin
                raiserror
(15250,-1,-1)
        
end

    begin

        DECLARE 
ms_crs_syscom  CURSOR LOCAL
        FOR SELECT 
so.name, sc.text FROM syscomments sc
    
inner join sysobjects so
    
on so.id = sc.id
    where 
so.type 'P'
    
AND sc.encrypted 0
                
ORDER BY number, colid

        
FOR READ ONLY
    
end

/*
**  Get the text.
*/
SELECT @LFCR 2
SELECT @LineId 1


OPEN ms_crs_syscom

FETCH NEXT FROM ms_crs_syscom into @obj_name, @SyscomText

WHILE @@fetch_status >0
BEGIN

    SELECT  
@BasePos    1
    
SELECT  @CurrentPos 1
    
SELECT  @TextLength = LEN(@SyscomText)

    
IF @prev_obj_name <> @obj_name
        
--start again
        
SELECT @LineId 1
    
    
WHILE @CurrentPos  !0
    
BEGIN
        
--Looking for end of line followed by carriage return
        
SELECT @CurrentPos =   CHARINDEX(char(13)+char(10), @SyscomText, @BasePos)

        
--If carriage return found
        
IF @CurrentPos !0
        
BEGIN
            
/*If new value for @Lines length will be > then the
            **set length then insert current contents of @line
            **and proceed.
            */
            
While (isnull(LEN(@Line),0) + @BlankSpaceAdded + @CurrentPos-@BasePos + @LFCR) > @DefinedLength
            
BEGIN
                SELECT 
@AddOnLen @DefinedLength-(isnull(LEN(@Line),0) + @BlankSpaceAdded)
                
INSERT #CommentText VALUES
                
( @obj_name,
          @LineId,
                  
isnull(@Line, N'') + isnull(SUBSTRING(@SyscomText, @BasePos, @AddOnLen), N''))
                
SELECT @Line = NULL, @LineId @LineId + 1,
                       @BasePos 
@BasePos + @AddOnLen, @BlankSpaceAdded 0
            
END
            SELECT 
@Line    = isnull(@Line, N'') + isnull(SUBSTRING(@SyscomText, @BasePos, @CurrentPos-@BasePos + @LFCR), N'')
            
SELECT @BasePos @CurrentPos+2
            
INSERT #CommentText VALUES( @obj_name, @LineId, @Line )
            
SELECT @LineId @LineId + 1
            
SELECT @Line = NULL
        END
        ELSE
        
--else carriage return not found
        
BEGIN
            IF 
@BasePos <@TextLength
            
BEGIN
                
/*If new value for @Lines length will be > then the
                **defined length
                */
                
While (isnull(LEN(@Line),0) + @BlankSpaceAdded + @TextLength-@BasePos+1 ) > @DefinedLength
                
BEGIN
                    SELECT 
@AddOnLen @DefinedLength - (isnull(LEN(@Line),0) + @BlankSpaceAdded)
                    
INSERT #CommentText VALUES
                    
( @obj_name,
              @LineId,
                      
isnull(@Line, N'') + isnull(SUBSTRING(@SyscomText, @BasePos, @AddOnLen), N''))
                    
SELECT @Line = NULL, @LineId @LineId + 1,
                        @BasePos 
@BasePos + @AddOnLen, @BlankSpaceAdded 0
                
END
                SELECT 
@Line = isnull(@Line, N'') + isnull(SUBSTRING(@SyscomText, @BasePos, @TextLength-@BasePos+1 ), N'')
                
if LEN(@Line) < @DefinedLength and charindex(' ', @SyscomText, @TextLength+1 ) > 0
                
BEGIN
                    SELECT 
@Line @Line + ' ', @BlankSpaceAdded 1
                
END
            END
        END
    END
    
--Save previous name
    
SET @prev_obj_name @obj_name
    
FETCH NEXT FROM ms_crs_syscom into @obj_name, @SyscomText
END

IF 
@Line is NOT NULL
    INSERT #
CommentText VALUES( @obj_name, @LineId, @Line )

select from #CommentText 
where  patindex(@SearchTerm, text) <> 0

CLOSE  ms_crs_syscom
DEALLOCATE     ms_crs_syscom

DROP TABLE     #CommentText


Colorized by: CarlosAg.CodeColorizer
posted on Tuesday, July 08, 2008 9:38:31 AM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]

Generate SQL Inserts from existing data

This is one from the archives that obviously got away.  I had need for this today so thought I'd post so I don't have to look so hard next time!

It's pretty much a case of create the SP in your chosen database and run with a tablename as the parameter.  I originally found this on one of the SQL sites (probably SQLServerCentral), and amended it slightly to add some basic filtering in the @where parameter.

There's a few limitations from memory - with blob, image and text fields, but for your average tables this will probably do the trick just fine.

I've also used a different method that basically builds up a single INSERT with SELECT 'literal values' UNION SELECT 'literal values'.  That works fine and is potentially easier to control transaction-wise as everthing's in a single statement.  Anyway here's the code....

if exists (select from dbo.sysobjects where id = object_id(N'[dbo].[usp_InsertGenerator]'and OBJECTPROPERTY(idN'IsProcedure'1)
drop procedure [dbo].[usp_InsertGenerator]
GO


CREATE PROC 
dbo.usp_InsertGenerator
(@tableName 
varchar(100), @where varchar(1000= NULLas
 
--Declare a cursor to retrieve column specific information for the specified table
DECLARE cursCol CURSOR FAST_FORWARD FOR 
SELECT 
column_name,data_type FROM information_schema.columns WHERE table_name @tableName
OPEN cursCol
DECLARE @string nvarchar(3000--for storing the first half of INSERT statement
DECLARE @stringData nvarchar(3000--for storing the data (VALUES) related statement
DECLARE @dataType nvarchar(1000--data types returned for respective columns
SET @string='INSERT '+@tableName+'('
SET @stringData=''
 
DECLARE @colName nvarchar(50)
 
FETCH NEXT FROM cursCol INTO @colName,@dataType
 
IF @@fetch_status<>0
 
begin
 print 
'Table '+@tableName+' not found, processing skipped.'
 
close curscol
 
deallocate curscol
 
return
END
 
WHILE 
@@FETCH_STATUS=0
BEGIN
IF 
@dataType in ('varchar','char','nchar','nvarchar')
BEGIN
 SET 
@stringData=@stringData+''''+'''+isnull('''''+'''''+'+@colName+'+'''''+''''',''NULL'')+'',''+'
END
ELSE
if 
@dataType in ('text','ntext'--if the datatype is text or something else 
BEGIN
 SET 
@stringData=@stringData+'''''''''+isnull(cast('+@colName+' as varchar(2000)),'''')+'''''',''+'
END
ELSE
IF 
@dataType 'money' --because money doesn't get converted from varchar implicitly
BEGIN
 SET 
@stringData=@stringData+'''convert(money,''''''+isnull(cast('+@colName+' as varchar(200)),''0.0000'')+''''''),''+'
END
ELSE 
IF 
@dataType='datetime'
BEGIN
 SET 
@stringData=@stringData+'''convert(datetime,'+'''+isnull('''''+'''''+convert(varchar(200),'+@colName+',121)+'''''+''''',''NULL'')+'',121),''+'
END
ELSE 
IF 
@dataType='image' 
BEGIN
 SET 
@stringData=@stringData+'''''''''+isnull(cast(convert(varbinary,'+@colName+') as varchar(6)),''0'')+'''''',''+'
END
ELSE 
--presuming the data type is int,bit,numeric,decimal 
BEGIN
 SET 
@stringData=@stringData+''''+'''+isnull('''''+'''''+convert(varchar(200),'+@colName+')+'''''+''''',''NULL'')+'',''+'
END
 
SET 
@string=@string+@colName+','
 
FETCH NEXT FROM cursCol INTO @colName,@dataType
END
DECLARE 
@Query nvarchar(4000)
 
SET @query ='SELECT '''+substring(@string,0,len(@string)) + ') VALUES(''+ ' substring(@stringData,0,len(@stringData)-2)+'''+'')'' FROM '+@tableName
IF @where IS NOT NULL
 SET 
@query @query + ' where ' + @where
 
exec sp_executesql @query
 
CLOSE cursCol
DEALLOCATE cursCol
GO

Colorized by: CarlosAg.CodeColorizer
posted on Tuesday, July 08, 2008 9:18:47 AM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]
# Wednesday, March 26, 2008

Data Dictionary from within SQL Server

I knew you could do this, but never took the time to find out exactly which sys... table the comments got stored in.  sysProperties is what you want, and this article on SQLServerCentral tells you how...

The easiest way to get them 'in' is in Enterprise Manager, and someone posted a slightly amended version of the code to get them out (I also added IsNullable)...


DECLARE @TableName VARCHAR(128)

SET @TableName 'Catalogue'

SELECT 
    
T.table_name AS TableName,
    C.ordinal_position 
AS ColumnOrder,
    C.[column_name] 
AS ColumnName,
    
ISNULL(P.value, ''AS ColumnDescription,
    C.DATA_TYPE 
AS DataType,
    
COALESCE(numeric_precision,character_maximum_length,0AS Length,
    
COALESCE(numeric_scale,0AS DecimalPos,
    C.is_nullable 
AS IsNullable,
    
ISNULL(C.COLUMN_DEFAULT, ''AS DefaultValue  
    
FROM information_schema.tables T 
        
INNER JOIN information_schema.columns C 
            
ON T.table_name C.table_name 
        
LEFT JOIN ::FN_LISTEXTENDEDPROPERTY(N'MS_Description'N'user',N'dbo',N'table', @TableName, N'column'null) P 
            
ON C.column_name P.objname
    
WHERE T.table_name @TableName

Colorized by: CarlosAg.CodeColorizer
posted on Wednesday, March 26, 2008 8:46:43 AM (AUS Eastern Daylight Time, UTC+11:00)  #    Comments [0]
# Saturday, March 08, 2008

ASP.NET Open Source Content Management Systems

I'm producing a site for a client at the moment, and it quickly became apparent that I'd have a number requirements that would lead me down the CMS path:

  1. Client makes direct updates to the site (with approval/preview possibilities if possible)
  2. Built-in membership, security, roles etc (why write this yourself? - I've got 2 kids and just don't have the time or inclination!)
  3. Skinning capability (OK ASP.NET gives you that 'kindof' for free - but I throw it in anyway)
  4. CSS-driven layout (rather than tables) - This is a technical requirement for me rather than the client, but I want to make the site accessible and SEO-friendly
  5. Capability to customise  and add generic hooks to other office-based systems
  6. Quick time to market (building all the above from scratch would be 'fun' I'm sure, but would also take time)

To be fair to the customer I couldn't really suggest that I do everything from scratch when there's plenty of open source and commercial CMS's that can do the job very well. 

I do .NET - so I'm sure there's plenty of great CMS's for other platforms, but I'm not about to use this as a re-skilling exercise, charging a customer for the pleasure (as I need to do some more 'significant' development around the site, so will be using .NET for that).

What follows is a personal view based on my requirements and my experience (rather than extensive research into each product).

DotNetNuke

This was the first one I considered as I'd had dealings with it a few years ago (a dev 'community' site at a previous employer).  After playing with just about every module on the planet at the time ('cos you do), we realised we just needed a blog, so converted to Community Server as that seemed a bit less cluttered.  (BTW if you want to run .NET blog without SQL Server (like me) then DasBlog's what you want.

Pro's

  • Price (free)
  • Supports pretty much everything I need (open architecture, skinning, module development etc)
  • Familiar from previous use
  • Easy to use and administer - very WYSIWYG
  • Documentation's comprehensive (if a little focused towards the high-level architectural view). 

Con's

  • Slow (I thought I imagined this as my own dev machine's not a 'beast' and it's running SQL 2005 express, but part of this is potentially partly due to the caching mechanism (as noted on the MojoPortal site)
  • Table-driven by default (if you want your skin to be CSS-driven you've either got to muck around for ages finding a 'good' CSS skin (there's plenty of bad/really basic ones).  This extends to the DNN standard controls, and it takes an awfully long time (using Firebug and the IE Dev toolbar) to work out why you've still got fixed size fonts when you've gone to great length in your skin to make everything sizable.  (I'm sure again the better CSS skins address this, but you there always appears to be something that gets in the way because the original DD design uses tabular layout throughout).
  • It's written in VB.NET (OK this isn't a 'real' reason, as there's no difference - I know!  You also shouldn't need to do much (if anything) in VB.NET if you're a C#'er and you want to extend things - I think I'm just a bit miffed I've got to work with a bit of VB.NET at work at the moment :-) )
  • Not that keen on the actual logical structure, and the limited url-rewriting.  The whole anonymous tab thing out of the box.  It seems to me that DNN was designed before lots of newer features became available and it's had to bolt them on as time's gone by.
  • Lastly - it doesn't feel very open source.  It's clear that the business model has evolved to keep the output freely available whilst making money out of books, consulting and conferences (fair play to Shaun Walker - he deserves the kudos for the product).  I just feel that things went a little 'corporate' a while ago, which turned me off.  I bought an iRiver rather than an iPod for the same reason (too much marketing noise that turned me off, almost trying to bombard you so you don't even consider other products, and also trying to deter competition).  There's also many people now trying to make money out of skins (most of which aren't terribly good) - another indicator that the 'community' isn't very focused on open source development.

MojoPortal

Until a couple of days ago I'd not heard of this, but the performance discussion (above) sparked my interest.  It's clearly got less out-of-the-box functionality and community support than DNN and is lacking more in the documentation department, but if you're after a no fuss implementation then this might be the way to go.

Pro's (from the site - not my experience)

  • Free and open source
  • Multi DB support (just about any DB you'd care to use)
  • Approval/workflow support (although haven't looked into this yet)
  • Provides major functionality required
  • Skinning uses standard ASP.NET theme-based approach
  • Aligned relatively well with accessibility guidelines (CSS layout) - XHTML compliancy 

Con's

  • Download is actually a little confusing.  You've got a number of zip files and you're left to work out which one/s you need.  I downloaded a couple then reaslied I only needed the mssql release. 
  • Documentation a little lacking (although installation pretty easy once you've just changed a config setting). 
  • More of a 'developer' product - some understanding of .NET required.  This makes it more difficult for non-techy people to get it off the ground  

Jury's still out on this one as I've literally installed and that's it.  I need to get to grips with the integration and skinning support, as I want to change the default 3 pane layout and remove the left panel, so we'll see how that goes...

Others (That I don't know anything about)

Rainbow Portal (looks like it might be more worth a look in a year or two when there's a bit more support

Umbraco (not all versions free)

 

So what am I going to do now?  Install MojoPortal and see if it 'feels' nicer than DotNetNuke.  I'm not really too worried about future support, and upgrades as I'm providing a site to a client who will run with it once I'm done (based on what it does 'now' - and how stable it is 'now').

I'll post the results - or add to the list when I've come to a conclusion after some more research

posted on Saturday, March 08, 2008 3:46:56 PM (AUS Eastern Daylight Time, UTC+11:00)  #    Comments [0]
# Wednesday, July 25, 2007

SQL Server - Save all DTS Packages

Here's another useful proc I found to back up all your DTS packages (from I can't remember where).  Put this in your master database: 

Create procedure usp_SavePackages
@Path    
varchar(128
)
as
/*

*/

    
set nocount on

declare 
@objPackage 
int
declare 
@PackageName varchar(128
)
declare @rc 
int
declare 
@ServerName varchar(128
)
declare @FileName varchar(128
)
declare    @FilePath varchar(128
)
declare    @cmd varchar(2000
)
    
    
select     @ServerName 
@@ServerName ,
        @FilePath 
@Path
    
    
if right(@Path,1) <> 
'\'
    begin
        select @Path = @Path + '
\
'
    end
    
    -- create output directory - will fail if already exists but ...
    select    @cmd = '
mkdir 
' + @FilePath
    exec master..xp_cmdshell @cmd
    
    
create table #packages (PackageName varchar(128))
    insert     #packages
        (PackageName)
    select     distinct name
    from    msdb..sysdtspackages
    
    select    @PackageName = ''
    while @PackageName < (select max(PackageName) from #packages)
    begin
        select    @PackageName = min(PackageName) from #packages where PackageName > @PackageName

        select    @FileName = @FilePath + @PackageName + '
.dts
'

        exec @rc = sp_OACreate '
DTS.Package
', @objPackage output
        if @rc <> 0
        begin
            raiserror('
failed to create package rc %d
', 16, -1, @rc)
            return
        end

        exec @rc = sp_OAMethod @objPackage, '
LoadFromSQLServer
' , null,
            @ServerName = @ServerName, @Flags = 256, @PackageName = @PackageName
        if @rc <> 0
        begin
            raiserror('
failed to load package rc %d, package %s
', 16, -1, @rc, @PackageName)
            return
        end
        
        -- delete old file
        select @cmd = '
del 
' + @FileName
        exec master..xp_cmdshell @cmd, no_output
        
        exec @rc = sp_OAMethod @objPackage, '
SaveToStorageFile
', null, @FileName
        if @rc <> 0
        begin
            raiserror('
failed to save package rc %d, package %s
', 16, -1, @rc, @PackageName)
            return
        end
        
        exec @rc = sp_OADestroy @objPackage
    end

GO

Colorized by: CarlosAg.CodeColorizer

posted on Wednesday, July 25, 2007 9:05:39 AM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]
# Monday, July 23, 2007

Microsoft CRM - Data Migration Framework (DMF) Bugs / Fixes

I've been migrating data for a client recently and part of this has required the use of the MS Data Migration Framework (DMF).  I thought I'd post a couple of bugs I found and(thankfully) how I managed to get past them with the use of SQL Profiler!

The Migration Framework works - albeit slowly as it uses the CRM Web Service to stuff the records into the CRM database.  The CRM structure I've been trying to migrate to contains a large number of custom fields.  The DMF calls many stored procs to get its data (from the CDF database); these are structured by entity e.g

p_dm_GetContact
p_dm_GetContract

I was having a problem with Contacts, and after tracing a migration attempt with cut-down data (for speed), I found that the issue was that the lid was being blown off a VARCHAR(8000) in p_dm_GetContact giving the following SQL Error:

Error: 207, Severity: 16, State: 3

The error is 'Invalid Column Name'.  Further investigation revealed that a slightly dodgy piece of code (replicated in all the other similar procs) effectively truncates the dynamically created field list in the 'SELECT' statement...

-- if there are extended fields, then attached them to the end 
--   of the base field list

if len(@metadatafields) > 0
    set @entityfields = @entityfields + ', ' + rtrim(@metadatafields)

@entityfields contains a comma delimited list of fields from the 'cdf_Contact' table, and @metadatafields contains fields from 'cdf_Contact_Ext' (custom fields).  If you concatenate the two you're obviously asking for trouble.

The following shows the 'fix' to enable a bit more room (it's still not very good - but as the code's not available we can't make a nicer fix right now)

CREATE procedure p_dm_GetContact
as
begin
    set nocount on
    
    declare 
@entity            varchar(100
)
    
declare @entityfields     varchar(8000
)
    
declare @metadatafields varchar(8000
)
    
-- set target entity name
    
set @entity 
'Contact'

    
-- query base field list for targeted entity
    
exec p_dm_GetFields @entity, 'c', @entityfields 
output

    
-- query extended field list for targeted entity
    
exec p_dm_GetMetaDataFields @entity, 'ext', @metadatafields 
output

    
-- if there are fields, then attached them to the end 
        --   of the base field list    
    
if len(@entityfields) > 
0
       
set @entityfields ', ' rtrim
(@entityfields)

    
-- if there are extended fields, then attached them to the end 
        --   of the base field list
------    if len(@metadatafields) > 0
------        set @entityfields = @entityfields + ', ' + rtrim(@metadatafields)

    
if len(@metadatafields) > 0
        
set @metadatafields ', ' rtrim
(@metadatafields)

    
-- build/execute query statement
    -- the status code must be null
    
exec (
'    select top 1 '''' As CRMUpdateStatus, 
            c.ContactId As entityid,
            c.StateCode As statecode,
            dbo.p_fn_GetID(l.DestinationId) As originatingleadid, 
            dbo.p_fn_GetID(p.DestinationId) As defaultpriceLevelid, 
            dbo.p_fn_GetID(a.DestinationId) As accountid, 
            dbo.p_fn_GetID(su.DestinationId) As owninguser 
            ' 
+ @entityfields + @metadatafields + 
'

                 from cdf_Contact c

       inner join cdf_Contact_info info 
                   on c.ContactId = info.ContactId

       left outer join cdf_Contact_ext ext 
                   on c.ContactId = ext.ContactId

      left outer join cdf_Lead_info l 
                   on c.OriginatingLeadId = l.LeadId

      left outer join cdf_PriceLevel_info p 
                   on c.DefaultPriceLevelId = p.PriceLevelId

      left outer join cdf_Account_info a 
                   on c.AccountId = a.AccountId

           inner join cdf_SystemUser_info su 
                   on c.OwningUser = su.SystemUserId

           where (info.StatusCode is null)
         and (info.DestinationId is null)'
)
end
GO


Colorized by: CarlosAg.CodeColorizer

 

posted on Monday, July 23, 2007 2:48:07 PM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]
# Thursday, July 19, 2007

SQL Server - Get Matrix for two different result sets

I had a need today to splice 2 different result sets together and effectively give all permutations of the two tables.  I'd seen different examples on doing this with cursors and loops etc, but I thought - there 'must' be an easier way - there's lots of different join types right?

The first issue I'd got is that there's no common fields between the two tables I need to merge because otherwise a 'simple' FULL OUTER JOIN would do the biz.

In the end it didn't amount to much more than that because a slightly dodgy hack (but functional nontheless), to guarantee full coverage from the join is to simply place the same literal on both sides of the join :-)

With 2 sets of 'different' data (let's use Northwind as I'm not about to bore you with my own data).  Say you wanted to generate a list of all products for all regions given the following result sets...

SELECT ProductName
FROM Products

SELECT RegionDescription
FROM Region

You can splice them together with this....

SELECT  R.RegionDescription,
        P.ProductName
FROM Products P
FULL OUTER JOIN Region R
    ON 1 = 1

and you get something like this... (snippet)

RegionDescription                                  ProductName                             
-------------------------------------------------- ----------------------------------------
Eastern                                            Valkoinen suklaa
Eastern                                            Vegie-spread
Eastern                                            Wimmers gute Semmelknödel
Eastern                                            Zaanse koeken
Western                                            Alice Mutton
Western                                            Aniseed Syrup
Western                                            Boston Crab Meat

If there's actually a common field then your join can obviously use that.  Any other filtering you want to do can also go into the WHERE clause as required.

This is another reminder of the lesson that if you're tempted to use a CURSOR, then resist that urge until you've exhausted your Google finger!

SQLServerCentral and SQLMag often have the answer you're looking for.

posted on Thursday, July 19, 2007 10:48:06 AM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]
# Tuesday, June 26, 2007

SQL Server - Kill all users script

Found this the other day - for when you're restoring SQL databases.  You need to kick out all current users right?

Run this little fella into your master database and away you go - just run this step before your restore.

CREATE PROCEDURE usp_KillUsers @dbname varchar(50as
SET NOCOUNT ON

DECLARE 
@strSQL varchar(255)
PRINT 'Killing Users'
PRINT '-----------------'

CREATE table #tmpUsers(
spid 
int,
dbname 
varchar(50))

INSERT INTO #tmpUsers
SELECT 
[Process ID]    l.req_spid, db_name(l.rsc_dbid)
from master.dbo.syslockinfo l with (NOLOCK
where db_name(l.rsc_dbid) @dbname
group by l.req_spid, db_name(l.rsc_dbid)


DECLARE LoginCursor CURSOR
READ_ONLY
FOR SELECT spid, dbname FROM #tmpUsers WHERE dbname @dbname

DECLARE @spid varchar(10)
DECLARE @dbname2 varchar(40)
OPEN LoginCursor

FETCH NEXT FROM LoginCursor INTO @spid, @dbname2
WHILE (@@fetch_status <> -1)
BEGIN
    IF 
(@@fetch_status <> -2)
    
BEGIN
    PRINT 
'Killing ' + @spid
    
SET @strSQL 'KILL ' + @spid
    
EXEC (@strSQL)
    
END
    FETCH 
NEXT FROM LoginCursor INTO  @spid, @dbname2
END

CLOSE 
LoginCursor
DEALLOCATE LoginCursor

DROP table #tmpUsers

Colorized by: CarlosAg.CodeColorizer
posted on Tuesday, June 26, 2007 8:15:39 AM (AUS Eastern Standard Time, UTC+10:00)  #    Comments [0]
# Monday, January 29, 2007

SQL Server Tips - Optional Parameters and Find Closest Match

It's funny how you can survive for years not knowing some really simple stuff.  I learned two useful things last week.  For instance...

Optional SQL Parameters

To have an 'optional' parameter in a SQL 'search' Stored Proc you'd often declare a parameter thus...

@MyParam VARCHAR(20) = NULL --to denote that we want to ignore this param

and you'd probably

  1. have some convaluted logic to possibly build dynamic SQL around whether to factor the parameter into your query
  2. build some nasty 'like' clause that defaults to '%'
  3. derive MIN and MAX values from the input parameter and use a BETWEEN
  4. use a 'magic value' in place of your NULL to denote 'ignore'
  5. something else equally ugly I've not considered

...or (assuming this parameter is expecting to do a direct match) you could just simply do

WHERE @MyParam = ISNULL(@MyParam, MyFieldIWantToMatch)

This will ensure if your field is nulled out it just does a simple (IF 1 = 1) type thing.

 

Find Closest Match

I had a need to find the closest match to a particular value within a 10% tolerance +-.  The following does that fairly simply (I did nick the idea from thescripts.com) but these sorts of things do avoid the need for nasty tempdb use with temp tables...

The following is an example - so don't think I design tables with one column!

DECLARE @Amount INT

SET @Amount = 50

DECLARE @MinAmount INT
DECLARE @MaxAmount INT

SET @MinAmount = @Amount * 0.9
SET @MaxAmount = @Amount * 1.1

SELECT  AT.Amount
FROM AT.AmountTable AT
WHERE Amount BETWEEN @MinAmount AND @MaxAmount --within range
ORDER BY ABS(@Amount - AT.Amount) ASC --Nifty 'closest' match

You can then take the first row in your result set to be the closest match.

 

posted on Monday, January 29, 2007 3:52:05 PM (AUS Eastern Daylight Time, UTC+11:00)  #    Comments [0]