Instead of working, I blog.

This post was particularly inspired while cleaning my house today.  It’s easy, if you think you’ve got a clean house then just stand up on a ladder some time and change your perspective.  I think you’ll find there are lots of things about your clean house you don’t know.  And while nobody’s officially complaining about the condition of the house (and in fact, everyone thinks it’s in great shape), it clearly is in disarray when you shift to that new bird’s eye view.

The same is true for your DB.  I honestly can’t count the number of times I’ve been told that there was no reason to look at DB performance because nobody was complaining about it.  Then when I dig into it and look at things from a DBA angle, the system is actually abysmal.  So if performance is so bad then why isn’t anyone complaining?  Well there can be a couple reasons for that.  First of all the DB could just not be very busy comparatively so the users won’t notice the issues.  And second, they may have just gotten used to things running slower so again they don’t notice it.  I’ve actually seen that several times.  Things get slower and slower, or better yet, they just start out slow and nobody ever knows the difference.  And what’s even WORSE is that more often than not there’s some cornhole in the background telling them that Oracle would solve all their problems.  Listen, if you can’t write a simple process against mssql and make it run right do you really think you’ll be able to hit Oracle ok?

So I’ve gotten off topic a little… back on track.  The point of all this is that just because nobody’s complaining about performance that doesn’t mean that your system is performing.  I quite often find that companies have no idea what their systems can scale to and they just assume that they’ll be able to handle whatever they throw at them.  And it typically takes more forethought than most companies have to look at a system objectively and proactively fix problems.

So the next time you’re considering your system performance, stand on a ladder or crawl down on the floor and see how things look from there.

Watch my free SQL Server Tutorials at:
http://MidnightDBA.ITBookworm.com

Read my book reviews at:
www.ITBookworm.com

Blog Author of:
Database Underground – http://www.infoworld.com/blogs/sean-mccown

Follow my Twitter:

http://twitter.com/MidnightDBA

As I promised yesterday I’m gonna talk about process and coding standards today and I want you to really think about what I’m saying here because it’s what separates the men from the boys.

This topic won’t ramble on quite as much as the past 2 days have though so don’t worry.  Basically what I want to say is that you should definitely be creating coding standards and following them, but don’t rest there and don’t allow any of your standards to rest either.  What I mean by that is just because you come up with a standard, say using ‘insert into’ instead of ‘select into’ during your ETL loads, don’t always blindly follow that standard.  Every process is unique and every process deserves consideration.  You can adversely effect your servers if you do the same thing across the board without ever questioning whether this is the right tool for this particular job or not.  So where a ‘select into’ may be the perfect solution for one really huge snapshot load, an ‘insert into’ may be the right solution for other lesser loads. 

One argument I hear a lot when preaching this bit of DB wisdom is that you want to have a solid way of doing things and you want to standardize.  And frankly, that’s just ridiculous.  That argument doesn’t hold up to even the most basic scrutiny.  Ok, maybe ridiculous isn’t the word… let’s go with asinine instead.  So you’re telling me that for this big snapshot load you should use the exact same process as you do for your smaller ones because standardization is the key?  Then why aren’t all of your loads full snapshots?  Shouldn’t you be standardizing on a single load process?  And what about your indexes?  Do you have the exact same fill factor for every single index in your place?  If so you’re not paying attention to your index stats.  And are you using #table instead of @table absolutely everywhere?  Or are there cases where one is better than the other? 

So you see, with just a few examples I’ve shown you that you don’t do EVERYTHING the same way in your DBs.  So there’s no reason for you to scream standardization with something as important as a data load.  Take each load on an individual basis and decide what the best course of action is for each one.  And yeah, as the data grows or changes in nature you may end up revisiting your process and updating it to reflect the new nature of your objects.  And that’s ok.  So many people think that having to rewrite a process is a huge sin and that you’ve failed somehow.  It’s really not you know.  Not only are these types of things a learning process, but things also change that require you to change tactics. 

I really hope you take what I’m saying here and apply it.  And don’t be one of those idiots who tells me that he runs everything serially because he wrote the process on a single-CPU box and therefore he needs to standardize on that platform instead of taking advantage of a multi-CPU machine.

Watch my free SQL Server Tutorials at:
http://MidnightDBA.ITBookworm.com

Read my book reviews at:
www.ITBookworm.com

Blog Author of:
Database Underground – http://www.infoworld.com/blogs/sean-mccown

Follow my Twitter:

http://twitter.com/MidnightDBA

OK, so yesterday I was talking about how servers can take a long time to finally be brought to their knees with poor processes, and that you want to get to the point where your server is on a strict diet and you control every morsel that goes into it.

Today I want to talk about how that can play into a real performance strategy because unfortunately you have more to consider than just raw performance.  I know, I know, it sounded yesterday like I was advocating that performance was the most important thing.  And it is… but at the same time it isn’t.  So let me explain a little.

Performance is extremely important and some might even say it’s the most important thing on the server, but you also have to consider management et al.

See, quite often the code that leeches out the absolute best performance on the server isn’t as easy to manage as other solutions.  Something like this isn’t very concrete without an example so I’ll give you one.

Let’s look at what I call moving schema.  The guys in my shop are sick to death of hearing that term but it’s an important one.  Moving schema is where you drop and recreate objects all the time that should be permanent.  A good example of this is doing ‘select into’ instead of ‘insert into’.  Sure, the ‘select into’ performs better than the ‘insert into’, but it’s also harder to manage from a couple different perspectives.  For starters, even during your load you can’t specify a filegroup for the object so you’re stuck with the default.  Now for performance or growth reasons you may find it better in general if you put it on a different set of disks, but you can’t because ‘select into’ doesn’t allow it.  So from a space management aspect ‘select into’ traps you in a corner.  Also, if you have specific permissions granted on the table you have to keep track of them somewhere and reapply them every day when you recreate the table.  And you also have to remember to update your permission repository every time you change the perms on that table during the course of your other admin functions.  As well, most performance stats are gathered by objectID.  And if you’re recreating the object every day then your objectIDs are invalid tomorrow.  So you either have to keep track of what the objectID is every day so you can track performance and other issues (like fragmentation) from day to day, or you have to store the objectName in your repository as well which leads to an inefficient and larger schema to track production issues.  It also makes the timing of other processes more critical because with ‘insert into’ they’ll be blocked until the current load finishes, but with ‘select into’ they’ll fail because the object doesn’t even exist. 

So ok, I’ve shown you a few reasons why something as simple as moving schema can be a problem.  And like I said, it’s really one of my diatribes because of those issues mainly, and a couple out-lying ones.  And the guys at my shop have been hearing this battle cry for about 3yrs now and I’m sure they’re quite tired of it.  But when it comes right down to it, it’s good DB practice. 

There are some instances where the moving schema rule can be broken though.  Let’s look at a really large data load for instance.  If for some reason you can’t do any kind of incremental load and you have to do a full snapshot every night (those instances are becoming fewer and fewer with Katmai) on a HUGE table (several hundred million rows), then you may be better off doing a ‘select into’ because you don’t incur the logging and it can it can decrease your load time significantly.  It of course can also help control how big your log grows.  So this is a decision you have to make at the time, right? 

However, I will say that quite often, if you take the size of the log out of the picture, the ‘select into’ doesn’t out-perform the ‘insert into’ by all that much.  I recently tested the 2 methods against 30mill rows and the ‘insert into’ only took about 40secs longer.  That’s really not enough to worry about in the course of a full ETL load.  And for all the extra benefits you get from having a stable object, that 40secs is more than worth it.  So this would be a case where choosing the absolutely best performing process wouldn’t be the best way to go.  In the case above where I talked about loading several hundred million rows, the ‘insert into’ may increase the load time by as many as 30mins and that’s hard to ignore.  So depending on your requirements you may decide that dropping the object every day is worth it.  But in the instance where the difference is only a handful of minutes, you’d typically choose to keep your schema in tact and gain the other benefits.

So OK, you’ve got a process and you’ve decided to drop the schema every day to do your load.  And your process performs as well as it possibly could because of the lack of logging.  The problem is that people tend to think of performance in terms of the single piece they’re writing and nothing else.  Performance includes more than just the load.  Performance also includes up-time, and number of errors, and manageability, etc.  The load is just one part of the process for any given object.  Examples of different parts of an object’s process would be the load process, the reporting process, the troubleshooting process, the triage process, security process, the space management process, etc.  I’m sure there are others, but you guys get the point.  So all of these are all part of the process of managing this object and if they all don’t play well then the process for that object doesn’t work well.  So be careful in throwing all the other processes away and making them extremely more complicated just to appease a single piece.

So I’ve been in several shops and it’s always different everywhere.  Some shops are really big into management and others are really big into performance.  I’ve been in shops where the management of the objects was just a pain because the systems really were so busy that every last CPU cycle was counted and every last I/O had to be justified.  So they were willing to put up with the management headache of that kind of performance cost because they needed it to stay alive.  And again you have to decide what’s best for your place and your processes.  But whatever you decide, don’t rest on that decision.  Keep making it again and again.  I’ll talk about that tomorrow.

I guess that’s all I’ve got to say for now.

Watch my free SQL Server Tutorials at:
http://MidnightDBA.ITBookworm.com

Read my book reviews at:
www.ITBookworm.com

Blog Author of:
Database Underground – http://www.infoworld.com/blogs/sean-mccown

Follow my Twitter:

http://twitter.com/MidnightDBA

One question I get a lot is about performance and how systems can run well for years and then suddenly just stop performing well.  That’s an understandable question and one that’s both easy and complicated to answer.

The short answer is that there’s a huge margin for error in these types of things.  However, the margin is only huge if you’re not doing anything about it.  Let me explain.

It’s like being on a diet.  When you’re watching what you eat every little bit matters.  That extra helping of fries, that bowl of ice cream, and that soda are all death to a strict diet.  Even little things can be harmful when you’re on a diet and the more strict the diet the more the little things matter.  That’s why professional athletes of all kinds watch their intake like hawks.  So in their case that extra ounce of potatoes, or that extra ounce of meat can really make a difference.  And that’s not even to mention diabetics and other people on strict medical diets.  Think about someone with severely high blood pressure.  Their diet is extremely important and the slightest wrong food can have serious blowback on their system.

Now look at someone who’s already grossly overweight.  This guy eats whatever he likes up to thousands of extra calories a day.  He eats only fried and fatty foods and eats as much of it as he likes.  So that extra helping of ice cream or that extra few fries really doesn’t matter much on top of everything else.  That’s not to say that it doesn’t have a cumulative effect, just that day to day it doesn’t matter much.  Eventually though, it will take its toll as he get heavier and heavier and starts to feel health effects from it.  So while those extra fries do eventually catch up with him, they don’t cause any real immediate effect on top of all the other stuff he’s eating.

Well, that’s much the way it is with servers too.  If you have a bunch of stuff that runs kinda poorly or just not as well as it could, it’s not really that important on a daily basis because the server itself runs slow and what’s one more mediocre process going to hurt?  So a server can run for quite a while like that and nobody will ever really notice the difference.  Part of the problem is that so few people bother to investigate better ways to do things so they get used to their DB performing slowly.  It’s not necessarily their fault and these things can sneak up on them.  Even a fairly good DBA can have wrong decisions go undiagnosed for a long time and the poor performance can sneak up on him and next thing he knows his system is just dragging.  And it’s hard to go back and find that one thing that started the whole thing.  I find typically that performance problems are systemic.  By that I mean that whatever mistake is made, is made throughout the whole system.  It’s quite often not an isolated incident unless someone new comes into a shop where things are already running smoothly.

So anyway, a server can put up with a good deal of abuse before it goes belly-up, but it will eventually happen.  What you want to try to get to is a point where you can treat your server like it’s got diabetes.  You want it on a very strict code diet where you watch every single I/O, every single MB going into RAM, every CPU cycle, etc.  On servers like this, one single process that doesn’t behave can have a noticeable effect on many other processes and you’ll know you did something wrong right away.  But if you’ve got a system that’s slow to begin with, who’s really going to notice if it’s running a little slower or if the CPU is hitting 81% instead of 76%?

This is why I’m not only stressing performance considerations on servers, but also why I’m always answering this question.

This of course has to play hand in hand with tomorrow’s post on reasonable performance.

Watch my free SQL Server Tutorials at:
http://MidnightDBA.ITBookworm.com

Read my book reviews at:
www.ITBookworm.com

Blog Author of:
Database Underground – http://www.infoworld.com/blogs/sean-mccown

Follow my Twitter:

http://twitter.com/MidnightDBA

Be careful when you’re complaining about how much advice you get from your DBA because one day he might stop giving it.  And that’s not really a scenario anyone wants because it means he’s stopped caring about the systems and you won’t get any real help.  Because I don’t care what you devs out there think, you need your DBAs to keep your code playing nicely with others.

So when your DBA stops putting in his 2-cents your prayers haven’t been answered… in fact, just the opposite.  You’ve taken your DB support staff and turned him into a wallflower.

So now my question to you is what did you do to shut him up?  Because I guarantee you it was your fault.  Are you constantly going out of your way to circumvent procedures he puts in place?  Are you constantly disregarding his advice?  Do you refuse to setup things in a way that will make it easier for everyone involved to troubleshoot?  Do you refuse to follow any coding guidelines he gives you to keep things going smoothly?

OK so I realize that even though your code runs in double the time it could and it always deadlocks, that you’re the smartest dev out there and you don’t need advice from any of those dumb DBAs, but try shutting up and listening sometimes.

Watch my free SQL Server Tutorials at:
http://MidnightDBA.ITBookworm.com

Read my book reviews at:
www.ITBookworm.com

Blog Author of:
Database Underground – http://www.infoworld.com/blogs/sean-mccown

Follow my Twitter:

http://twitter.com/MidnightDBA

I recently ran across a perfectly delicious situation I wanted to share with you.  Someone actually wrote to tell me about this.  He just started a new gig and there was massive corruption in one of the dev DBs.  He went to ask the DBA where the backup was and he got a very snotty reply. 

Well, as it turns out this was something that the DBA had gone around and around with them about.  See, the devs didn’t want the DBA to do anything on their precious box so they refused to give him rights.  He tried to explain that they needed backups, etc but they wouldn’t hear of any DBAs pissing all over their dev box. 

And now when there’s massive corruption and they need to get their dev code back they call the DBA to ask for help.  Y, fat chance guys. 

I’m here to tell you that the DBA has full snotting rights here.  And it only goes to teach them a lesson I hope.  DBAs aren’t here to piss on your parade.  We’re here to make sure your parade lasts as long as you want and you can even start your parade over and over again if you need to.  Seriously guys all metaphors aside, we’re here to help.  So stop acting like we’re on different sides.

Watch my free SQL Server Tutorials at:
http://MidnightDBA.ITBookworm.com

Read my book reviews at:
www.ITBookworm.com

Blog Author of:
Database Underground – http://www.infoworld.com/blogs/sean-mccown

Follow my Twitter:

http://twitter.com/MidnightDBA

Hey guys... I'm starting a webcast series next week on Beginning Powershell for SQL Server DBAs.

This is an excellent chance for those of you who have always wanted to get started with powershell to do so.

It'll be a ground zero course so if you don't know anything at all about powershell I'll attempt to bring you up.

I hope to see you all there.

Here's a link to the webcast and I hope all of you can join me.

Signup and tell all your friends.  If we get enough people signup we'll be able to keep this series alive.

Watch my free SQL Server Tutorials at:
http://MidnightDBA.ITBookworm.com

Read my book reviews at:
www.ITBookworm.com

Blog Author of:
Database Underground – http://www.infoworld.com/blogs/sean-mccown

Follow my Twitter:

http://twitter.com/MidnightDBA

I had a comment on one of my last posts that suggested that I talk about a few things and this is the first thing that struck my interest because it’s a topic that’s close to my heart.  And of course, I’m talking about getting away from the GUI and learning to do things at the command line.

And I know that SQL Server was built around the cmdline and a lot of you cling to it for all of your daily tasks, but you’ve just got to get past it and start doing things in the GUI.  No wait, reverse that.

So why do I want you to get off the GUI?  Well for starters you never really know a system until you code in it.  You can go your entire life not knowing what goes on under the covers if you’re just clicking around and now you’re never learning anything.  So when problems come up you have no idea how to fix them because you’ve never bothered learning how anything works.

OK, I could sit here and write everything out in anecdotes but since time’s of the essence and my boys are going crazy this morning, I’ll just put it in a list with comments.

1.  GUI is limited – quite often the GUI won’t do everything you need it to do.  This comes up quite often working with security, but also things like CDC and data compression.  The GUI just doesn’t have all the features the cmdline does.  Backup/restore is another good example because you can’t do page-level restores in the GUI.

2.  GUI isn’t scalable – You can’t save GUI actions and reproduce them for several boxes.  If you code a solution you can code in parameters that allow you to run the same code against different objects.  I think user accts are a good example.  What if you have to add a user to all 200 DBs on your server, or on 10 servers?  What are you gonna do?  Click through those perms on all 2000 DBs?  It’ll take you days.  Meanwhile at the cmdline you can run it against all boxes at once and be done in just a couple mins.  Seriously, you can’t beat that.  So everything you do at the GUI you have to reproduce every time you need to perform that task.

3.  Gui provides no knowledge – Like I said already, if you always use the GUI then you never learn much because you don’t ever have any exposure to the inner workings.

4.  GUI is single-task based – Everything you do in the GUI is a single action.  You can’t do too multiple tasks as once and you can’t set them up in transactions so they all pass or fail together.  What if you needed to add a user to SQL and to Windows?  You can do it in powershell at the same time, but in the GUI you have to physically click through and do them yourself.  And what if you wanted to make it so that if one of them failed they both did?  You can’t do that in the GUI.

5.  GUI is typically longer – Even though the GUI is more friendly sometimes, it often takes longer to click through the screens than it does to type the cmds… at least for those of us who can type.

6.  GUI is slower – Many times the GUI has been slower to accomplish a task.  I’ve had the GUI take quite a bit of time or even freeze up on me while trying to do something when the cmdline finished in just a couple secs.  So the GUI isn’t nearly as efficient.

7.  GUIs don’t multitask well – While Yukon brought us the ability to have GUI tasks running in the background while we work in SSMS, you can still only do so many things that way.  You can multitask much better at the cmdline because you can offload it easier to other boxes or just schedule scripts to kick off.  Again, you can’t schedule a GUI operation.

Look… GUIs are fine when you’re a beginner.  They ease the burden of learning something new and when a new technology comes out I quite often start with the GUI just to get my feet wet.  But then I switch to the cmdline pretty quickly and do it in there the rest of the time. 

And there’s not much excuse for not learning the cmdline anymore since Yukon brought us the ability to generate the code for our actions and Katmai has improved on that.  Now if you do something in the GUI  you’re not sure how to do in the cmdline, you just script it from the GUI and learn something.  Now you can just modify it for your purposes.  I do that sometimes still.

I was really lucky when I started learning SQL in that the guy teaching me was a huge cmdline buff and insisted that I did everything that way.  Also, I emulated the MVPs who all touted the cmdline so I figured if it was good enough for them then I should be doing it too.  So in that respect I was lucky to learn that lesson early.

I hope some of you now learn that lesson here and start trying to do your jobs at the cmdline.  It’ll be tough at first, but in a yr you’ll be really glad you took the pains.

Watch my free SQL Server Tutorials at:
http://MidnightDBA.ITBookworm.com

Read my book reviews at:
www.ITBookworm.com

Blog Author of:
Database Underground – http://www.infoworld.com/blogs/sean-mccown

Follow my Twitter:

http://twitter.com/MidnightDBA

We’ve all had companies that didn’t listen to us.  They hire you with the intention of having you make things better and improving their processes.  Then when you come on and start making suggestions, they don’t want to implement any of them.  The reasons can be political, selfish, or just laziness, but the fact remains you’re not doing what you were told you would be.  And the company doesn’t seem to mind.

So what do you do now?  The company doesn’t seem to mind if things run the way they always have been so why should you?  It’s definitely easier to care about the quality of your job when the company does.  But why would any company choose to do things poorly especially when they’ve got to such great lengths to hire someone to fix them?  The answer is I just don’t know.  I’ve seen it too many times under too many companies and it confounds me every time.  I think a lot of time it’s an issue with a boss having too much faith in a guy who’s telling him you don’t know what you’re doing.  And even when you bring solid numbers to him he still doesn’t see the light.  That’s the one that gets me.  When I come to them with solid benchmark results and they still refuse to change the process because there’s a guy they’ve worked with longer who’s lying to them about how it’ll really behave in prod.

OK, so now what to do… well, you’ve really only got a couple choices.  You can quit or you can make your peace with it.  If you’re just not getting anything you need out of the gig then maybe it’s time to move on.  But if you’re getting something else out of it like a good schedule, or work from home, etc then it might be worth it to you to stick around.

If you do stick around then you need to make your peace with it.  And that’s the hard part.  Basically you have to stop caring about the systems and how they run.  Consider yourself more of an internal consultant.  They ask you questions and you advise them.  If they don’t take your advice then it’s not your problem.  It’s really as simple as that.  Of course, if you’re on the hook for support and this is gonna cause a bunch of extra work for you then that’s another story.  But you’ve gotta weigh that against everything else and make a decision.  If they’re really not gonna listen to anything you say then they really don’t understand DBs now do they?  Part of that is education, right?  You have to educate them about how to work with data.  Remember, this data explosion we’re in the middle of is still relatively new and many companies don’t have any idea how to work with it yet.  They have to be taught.  Now if only you can find a way to open them up to listening.

Just remember… numbers don’t lie.  If you keep showing them numbers eventually they may listen.  Of course, for some of these things you could always just do it yourself and say sorry later.  I’d never personally do anything like that(cough-cough), but I know some guys who have.

Watch my free SQL Server Tutorials at:
http://MidnightDBA.ITBookworm.com

Read my book reviews at:
www.ITBookworm.com

Blog Author of:
Database Underground – http://www.infoworld.com/blogs/sean-mccown

Follow my Twitter:

http://twitter.com/MidnightDBA