Instead of working, I blog.

There are some DBs that just can't be tuned any more than they already are (or aren't). A good example of this is an application that hits a DB and never qualifies any of its queries. They all hit with select * and no where clause. There's really nothing you can do to increase the performance short of just throwing more spindles at it. But that's not really what I'm thinking about right now. What I've got on my mind now is an application and DB that just can't be tuned no matter what you do because the business owners don't see the benefit of making the changes.

I saw that a lot when I first got to my current gig. We had queries doing horrendous things and taking several hours to return and nobody cared. The end users had been running these queries for years and were happy with them. They didn't care that the server was maxed out all the time and that they had to wait 12hrs for a report to return. Now, I don't have to tell you that as a DBA that just drives me insane. Not to mention that it gives me nothing to do. Why am I even here then?

So with that in mind, I had to go a little cowboy on them and just start making minor changes that proved my point. I really can't stress enough that I'm against going cowboy on any DB and I don't care who you are. But there are some instances where it's warranted. You have to get the ball rolling somehow. And how this DB got in such bad shape was definitely their fault, but their current view wasn't. They had just been so used to things working the way they were that they didn't see the need to change. They got their reports more or less when they expected them and even if they had to wait a couple extra hours for them they didn't really mind because they understood the server was busy.

So what I did was just start by indexing a couple huge #tables. I just picked a couple of the worst SPs and added a couple indexes. Then I went in and started commenting out cursors and replacing them with simple join queries. Both of these made a huge difference. Then I just sat back and waited. You really don't want to go too far with something like this. Then when they started noticing that their 12hr queries were coming back in just a few secs, then I had their attention. I was then able to convince them to let me go even further and start really tearing into some of these SPs.

And now, for the first time ever, we've got a near-realtime reporting effort in our company. They've come a long way from 'I don't care if it takes 12hrs' to 'I have to have it now'. The problem is they still slip back into their old habits now and then. They currently want to implement an encryption solution that will take around 2mins to return for each report when the solution I suggested returns in about 2secs. And sure, 2mins isn't really going to break the bank, but as those of you who have done a lot of query tuning should know, you have to be hungry for resources. You have to treat every single CPU tick as a drop of water in the desert. If you don't, you'll wake up one day and be in the middle of the same shit you were in before. You have to fight over milliseconds and squeeze every last drop of performance out of every query you can find. That's the only way to run a real tuning effort.

But it's amazing how politics and perception find their way into every aspect of your job.

I just came across a training site I haven't seen before. They've got a couple free vids, but it's hard to really get a feel for the training from just those.
So I'm trying to get a pass to the site so I can review it for you guys and let you know if it's worth it.

I'll keep you posted.

I've always had a soft spot for celebrities who have their entire lives spilled out in public. It's gotta be tough to not only have everyone see how good or bad you do you in your job, but also in your personal life as well. And I've often thought for everything I'm not (rich, well-known, etc) at least I don't have all my failures made public.

At least, that's the way it is until you have a disaster with one of your DBs. Then everyone wants to come stand over your shoulder and watch you bring it back online. Now the heat is on because you have to remember every last command and parameter in front of the crowd. And knowing that is like trying to stop laughing in church. The pressure is just too great. Some DBAs fold at this time. Others just do their jobs like nothing's going on. Me, I clear my desk of on-lookers. I had that happen just this morning. I can't stand to have people just standing there watching me work. I like to be able to follow a train of thought without worrying about how I come off to my audience. So I always tell them... the sooner you leave the sooner I can get to work fixing this.
This morning's disaster came in the form of a dev sending me an email telling me his system was down. He then followed up with a trip to my desk. So I asked him... did you come over just to stand over my shoulder and watch me work? Thankfully, he took the hint and left. But it's not always that easy.

I've always been very sensitive about that with other people too. Unless invited, I always try to stay on the backside of the monitor... especially when someone is trying to get something done in a hurry or needs to concentrate.

There are those who don't really mind people watching them. These would be those who present at conferences all the time and are used to it. But I'm not one of them. Anyway though... give your DBAs a break. If you want them to do something for you, don't stand there and watch them. Just leave and they'll call you when it's ready.

I've started making some SQL Server tutorial videos. I've done like 3 or 4 so far and I'm going to post them soon. I'm starting with a basic topic and building on that. So this time it's backup. I started with the basic backup syntax and am working up to complicated backup procedures. This is meant to be kind of an inclusive tutorial series for someone who doesn't know backups at all to taking them through some decent types of procedures. I'll be branching out too, this is just where I started. I'll post links once they're online.

As you know our other DBA just left and he just started his new job. I was IMing him today and asked him how it was going. He said, the last DBA was an idiot. It's funny, cause I can't count the number of times I've said that. I don't thing I've ever started a job where the guy before me knew what he was doing.

The question is though, am I really all that good or do I just have an inflated ego? I'd probably have to say it's a bit of both really. I've seen a lot of DBAs who just don't know the simplest things about SQL. I've talked about this several times in both my blogs so I'm not going to harp on it too much right now, but it holds more true every year I'm in this industry.

There's a difference between just doing things differently than the other guy, and his systems actually being neglected. Not performing backups or index maint. is bad DBAing. It's not just a different way of doing things. I remember talking to a guy who was a very high DBA at a company we all know last year. I was at PASS come to think of it. And he sat there proudly and told me that they NEVER change their sa passwords on any of their systems. I would love to tell you his reasoning, but I just couldn't get into something like that. But to be proud that you never change your sa password is just assinine. You know what they say dude, if you look around the room and you can't find the asshole, it's you. The same goes with idiots.

It's hard to measure skill though. Everyone has such different experiences. Things I've come to know well may be completely foreign to a different DBA who's far better than I at something else. So does it make him an idiot because he doesn't know what I know? Yeah, sometimes. The basics should be covered. Every DBA should know what it is to backup a system and do maint. and basic security. And so often it's these basics that aren't covered.

So now it comes down to simplicity. What makes a really good DBA? I've had several talks with guys all over IT about this same topic, and in almost every session, we've pretty much concluded that you only have to try a little bit to be better than the average guy. The average guy does very very little to further his knowledge or to get really good at his job. Most people just skate by. So if you try just a little bit you can rise above the crowd. That's what I think anyway.

This is the perfect topic to go along with what I wrote on my other blog today in The real difference between SQL Server and Oracle.

I just got an email from the producer of the new Kalen Delaney series on SQL Server giving me my press pass into the online content for this series. I've only watched the 1st 9mins so far and already it's exactly what I'm talking about in my other blog. Here's Kalen Delaney who writes one of the most successful series on SQL Server (the other one is by the late Ken Henderson. I still have a hard time saying that), and she's going the extra mile to put her book into a video training series where she explains the concepts herself.
I, like many other people learn better when things are explained to me than I do from a lifeless page. And Kalen's an experienced teacher so she has a way of explaining things that make you just get it. Already in this video she's already covered security of metadata and the sys schema. She's actually explaining how this stuff fits together from the ground up. That's how it's done. I have no doubt that the rest of the series will contain the same deep-level understanding.

I think I'm going to enjoy this series and I'll try to write-up a full report when I'm done. Or maybe I'll just do it as I go along.

OK, so here's the link to the site. You can order the DVD or you can watch it online. It's good stuff. Seriously, go check it out if you haven't.
I was recently chatting with Kalen in email and she told me that this is basically the course she teaches when she's brought into a company to teach a class.

Actually, I didn't mean this to be an official interview, but I'm going to go ahead and paste her email here. I'm sure she won't mind (at least I hope not) and she explains it better than I would anyway. I typically don't post emails without asking first, but she knows who I am and she answered my questions like she was being interviewed, so this one time I'm going to do it. But you'll almost never see me take this liberty.

1. What material will this first DVD cover…

You can get information about my course here: http://www.insidesqlserver.com/Course%20Description%20and%20Outline.htm
The first DVD covers most of what is in Module 1.

2. What format will it take… will be be a group of slides and whitepapers, or screencast instruction by you…

The DVD will be a mixture of live capture of me talking, and screen captures of my slides and my demos.


3. Who all is involved in the project…

I am recording the class that I have been presenting all over the world for the last several years. Chuck Boyce, of AskaSQLGuru.com is doing the filming and editing. The business side is being managed by Peter Ward of www.WardyIT.com in Brisbane, Australia


4. How often can we expect to see a new DVD come out…

Since I have to fly to New York for filming, we are only able to do about one a month. In fact, I am just about to leave for the airport for the second round of filming.


5. What advantage will one have in ordering these over just getting the books…

Different people learn in different ways. If you like to hear and see someone explaining concepts, this can add to the benefit of the books. People pay a lot of money to attend my classes, but since I’m only one person, I can’t offer them that often. The DVDs are a chance to for anyone, anywhere to get to take my class. If you can read and absorb everything in the books on your own, the DVDs might not offer anything more.

So again, here's the link to SQLServerDVD.com.

I've had to call a vendor for support and it's already not going well. I'm not going to say which vendor, but I'll tell you that I'm having problems with my SQL backups.
Anyway, I went online and filled out the form with all the info... OS, SQL version, etc... then I got the email from the support tech asking me for all this info.

I replied by saying I had already filled that stuff out when I created the ticket so why did I have to do it again? I don't mean to be a difficult customer, but come on. Why do I have to do everything twice?

Every morning when I leave the house I get in the car and the first thing I do is throw my jellied toast face down in the floor of my car. Why not, that's where it's going to end up anyway when the asshole in front of me slams on his breaks. Now at least it doens't piss me off when it happens. Then when I get to work the first thing I do is run some kind of wild query that fills up all the memory and CPU and locks everyone out of the system. Why not, that's what's going to happen as soon as my favorite report writer logs in.

Some days it just doesn't pay to get out of bed and go through the hassle. Then again, if you've done what you should as a DBA, it shouldn't be that bad. Hopefully you've setup things in your DB that keep things from getting too out of hand. Hopefully you've been able to teach your report writer some basic dos and don'ts (sp?).
But what if you're in a new place? Can you still be effective? Of course you can. Again, if you're a good DBA you've collected a nice gaggle of scripts that you take from place to place. Years ago when I was just getting started, I didn't get that. I always thought, why be lazy, just write the damn scripts when you need them. But that's the wrong attitude. It's not laziness, it's practicality. There's just no reason to work out that logic every time.

So if you follow some of your own best practices and set yourself up for success, maybe you can withstand the bad times. But there's nothing you can do about traffic, so I guess you'll have to keep throwing your toast on the floor. Life goes on.

For those of you who follow my blog, you know that I've used LiteSpeed for many years now. And one of the things that's always bugged me is the lousy way they run their upgrades. To upgrade LS typically consists of having to do some version of manually deleting the current repository tables and then running the install program. And this is after uninstalling your previous version manually as well. And if you want to save your data in your repo tables you have to create new ones and copy the data over, or just rename the old tables. And it still isn't that easy because of the relationships. If you just rename the tables you have to rename the relationships too or the new install will fail. And you have to do this for every box you own. What a pain in the ass!!

Well, I have a problem with LS not backing up one of my DBs even though it's been working just fine for almost 2yrs. So I just upgraded from v.4.7 to the latest 4.8 (yes, I know v.5 is coming out soon, but I can't wait). And let me tell you this... it looks like they finally got these upgrade problems fixed. My upgrade went smooth. I didn't have to manually uninstall the old version, I didn't have to play my usual shell game with the repo tables, and I didn't have to burp the service. Life is good.

Now, I'll have to see if it actually fixed my problem. But I'm happy to be able to report something good about LS for the first time in a long time. They've mostly been adding features, and very slowly at that, and haven't been really concerned with our actual problems. Maybe this is a turning point.

Anyway, good job LS guys.

One thing that's as varied as ways to code is how to manage the code itself. Or I guess I should say architect instead of manage, but it all comes down to management.

The 2 major ways are to componentize and to not. And by componentize, I mean taking all the individual components and turning them into small independent chunks that can be called from many sources. A good example is taking a date calculator routine in your SPs and turning it into a function so the SPs just call it instead.
So those are the 2 ways. You either put it inline with the code or you make it into a component and call it from all the modules you need.

Of course, I'm mainly talking about SQL code here, but it really applies to any kind of code I suppose.

It's easy to see the advantages of developing a component solution. This is the DB equivalent of COM, right? But what's not easy to see is what it does for you (or to you) support-wise. What this solution can do is put you in a new form of .dll hell from the old days.

Let's take a simple example...
Let's say that you have sn SP in prod that's giving you problems and you want to track down where the degraded performance is coming from. So you open up the SP and start looking at the code. You see that a major component of it is an external SP call. OK, so you open up that SP. Now you see that that SP calls 3 other SPs. And each one of them is calling more SPs. And so on and so on. This is the definition of spaghetti code isn't it?

I'm actually in the middle of doing this very thing right now. I wanted to stop and blog about it while I was still pissed off. I've been doing this for an hour now and I'm not any closer to a solution than I was when I started. So you can see that while going the COM route can be helpful, you can take it too far. And that's really what happens, isn't it? Devs take a good idea and drive it into the ground until it's so hard to manage it's not even worth having it anymore.

I've been trying to come up with some guidelines for this and it's tough. But roughly, I like to say that if several SPs use the same code, or are calculating the same thing, it's ok to pull out into COM. The same goes with code that gets changed a lot. If you've got code that changes often and is used in a lot of SPs, by all means put it into it's own SP so you only have one place to change it.

Anyway, I guess I'll get back to my plate of spaghetti now.

This post is dedicated to all those field DBAs who like to call up the prod DBAs and tell them how busy the server is based on the number of spids or short-term blocks returned by sp_who(2).

See this is the hidden cost of generic logins that have too many rights. Everyone on the planet can read BOL and try to interpret the results. And whatever they mark in their heads as being the sign of a server being too busy is what they're going to call you with. Our users have 2 criteria for a busy server. The number of spids (active or inactive), and how many short-term blocks they see.

Of course I used to try to explain to them that you could bring the server down with a single spid so the number doesn't matter... and that blocks are fine as long as they don't persist. Since I've been here for quite a while though, and none of them have gotten the hint yet, I usually just thank them for letting us know and that we'll get right on it.

One of my favorite analogies used to be that judging the server on the amount of spids is like loading your car up with people and declaring that traffic is really bad today. Somehow that still didn't get the point across.

It's almost never fun to lose a DBA, but it's a fact of life. People leave jobs, and sometimes jobs leave people. This is another reason why it's really important to not tax your DBAs too heavily. If you've got 2 DBAs and they're both working like dogs, what happens when you lose one? I'll tell you what happens... deadlines start to slip, backups start failing and don't get looked into, maint starts getting behind, security gets relaxed, etc. You want your DBA to have time to do his job and be able to pick up some slack when you lose someone. And it could be quite a while before you get a replacement. Good DBAs are really hard to find and you don't want someone to just warm the seat.
I talked about this recently in my IW blog. Of course, this really only goes for production DBAs, right? I mean, you can work your devs as much as you want. They'll never get a call in the middle of the night because the server's down or because a package failed. So again, DBAs are insurance policies. We're kinda like a clustered server. You don't use the inactive node. It just sits there waiting for something to happen to the primary node. It seems a terrible waste and managers hate spending that money for a box that just sits there. And while DBAs aren't quite that useless, we really should be used in the right way. So it really is just like a multi-node cluster. You never run the primaries at full capacity because one day something will happen and one box will have to take on its workload and one of the downed nodes. So if they're all running at 100%, they can't failover and resume work. So you run them at 50%... give or take, right?

So again, let your prod people do their prod jobs and don't put them on too many actual projects. Afterall, that's what prod means.

And yeah, we're losing our other DBA so I'm all alone now. We'll see how it turns out.