Exchange 2007 Catalog (a.k.a. Index) Maintenance

If you don’t want to hear the back story, go ahead and skip down to ‘Rubber, Meet Road’ section, I tend to be windy. Is it even called windy when you type incessantly? I digress….

So, we’re in the middle of an Exchange migration. As most of you know, sometimes these migrations can be painfully slow, suffer setbacks, delays, etc. So, to that end, we had to move mailboxes off of server A and over to server B due to disk storage issues. Sadly, server A doesn’t have enough room on the drive to do an offline defrag after moving the email to server B. Needless to say, the drive is REALLY low on space, I’m talking under 1GB out of 600GB low. I monitored the disk carefully for a while, making sure that the almost 400GB of whitespace in the databases I have freed up is doing the trick keeping the drive from filling up. After a week or two, things are looking good with no substantial decrease in disk space, which was what I expected. Alas, a month later (yes, the migration was only supposed to be delayed by a few weeks, but again, we all know how that goes) I get more disk space alerts, and about 100 more MB of space had been chewed up. I filter the event log for event ID 1221 which tells me that I have ample whitespace remaining in the two databases…. So what is taking up my space? Following best practices, log files are on a different drive, so I know that isn’t the culprit. So, as you likely already figured out from the title, it was in fact the Exchange CatalogData-<GUID> folder(s). There were two folders, one for each database, and their total size was 15GB. Now, that is a lot of index for only 300GB of mail. I knew that even 10GB would likely buy me the space I needed to ride this migration delay out and not have to perform any unnecessary maintenance and downtime.

In Exchange 2007 the CatalogData-<GUID> folder is located in the same folder as the mailbox database by default. I had initially set out to move the CatalogData-<GUID> folders as I have some other drives on that server with some space. One of the first few links returned on my search sent me to this article by Vidad Cosonok. Vidad’s article was a quick ‘how to’ on freeing up space in short order on a filled up drive to buy some time. I read through it, and it looked easy enough, and it also didn’t look like it was going to cause any downtime so I gave it a go. Right around 5 p.m. (to be on the safe side) just like the article said, I stopped the Microsoft Exchange Search Indexer service, deleted two CatalogData-<GUID> folders with a hard delete (shift-delete) and restarted the service. Bing Bang, just like that, I had 15GB of free space on the drive, no errors, and the service had re-created the folders and was re-indexing the two databases. It is important to point out, that if a user tries to run a search on their email while the index is being rebuilt, it will take an extremely long time and may return a “No Items Found” when the item is actually there (false negative). It might behoove you to do this after hours, and depending on your user tolerance, you might consider doing it over a weekend as it will tax the server during the rebuilding of the index. The next morning, I checked the CatalogData folders and their combined size was 1.5GB. How about that! This leads me to believe that Microsoft Exchange 2007 does zero maintenance on catalog files, and an administrator should probably add it to their best practices and list of yearly maintenance tasks to rebuild the indexes. I would also encourage you to do this shortly after a migration on the source server, as is the case in this scenario, assuming you need the space. I monitored the index over the next couple of days and found the CatalogData-<GUID> folder did not increase much, which suggests to me that it is done with the initial indexing pass on the database and is simply keeping up with new email.

Rubber, Meet Road

Initial Problem: Free Space on Exchange 2007 Database Drive

Initial Solution: Move CatalogData-<GUID> File(s)

Actual Solution: Regenerate Exchange 2007 Database Index to flush old index files and return space to the OS.

Steps Taken:

  1. Stop Microsoft Exchange Indexer Service
  2. Delete CatalogData-<GUID> folders*
  3. Start Microsoft Exchange Indexer Service
  4. Monitor for full re-index to complete.

It is important to point out, that if a user tries to run a search on their email while the initial re-index is occurring, it will take an extremely long time and may return a “No Items Found” when the item is actually there (false negative). It might behoove you to do this after hours, and depending on your user tolerance, you might consider doing it over a weekend.

One more quick disclaimer, I bugged my Exchange Guru colleague Robert Durkin and asked about the application of this situation to Exchange 2010, and he said that while there are some differences with the distribution of the catalog when a 2010 mailbox database is in a DAG, in general the process should be the same in 2010 as mentioned by the article I linked above by Vidad Cosonok.  I did not test this on 2010 and cannot speak to its applicability, so as always I’d recommend testing first.

Manually Remove a Service with PowerShell

From time to time, you’ll be faced with a piece of software whose uninstall is poorly written, a virus or malware, or a freak power failure during an uninstall. In instances like these, you might have to remove an orphaned service in Windows. In my particular case, our old monitoring software was Zenith Infotech, and their software left behind two services that can really booger up an Exchange server if you don’t get rid of them.

The first thing you need to do is open up Server Manager, and drill down to the server’s services and get the name of the service(s) you need to remove by right clicking on the service and selecting properties from the sub menu:

At this point, I personally opened up regedit and verified the location of the service in the registry for sanity’s sake:

Now we have the information we need to delete the service. If you have just one service on one server, then you can just delete the service’s registry key from Registry Editor and be done with it. Since I have over 90 servers I need to do this for, I strung together these PowerShell commands to remove these services.

The first thing I decided to do was stop the service, just in case it was actually trying to do something to the OS:

Stop-Service SAAZappr

The next command identifies the registry key to be removed (everything after the HKLM: part as it appears in the bottom of the Registry Editor window highlighted above) and removes it, and by adding the –Recurse switch, we’re also telling it to automatically remove all of its sub-containers, keys and parts. For good measure, I tagged –Force on the end in the event some sort of permissions issue decided to rear its ugly head:

Get-ChildItem HKLM:\SYSTEM\CurrentControlSet\Services\SaaZAppr | Where-Object {$_.PSChildName -ne ‘CLSID’} | Remove-Item –Force

Finally, I took the data from the “ImagePath” section of the Registry Key and made sure I deleted all of the folders, subfolders, and files etc. from the server that were also potentially left behind, also using the –Recurse and –Force switches:

Remove-Item “C:\program files\SAAZOD” -Recurse –Force

The last thing I did was compile the script into an .exe to ease deployment on all of my servers. I complied into an .exe using PowerGUI Pro.

So, the final script, removing both of the SAAZ services, covering both 32 and 64 bit installations, looked like this:

# SaaZ Services Killer

# Written by Matt Richardson

# 02/14/2012

Stop-Service SAAZappr

Stop-Service SAAZapsc

Get-ChildItem HKLM:\SYSTEM\CurrentControlSet\Services\SaaZAppr | Where-Object {$_.PSChildName -ne ‘CLSID’} | Remove-Item -Force 

Get-ChildItem HKLM:\SYSTEM\CurrentControlSet\Services\SaaZapsc | Where-Object {$_.PSChildName -ne ‘CLSID’} | Remove-Item -Force

Remove-Item “C:\program files\SAAZExmonScripts” -Recurse -Force

Remove-Item “C:\program files\SAAZOD” -Recurse -Force

Remove-Item “C:\program files (x86)\SAAZExmonScripts” -Recurse -Force

Remove-Item “C:\program files (x86)\SAAZOD” -Recurse –Force

This script will get an error every time since it is trying to delete both 32 bit and 64 bit installation folders, but the errors are ambiguous and don’t stop the script from completing so I didn’t see the harm in it or the value in building in the logic to identify the version and delete accordingly.