Tuesday, September 11, 2012

Mailboxes Quarantined due to Troubleshoot-DatabaseSpace.ps1 Script

Exchange 2010 SP1 added a new script called Troubleshoot-DatabaseSpace.ps1 that is used to detect excessive growth of database and log drive volumes. This script can be run manually by administrators but if you have Microsoft System Center Operations Manager (SCOM) 2007 it is run automatically every 15 minutes.

This script performs the following actions:
1. Track log generation rate for the highest log generators (mailboxes) per database. This helps determine which users are logging too heavily and potentially causing space issues;
2. Track available disk space for both database and log files. If either of these is within a configurable threshold of being full (25% by default), further action must is taken;
3. Track log generation rate. If it appears that the disk is going to run out of space within the value specified by the HourThreshold parameter (12 hours by default and based on the log generation rate), further action is taken;
4. If all of the above conditions are fulfilled, the script determines the list of top 25 users who accessed the database during the last 1 hour period. The script then quarantines the top high-usage mailboxes for 6 hours, during which they will not have access to e-mail;
5. If the troubleshooter is unsuccessful at dropping the log generation rate to below the threshold level, it will write out events that translate into health model alerts. At this point, the script removes the database from provisioning by running the Set-MailboxDatabase cmdlet with the ExcludeFromProvisioning parameter set to $True against the specified database;


When mailboxes are quarantined, you will see entries in the Registry of the mailbox server hosting that database/mailbox in:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\"server_name"\Private-"DB.Guid"\"Mailbox_Guid"


To check the events logged by this script go to the mailbox server you want to check and then Event Viewer -> Application and Services Logs -> Microsoft-Exchange-Troubleshooters/Operational. Note that if you run the script manually through the Shell it will not produce any output - you have to check the Event Viewer.


If you run the script manually without any parameters or if you have SCOM running this script automatically, then it will use default values specified in the StoreTSConstants.ps1 script located in the same folder. This is where you can customize the 25% threshold, for example, if it is not ideal for your environment:

   # The percentage of disk space for the EDB file at which we should start quarantining users.
   $PercentEdbFreeSpaceDefaultThreshold = 25

   # The percentage of disk space for the logs at which we should start quarantining users.
   $PercentLogFreeSpaceDefaultThreshold = 25

   # The percentage of disk space for the EDB file at which we are at alert levels.
   $PercentEdbFreeSpaceAlertThreshold = 16

   # The percentage of disk space for the EDB file at which we are at critical levels.
   $PercentEdbFreeSpaceCriticalThreshold = 8

   #The number of hours we can wait before running out of space.
   $HourDefaultThreshold = 12


In order for mailboxes to be quarantined, the –Quarantine parameter must be passed to the script, which the SCOM monitor uses by default. Because the Exchange Management Pack is sealed, you can’t change this...

So, if 25% is too high for your environment, you can change the value by updating the StoreTSConstants.ps1 script across all your mailbox servers or, ultimately, simply disable the SCOM monitors that run this script:
• KHI: Failed to execute Troubleshoot-DatabaseSpace.ps1
• KHI:The database copy is low on database volume space and continues to grow. The volume has reached critical levels 8% free.
• KHI:The database copy is low on database volume space and continues to grow. The volume has reached error levels under 16% free.
• KHI:The database copy is low on database volume space and continues to grow. The volume is under 25% free.

6 comments:

  1. Anyone have trouble editing the values in the StoreTsConstants.ps1 script file? I have changed it on our servers yet it still takes the default 25% value. I cannot find the script on any of our scom servers. Any information would be very helpful. Thanks.

    ReplyDelete
  2. Hi,

    Are mailboxes still being Quarantined for you then?
    In my environment I changed:
    $PercentEdbFreeSpaceDefaultThreshold
    $PercentLogFreeSpaceDefaultThreshold
    $PercentEdbFreeSpaceAlertThreshold
    $PercentEdbFreeSpaceCriticalThreshold

    and "SCOM" picked those values straight away because all it does is run the scripts that are on the Exchange servers.
    Have you updated the script across all mailbox servers?

    Regards,
    Nuno

    ReplyDelete
  3. Hi Nuno. Sorry about the late reply. I wasn't aware that you had replied. I only changed the values on the servers that have the space issue. On servers that had more than 25% we left the StoreTsConstants.ps1 file alone. i used 12,12,10,8 as the values.

    ReplyDelete
  4. Hi,
    No worries! Now it's me who says sorry for the late reply...
    Are your DBs part of a DAG?
    What do you get in the event viewer if you run the script manually?
    To be honest, I would still change it across all servers, unless you have a reason not to.
    Let me know how it goes.

    Regards,
    Nuno

    ReplyDelete
  5. Hi Nuno,
    Setting the values in the default StoreTSConstants.ps1 won't help in the case of SCOM, as SCOM has a hard-coded storeTSConstants.ps1 built into the management pack. (On an Exchange 2010 server with SCOM installed, search for instances of StoreTSConstants.ps1, and you'll find another copy in the SCOM temporary folders.) Therefore the only possible way is to disable the monitors or create a crazy script that continually replaces the temporary ones. :-( (Well OK, you could of course change the values in the management pack, and then resign it if you happen to have the appropriate code-signing certificate just lying around somewhere)

    Regards,
    Alginald

    ReplyDelete
    Replies
    1. Hi Alginald,

      Thank you very much for the clarifying that. I must confess I updated the values but in my environment the default ones haven’t been reached yet, so I am not seeing any SCOM alerts or anything.
      That explains the issues experienced by “SeekExchangeKnowledge”…
      Thanks a lot once again! I will update the post accordingly soon.

      Regards, Nuno

      Delete