Meeting Room Random Free/Busy Error MapiExceptionRpcServerTooBusy

The other day I was faced with an issue where one or more meeting rooms would randomly not show their free/busy information depending on how many meeting room I was querying at the same time in Outlook. If using Outlook on the Web (aka OWA), a few rooms would not have free/busy information available.

After some digging, I found the following two events logged in the Exchange server (the output has been truncated):

Log Name:      Application
Source:        MSExchange Availability
Event ID:      4009
Task Category: Availability Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EXAIO
Description:
Process Microsoft.Exchange.InfoWorker.Common.Delayed`1[System.String]: Unable to open connection for mailbox SMTP:R_IT_01B_CF@nunomota.pt. Exception returned is: Microsoft.Exchange.Data.Storage.StorageTransientException: Cannot query rows in a table. ---> Microsoft.Mapi.MapiExceptionRpcServerTooBusy: MapiExceptionRpcServerTooBusy: Unable to query table rows.

Log Name:      Application
Source:        MSExchange Availability
Event ID:      4009
Task Category: Availability Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EXAIO
Description:
Process Microsoft.Exchange.InfoWorker.Common.Delayed`1[System.String]: Unable to open connection for mailbox SMTP:R_HR_04B_CF@nunomota.pt. Exception returned is: Microsoft.Exchange.Data.Storage.StorageTransientException: Cannot open mailbox /o=ADIAEXCHANGE/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=EXAIO/cn=Microsoft System Attendant. ---> Microsoft.Mapi.MapiExceptionRpcServerTooBusy: MapiExceptionRpcServerTooBusy: Unable to make connection to the server.

The problem was that the RPC Pool limit was being reached. For each database we can have up to 32 concurrent RPC connections from the same client process. In my case, most meeting rooms were distributed across mainly two databases:

Count Name
----- ----
   83 MDB01
   51 MDB02
    5 MDB03
    3 MDB04
    7 MDB05
    5 MDB06
    4 MDB07
    1 MDB08
    1 MDB09

As you probably guessed, the resolution is to simply split meeting rooms across all databases (when possible). Once I distributed them so there is only 17/18 meeting rooms per database, this error no longer happened.

Create Shadow Copies on the same Active Directory Site

In the Transport High Availability in Exchange 2013 article we discussed, amongst other topics, Exchange’s Shadow Redundancy feature and how it generates redundant copies of e-mails before these being delivered to mailboxes and before Exchange acknowledging to the sending server successfully receiving them.

We also saw that in DAG environments, Exchange gives preference to creating a shadow copy of an email on a DAG member that is located in a different Active Directory (AD) site, if any. But what if we have a DAG that extends one or more AD sites but we want shadow copies to be created only within the same site? If we look at our Transport Config, we will see a ShadowMessagePreferenceSetting parameter (the following output has been truncated):

[PS] C:\> Get-TransportConfig | FL

MaxRetriesForLocalSiteShadow       : 2
MaxRetriesForRemoteSiteShadow      : 4
ShadowHeartbeatFrequency           : 00:02:00
ShadowMessageAutoDiscardInterval   : 2.00:00:00
ShadowMessagePreferenceSetting     : PreferRemote
ShadowRedundancyEnabled            : True
ShadowResubmitTimeSpan             : 03:00:00

The ShadowMessagePreferenceSetting parameters has three possible settings:

• PreferRemote: Exchange tries to make a shadow copy of the message on a server in a different AD site. If the operation fails, it tries a server in the local AD site;
• LocalOnly: a shadow copy of the message should only be made on a server in the local AD site;
• RemoteOnly: a shadow copy of the message should only be made on a server in a different AD site.

Please have in mind that this parameter is only meaningful when the primary server that is trying to make a shadow copy of the message is a Mailbox server that is a member of a DAG that spans multiple AD sites.

As such, if we want our shadow copies to be created within the same site, all we have to do is update the parameter to LocalOnly. However, when we try to do so we might get the following error depending if the other parameters still have their default values:

[PS] C:\> Set-TransportConfig -ShadowMessagePreferenceSetting LocalOnly

The value for MaxRetriesForRemoteSiteShadow must be set to zero for the LocalOnly shadow redundancy preference setting.

So what we need to do is to also update the MaxRetriesForRemoteSiteShadow parameter:

[PS] C:\> Set-TransportConfig -ShadowMessagePreferenceSetting LocalOnly -MaxRetriesForRemoteSiteShadow 0

ExMon for Exchange 2013/2016 Now Available

ExMon, also known as Microsoft Exchange Server User Monitor, has finally been updated for Exchange 2013 and 2016!

You can download the installation file here, as well as a PDF how-to guide on how to use ExMon.

Distribution Groups Statistics

As an Exchange Administrator, have you ever wondered if all those Distribution Groups are actually being used? Organizations running Microsoft Exchange Server are likely to have been running Exchange for a at least a few years and also likely to continue to do so for a while. After all those years, more and more distribution groups get created, some of them stop being used, some are simply forgotten, etc.

Maybe you got asked by the Audit department or by your manager for a list of the 20 most utilized groups, or maybe you are just curious. The good news is that as long as you have Message Tracking Logs enabled, you can easily get this information!

The easiest way to track messages sent to distribution groups is to list all the expansion events. When a user sends an email to a group, Exchange needs to expand that group in order to know who to send the email to. This gets registered with an EventID of EXPAND. Additionally, the RelatedRecipientAddress field in the EXPAND entry contains the PrimarySmtpAddress of the expanded group. And this is pretty much all the information we need.

Using the following cmdlet, we can get a list of all the emails sent to distribution groups on the 1st of January 2016:

Get-MessageTrackingLog -Start 01/01/2016 –End 01/02/2016 -EventID Expand | Select Timestamp, RelatedRecipientAddress, MessageSubject

But what we really want is to know how many emails were sent to which groups during a particular time. Not a problem! All we need to do is tweak the cmdlet above and add Group-Object:

Get-MessageTrackingLog -Start 01/01/2016 –End 01/02/2016 -EventID Expand | Group RelatedRecipientAddress | Sort Count –Descending | Select Count, Name

Do not forget to add Get-TransportService (or Get-TransportServer in Exchange 2007 and 2010) to get the logs across all transport servers if you have more than one!

If you are interested in getting the Top 20 most used groups, for example, this is also very easy:

Get-TransportService | Get-MessageTrackingLog -ResultSize Unlimited -Start 01/01/2016 -EventID Expand | Group RelatedRecipientAddress | Sort Count -Descending | Select -First 20 | FT Count, Name -AutoSize

Exchange Message Queue RiskLevel

When using the Get-Queue cmdlet to view configuration information for queues on Exchange servers, several properties are available for each queue such as DeliveryType, NextHopDomain, Status, MessageCount, LastError, Velocity, RetryCount, and many, many more.

Some of these properties have already been discussed in MSExchange.org articles and/or tips such as Exchange 2013 Queue Velocity. Another interesting one is RiskLevel:

So what is this Risk Level property? Well, according to the Queue filters TechNet article, “this property is reserved for internal Microsoft use, and isn't used in on-premises Exchange 2013 organizations.” As such, I am assuming on-premises customers do not need to worry about this. But just for information, according to the MSDN RiskLevel enumeration webpage, this property has four possible values:

• Normal: The associated message is normal risk.
• Bulk: The associated message is part of a bulk mailing.
• High: The associated message is high risk.
• Low: The associated message is low risk.

Unable to Download Offline Address Book

The other day I was troubleshooting an issue where users were not able to download the Offline Address Book (OAB). After some troubleshooting steps, I ran the Outlook AutoConfiguration Test and noticed that the URL for the OAB was not listed:

Looking at the properties of the OAB, I noticed that Global Web Distribution was disabled and Virtual Directories was empty:

So what do these two properties do exactly?

• GlobalWebDistributionEnabled specifies whether distribution occurs to all virtual directories in the organization. If the value of the GlobalWebDistributionEnabled parameter is $True, distribution occurs to all virtual directories in the organization, and we cannot add values to the VirtualDirectories parameter;
• VirtualDirectories specifies the array of OABVirtualDirectory objects. If this parameter is specified, Version4 of the OAB must be generated.

Because, for some strange reason, none of these properties was set, there was no virtual directory for users to download the OAB from.

Remember that in Exchange 2013:

• OAB files are not stored locally on the CAS;
• OAB files are generated and stored in the Organization Mailbox first and later copied to the %ExchangeInstallPath%\ClientAccess\OAB\ folder on the Mailbox server responsible for generating the OAB;
• CAS 2013 proxies all OAB download requests to the appropriate Mailbox server (the one above);
• Outlook receives OAB URL from Autodiscover and reaches designated CAS 2013 through OAB URL.

So, without OAB URL users simply do not know where to go to get the OAB! Depending on the environment, we now have two options. The first is to specify which virtual directory(ies) to use:

Or simply enable Global Web Distribution:

After resetting IIS (or recycling the OAB application pool), if we run the Outlook AutoConfiguration Test again we should see the OAB URL:

To further test the OAB, beside manually downloading it in Outlook, you can write down the OAB URL, paste it into the address box in Windows Internet Explorer, add /OAB.xml to the end of the path, and then press Enter. The following is an example of what the URL and response resembles:

 

https://mail.domain.com/OAB/6b7ccc1d-7313-453b-9870-5e4708068e95/OAB.xml

Outlook 2016 New Sync Slider Options

With Outlook 2013 we saw the introduction of the Sync Slider, which helps limit the size of OST’s by controlling how much recent mail was synchronized locally on the device. For Exchange accounts, this was 12 months by default, with the option to go down to 1 month or up to all mail.

In Outlook 2016, Microsoft added more sync options that allow for better storage management by only retaining this much mail on the device:

• 3 days;

• 1 or 2 weeks;

• 1, 3, 6, 12 or 24 months;

• All.

 

As before, this setting can also be adjusted using Group Policy.

Restricting Outlook Clients to Cached Mode

Some organizations prefer all Outlook clients to connect to their Exchange servers using Cached Exchange Mode (also known as offline mode) because clients in Online Mode can generate significantly more disk I/O traffic. Besides using Group Policy to control this from the client side, in Exchange 2007, 2010, 2013 and 2016 we can force all clients to use Cached Mode at the server end by running the following cmdlet:

Get-Mailbox | Set-CASMailbox -MAPIBlockOutlookNonCachedMode $True

The Set-CASMailbox cmdlet is used to configure client access settings on a mailbox. For example, we can configure settings for ActiveSync, Outlook, Outlook on the web, POP3, and IMAP4. The MAPIBlockOutlookNonCachedMode parameters, which is available only in on-premises Exchange servers, controls access to the mailbox by using Outlook in online or offline mode. Valid values for this parameter are:

• True: only Outlook clients that are configured to use Cached mode are allowed to access the mailbox;
• False: the state of the Cached mode setting is not checked before Outlook clients are allowed to access the mailbox (both online and cached mode are allowed). This is the default value.

However, be careful with this setting! If there are Outlook clients currently configured in Online mode and you set this parameter to True, they will receive the following error message and will not be able to connect until they change their Outlook profile to Cached mode:

 

Prevent Users from Changing Photo in OWA

Using Outlook Web App (OWA, or Outlook on the Web as it is now known), users can change/update their photo using two methods:

1.    By clicking on the Display Name (Exchange 2013) or user photo (Exchange 2016) in the main OWA window and then Change:

 

2.    Using the options menu and then account followed by Edit information... in Exchange 2013, or “My account” section:

 

 

 

Some companies do not like the idea of users updating their pictures themselves. There are already many posts out there on how to prevent users from changing their photo in OWA. However, some posts only say to change the SetPhotoEnabled parameter to False for all OWA virtual directories:

Get-OwaVirtualDirectory | Set-OwaVirtualDirectory -SetPhotoEnabled $False

 

If we now click on our display name or photo, the Change button is now gone:

 

The problem is that this method might only prevents users from clicking their photo to change their photo! It is possible that users can still change it through the options menu:

The reason why this might happen is because this setting only applies to mailboxes if they do not have an OWA Mailbox Policy applied to them! If they do, then we also need to update this policy, whether it is the Default policy or not. In this case I am updating all policies and applying the default one just to my account for testing:

Get-OWAMailboxPolicy | Set-OWAMailboxPolicy -SetPhotoEnabled $False
Set-CASMailbox nuno -OWAMailboxPolicy Default

Now the change button is finally gone:

Exchange Web Services Editor

Using PowerShell, Exchange administrators can develop scripts to do almost everything. However, there are occasions where a script that uses Exchange Web Services (EWS) is required, may that be for an administrative task or to develop an entire application.

 

I have developed a few EWS scripts but I admit I am no expert on the subject. As such, sometimes I rely on a great tool that not many people are aware of, the EWS Editor. This tool has 3 main goals:

1.      "Demonstrate the Exchange Web Services Managed API functionality and simplicity to developers through its source code;

2.      Demonstrate the Exchange Web Services SOAP traffic used to perform actions initiated through an explorer user interface;

3.      Assist non-developers in debugging and understanding Exchange stores by exploring items, folders, and their properties in depth."

 

Each release of EWS Editor includes the distribution of the EWS Managed API it was built for. EWSEditor uses .NET Framework version 4.5 which can be downloaded here: .NET Framework 4.5. The latest version of EWS Editor (from May 2015) uses .EWS Managed API 2.2, which requires a minimal of .NET 3.5. EWS Editor can be downloaded from CodePlex.

 

EWS Editor is, in some ways, similar to MFCMapi, another great tool. It does not require installation and unlike MFCMapi it does not require Outlook profiles in order to access mailbox items as it does everything over EWS.

We can check mailbox items and their properties:

We can look into particular folders, its emails, attachments, properties (notice how all the property names are easy to read), etc.:

 

We can test and get information regarding AutoDiscover:

The tool comes with a multitude of EWS Posts examples that we can use to see how they are built or even post them against Exchange and analyze the response:

 

We can also check and/or set Out-Of-Office messages for users (assuming we have the right permissions to do so):

It even comes with a handy Distribution Group expansion tool that allows us to see exactly who is a member of a particular group:

Another great tool is the Debug Log Viewer where we can see all the EWS requests and responses for all the actions we do using this tool (in this case we can see the response from the Messaging group expansion above):

There are many, many other things we can use this tool for:

Identifying IIS Worker Process in Exchange 2013/2016

It might come a time when troubleshooting performance issues where we need to identify exactly what a particular IIS Worker Process is in an Exchange 2013 environment:

First thing we need to do is get the Process ID (or PID). To do this, right-click on one of the columns and select PID:

We can now see that that PID for the IIS Worker Process we are trying to identify is 10300:

Now open a Command line or PowerShell console, navigate to Windows\System32\Inetsrv and then run the command “appcmd list wp” or “.\appcmd list wp” if you are using PowerShell:

We now know that the IIS Worker Process with the PID of 10300 is the Exchange OWA Application Pool.

Message Tracking Logs Missing

The other day someone asked me for help regarding Message Tracking Logs missing. The problem was that the person had the message tracking logs configured to save 60 days’ worth of data but he could not find a particular email that was supposedly delivered before those 60 days.

 

The first step was to verify that 60 days of logs were actually being saved:

Get-TransportService | Select MessageTrackingLogMax*
MessageTrackingLogMaxAge:           60 Days
MessageTrackingLogMaxDirectorySize: 1000 MB
MessageTrackingLogMaxFileSize:      10MB

Going to the location where the logs are saved (by default %ExchangeInstallPath%TransportRoles\Logs\MessageTracking), there were indeed 60 days’ worth of logs, so why couldn’t we find the email?! Also, the overall folder size was over 1GB in size... But how can that be if we specified to only use 1000MB?! Let us go back a bit first...

 

The naming convention for message tracking log files in Exchange 2013 is MSGTRKyyyymmdd-nnnn.log, MSGTRKMAyyyymmdd-nnnn.log, MSGTRKMDyyyymmdd-nnnn.log and MSGTRKMSyyyymmdd-nnnn.log . The different logs are used by the following services:

• MSGTRK: these logs are associated with the Transport service;
• MSGTRKMA: these logs are associated with the approvals and rejections used by moderated transport;
• MSGTRKMD: these logs are associated with messages delivered to mailboxes by the Mailbox Transport Delivery service;
• MSGTRKMS: these logs are associated with messages sent from mailboxes by the Mailbox Transport Submission service.

 

The placeholders in the log file names represent the following information:

• The placeholder yyyymmdd is the coordinated universal time (UTC) date on which the log file was created. Yyyy = year, mm = month and dd = day;
• The placeholder nnnn is an instance number that starts at the value of 1 daily for each message tracking log file name prefix.

 

Information is written to each log file until the file size reaches its maximum specified value (MessageTrackingLogMaxFileSize) for each log file. Then, a new log file that has an incremented instance number is opened. This process is repeated throughout the day. The log file rotation functionality deletes the oldest log files when either of the following conditions is true:

• A log file reaches its maximum specified age;
• The message tracking log directory reaches its maximum specified size.

 

Now here comes the important part! The maximum size of the message tracking log directory is calculated as the total size of all log files that have the same name prefix. Other files that do not follow the name prefix convention are not counted in the total directory size calculation.

On Exchange 2013 Mailbox servers, the maximum size of the message tracking log directory is three times the specified value. Although the message tracking log files that are generated by the four different services have four different name prefixes, the amount and frequency of data written to the MSGTRKMA log files is negligible compared to the three other log file prefixes.

 

Going back to the initial issue, the problem was that there were less than 60 days’ worth of MSGTRKMD logs as the combined log files size has met the 1000 MB limit...

So, the bottom line is that, as with Exchange 2010, if you want to keep X amount of days of message tracking logs, ensure you set MessageTrackingLogMaxDirectorySize to a high enough value.

How to Bypass the Clutter in Exchange Online

Clutter is a feature in Office 365 designed to help users focus on the most important messages in their Inbox by moving lower priority messages out of their way and into a new Clutter folder. Exchange Online keeps track of the emails users read and the ones they don't read. Once it is turned it on, Clutter is automatic. As new email comes in, it takes messages users are most likely to ignore and puts them into the Clutter folder. The more users use it, the better it gets.

For organizations, it is important to ensure that certain messages do not end up in users’ Clutter folders. These could be messages from the CEO, from the HR or Finance departments, from the ServiceDesk, etc. In order to ensure this is the case, we need to create a transport rule that makes sure that these messages bypass the clutter folder.

We can do this by using PowerShell to create a rule similar to the following for example:

New-TransportRule -Name “Bypass Clutter” –From servicedesk@nunomota.pt -SetHeaderName “X-MS-Exchange-Organization-BypassClutter” -SetHeaderValue “true”

Please note that both “X-MS-Exchange-Organization-BypassClutter” and “true” are case sensitive!

If we prefer to use the UI, the transport rule will look like this:

Office 365 Support and Recovery Assistance tool

If you are experiencing problems with Outlook or are having trouble installing Office apps, Microsoft's new Office 365 Support and Recovery Assistant tool (aka SaRA) can help diagnose and fix many common user or client side issues.

The tool performs a series of diagnostics tests to help identify the root cause of issues, such as verifying users’ credentials, licenses, updates to Outlook clients, and whether Outlook servers are reachable. Depending on the test results, it can offer to automatically fix problems for users or provide instruction on recommended solutions. All the diagnostics results are saved in a log file for users to share with their Outlook admin or support engineers for further investigation.

Each time we run SaRa, it automatically gets updated to its latest version, so it can troubleshoot any new Outlook problems.

SaRA, still in the pre-release (beta) stage, can be downloaded from this link: http://aka.ms/snrpublic

How to determine which store worker process is responsible for which mailbox database?

As we all know by now, the Exchange Store service in Exchange 2013 has been rewritten in such a way that each database now runs under its own process, thus preventing store issues to affect all databases in the server. Managed Store is the new name for the rewritten Information Store process (store.exe). It is now written in C#, designed to enable a more granular management of resources (additional I/O reduction, for example) and is even more integrated with the Exchange Replication service (MSExchangeRepl.exe), in order to provide a higher level of availability.

The database engine continues to be ESE, but the mailbox database schema itself has changed in order to provide many optimizations.

The Managed Store is composed of two processes. The first one is the Store Worker Process (Microsoft.Exchange.Store.Worker.exe) that is similar to the old store.exe process. The difference is, as already mentioned, that there is one Store Worker Process for each database. This means that if one of these processes fails, only the database it is responsible for will be affected, while all the other databases will remain operational.

The second one is the Store Service Process (Microsoft.Exchange.Store.Service.exe) that controls all store worker processes. For example, when a database is mounted, the store service process will start a new store worker process for that particular database. On the other hand, when a database is dismounted, it will terminate the store worker process responsible for that database.

The question that sometimes arises is “how do we determine which store worker process is responsible for which mailbox database?” To show how to do this, I am going to use a test server where I have 2 mailbox databases, and therefore two Microsoft.Exchange.Store.Worker.exe):

First thing we need to do is get the Process ID (or PID). To do this, right-click on one of the columns and select PID:

 

 

We can now see that that PID for the store worker we are trying to identify is 3308:

 

Now open an Exchange Management Shell console and run the following cmdlet:

Get-MailboxDatabase -Status | Sort Name | FT Name, WorkerProcessID

 

We now know that the store worker process with the PID of 3308 is responsible for the mailbox database DB01.

How to Generate Exchange OAB Multiple Times a Day

The generation of the Offline Address Book (OAB) back in Exchange 2010 was based on a schedule set on the OAB’s properties. In Exchange 2013 (at least in CU9 at the time of writing this) and in Exchange 2016 "beta" we still see this property but it is no longer used:

 

 

Instead, Exchange Server 2013 OAB Generation takes place according to OABGeneratorWorkCycle and OABGeneratorWorkCycleCheckpoint properties configured at the Mailbox Server. According to TechNet:

• The OABGeneratorWorkCycle parameter specifies the time span in which the OAB generation on the Mailbox server will be processed. The default value is 8 hours;
• The OABGeneratorWorkCycleCheckpoint parameter specifies the time span at which to run OAB generation on the Mailbox server. The default value is 1 hour.

 

 

Maybe it is because English is not my first language, but it took me a while to understand exactly what the above means... Eventually I got to the conclusion that what the above text is trying to tell us is that, by default, Exchange updates the OAB every 8 hours. If I look at my environment, I can see this is indeed the case:

 

 

For organizations where recipients change at a considerable rate, this might not be ideal. Let us say that we want to want to update the OAB every 2 hours instead. To achieve this, we simply run the following cmdlet:

Set-MailboxServer “server” -OABGeneratorWorkCycle 00.02:00:00 -OABGeneratorWorkCycleCheckpoint 00:30:00

 

 

 

After a few hours, we can check the Application log and look for Event ID 17002 to ensure the OAB is actually being generated every two hours (Event ID 17001 shows when the generation started while 17002 when it completed):

 

 

 

Don’t forget that to set these properties across all your Mailbox servers that might be generating the OAB. The server hosting the arbitration mailbox with Persisted Capability “OrganizationCapabilityOABGen” is the one responsible for the OAB generation. For a non-DAG environment, you can use the following cmdlet to identify the OAB Generation server:

Get-Mailbox -Arbitration | Where {$_.PersistedCapabilities -match "oab"} | FT Name, ServerName

For a DAG environment, use the following one instead:

Get-Mailbox -Arbitration | Where {$_.PersistedCapabilities -match "oab"} | FT Name, Database

And then check which server currently has that database mounted:

Get-MailboxDatabaseCopyStatus “database”