Exchange Install Error Database is mandatory on UserMailbox

The other day I was trying to update the first Exchange 2013 server in this particular test environment to CU13 when I got the following error when installing the Mailbox role:

The error itself contained the code Exchange tried to run, so it wasn't clear what the exact problem was. An event in the Application event log mentioned the exact same thing:

Log Name:      Application
Source:        MSExchangeSetup
Date:          8/7/2016 2:34:52 PM
Event ID:      1002
Task Category: Microsoft Exchange Setup
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX...
Description:   Exchange Server component Mailbox role: Mailbox service failed.
Error: Error:  The following error was generated when $error.Clear();
(...)

Time do dig deeper and look at the ExchangeSetup.log file! In here, I got some more information about what the problem was: Exchange was failing while trying to do something to the arbitration mailbox SystemMailbox{bb55...} (the one used in the Offline Address Book generation process):

[08/07/2016 10:34:51.0582] [2] Processing object "domain/Users/SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}".
[08/07/2016 10:34:51.0582] [2] [ERROR] Database is mandatory on UserMailbox.
[08/07/2016 10:34:51.0598] [2] [ERROR] Database is mandatory on UserMailbox.
[08/07/2016 10:34:51.0598] [2] [ERROR] Database is mandatory on UserMailbox.
[08/07/2016 10:34:51.0598] [2] Ending processing set-mailbox
(...)
[08/07/2016 10:34:52.0004] [1] The following 2 error(s) occurred during task execution:
[08/07/2016 10:34:52.0004] [1] 0.  ErrorRecord: Database is mandatory on UserMailbox.
[08/07/2016 10:34:52.0004] [1] 0.  ErrorRecord: Microsoft.Exchange.Data.DataValidationException: Database is mandatory on UserMailbox.
(...)
[08/07/2016 10:34:52.0004] [1] [ERROR] Database is mandatory on UserMailbox.
[08/07/2016 10:34:52.0004] [1] 1.  ErrorRecord: Database is mandatory on UserMailbox.
[08/07/2016 10:34:52.0004] [1] 1.  ErrorRecord: Microsoft.Exchange.Data.DataValidationException: Database is mandatory on UserMailbox.
(...)
[08/07/2016 10:34:52.0004] [1] [ERROR] Database is mandatory on UserMailbox.
[08/07/2016 10:34:52.0004] [1] [ERROR-REFERENCE] Id=SystemAttendantDependent___1DEE95834DBA48F2BB211C2FB6765A5A Component=EXCHANGE14:\Current\Release\Client Access\Unified Messaging
[08/07/2016 10:34:52.0004] [1] Setup is stopping now because of one or more critical errors.
[08/07/2016 10:34:52.0004] [1] Finished executing component tasks.
[08/07/2016 10:34:52.0036] [1] Ending processing Install-MailboxRole
[08/07/2016 10:36:39.0444] [0] CurrentResult setupbase.maincore:396: 0
[08/07/2016 10:36:39.0444] [0] End of Setup
[08/07/2016 10:36:39.0444] [0] **********************************************

But what exactly does it mean by "database is mandatory"? Checking all the arbitration mailboxes using the Get-Mailbox cmdlet, I noticed that this mailbox was corrupted, it did not have a database, and I received the exact same error as during the installation:

Looking at its properties in Active Directory, the homeMDB attribute was indeed blank:

There are two ways to fix this: either remove this arbitration mailbox using ADSIEdit, re-run the Exchange setup to re-create it and then use the New-Mailbox cmdlet to complete its configuration; or rehome the mailbox by running the following cmdlet:

After this, the homeMDB attribute had the correct value:

And the installation process completed successfully!

Identifying IIS Worker Process in Exchange 2013/2016

It might come a time when troubleshooting performance issues where we need to identify exactly what a particular IIS Worker Process is in an Exchange 2013 environment:

First thing we need to do is get the Process ID (or PID). To do this, right-click on one of the columns and select PID:

We can now see that that PID for the IIS Worker Process we are trying to identify is 10300:

Now open a Command line or PowerShell console, navigate to Windows\System32\Inetsrv and then run the command “appcmd list wp” or “.\appcmd list wp” if you are using PowerShell:

We now know that the IIS Worker Process with the PID of 10300 is the Exchange OWA Application Pool.

Message Tracking Logs Missing

The other day someone asked me for help regarding Message Tracking Logs missing. The problem was that the person had the message tracking logs configured to save 60 days’ worth of data but he could not find a particular email that was supposedly delivered before those 60 days.

 

The first step was to verify that 60 days of logs were actually being saved:

Get-TransportService | Select MessageTrackingLogMax*
MessageTrackingLogMaxAge:           60 Days
MessageTrackingLogMaxDirectorySize: 1000 MB
MessageTrackingLogMaxFileSize:      10MB

Going to the location where the logs are saved (by default %ExchangeInstallPath%TransportRoles\Logs\MessageTracking), there were indeed 60 days’ worth of logs, so why couldn’t we find the email?! Also, the overall folder size was over 1GB in size... But how can that be if we specified to only use 1000MB?! Let us go back a bit first...

 

The naming convention for message tracking log files in Exchange 2013 is MSGTRKyyyymmdd-nnnn.log, MSGTRKMAyyyymmdd-nnnn.log, MSGTRKMDyyyymmdd-nnnn.log and MSGTRKMSyyyymmdd-nnnn.log . The different logs are used by the following services:

• MSGTRK: these logs are associated with the Transport service;
• MSGTRKMA: these logs are associated with the approvals and rejections used by moderated transport;
• MSGTRKMD: these logs are associated with messages delivered to mailboxes by the Mailbox Transport Delivery service;
• MSGTRKMS: these logs are associated with messages sent from mailboxes by the Mailbox Transport Submission service.

 

The placeholders in the log file names represent the following information:

• The placeholder yyyymmdd is the coordinated universal time (UTC) date on which the log file was created. Yyyy = year, mm = month and dd = day;
• The placeholder nnnn is an instance number that starts at the value of 1 daily for each message tracking log file name prefix.

 

Information is written to each log file until the file size reaches its maximum specified value (MessageTrackingLogMaxFileSize) for each log file. Then, a new log file that has an incremented instance number is opened. This process is repeated throughout the day. The log file rotation functionality deletes the oldest log files when either of the following conditions is true:

• A log file reaches its maximum specified age;
• The message tracking log directory reaches its maximum specified size.

 

Now here comes the important part! The maximum size of the message tracking log directory is calculated as the total size of all log files that have the same name prefix. Other files that do not follow the name prefix convention are not counted in the total directory size calculation.

On Exchange 2013 Mailbox servers, the maximum size of the message tracking log directory is three times the specified value. Although the message tracking log files that are generated by the four different services have four different name prefixes, the amount and frequency of data written to the MSGTRKMA log files is negligible compared to the three other log file prefixes.

 

Going back to the initial issue, the problem was that there were less than 60 days’ worth of MSGTRKMD logs as the combined log files size has met the 1000 MB limit...

So, the bottom line is that, as with Exchange 2010, if you want to keep X amount of days of message tracking logs, ensure you set MessageTrackingLogMaxDirectorySize to a high enough value.

Office 365 Support and Recovery Assistance tool

If you are experiencing problems with Outlook or are having trouble installing Office apps, Microsoft's new Office 365 Support and Recovery Assistant tool (aka SaRA) can help diagnose and fix many common user or client side issues.

The tool performs a series of diagnostics tests to help identify the root cause of issues, such as verifying users’ credentials, licenses, updates to Outlook clients, and whether Outlook servers are reachable. Depending on the test results, it can offer to automatically fix problems for users or provide instruction on recommended solutions. All the diagnostics results are saved in a log file for users to share with their Outlook admin or support engineers for further investigation.

Each time we run SaRa, it automatically gets updated to its latest version, so it can troubleshoot any new Outlook problems.

SaRA, still in the pre-release (beta) stage, can be downloaded from this link: http://aka.ms/snrpublic

Exchange 2013 EAC Performance Console

Almost 3 years ago, on my Exchange 2010 ECP Performance Console article on MSExchange.org, I explored the Performance Console of the Exchange Control Panel in Exchange 2010. Did you know that this console is still present in Exchange 2013?

 

This console, which is not visible by default, provides numerous counters regarding the performance of the EAC. We can use it to check how long it takes to authenticate a user, how many PowerShell cmdlets have been invoked and even how long the server took to process requests, and much more.

 

To enable it, we have to manually edit the web.config file located at:

%ExchangeInstallPath%\V15\ClientAccess\ecp\web.config

 

Open the file with Notepad and look for the "appSettings" section, right in the first few lines. In there we will find the following key:

 

<!-- Set ShowPerformanceConsole to "true" to show ECP's Perf Console: -->
<add key="ShowPerformanceConsole" value="false" />

 

 

As the comment explains, all we have to do to enable the console is update the value of the ShowPerformanceConsole key from false to true. Save the file, run the usual IISRESET /NOFORCE to restart IIS and we are good to go!

 

If we now log in to the EAC, we will have a Performance console link:

 

Clicking on this link opens the console itself:

 

 

To learn more about this console, check my Exchange 2010 ECP Performance Console article at MSExchange.org.

Exchange Server 2013 SP1 Transport service stops and does not restart

When upgrading Exchange 20013 to Service Pack 1, if you have servers running both back-end and front-end roles and you run the front-end Microsoft Exchange Transport service on the server, you might experience the Microsoft Exchange Transport stopping and not restarting.

 

If you stop the Microsoft Exchange Transport service, you can start the front-end Microsoft Exchange Transport service. Then, when you try to start the Microsoft Exchange Transport service, that service does not start and the following events are logged in the Application log:

Source: MSExchangeTransport

Event ID: 1019

Task Category: SmtpReceive

Level: Error

Description: Failed to start listening (Error: 10048). Binding: 0.0.0.0:25.

Source: MSExchangeTransport

Event ID: 1018

Task Category: SmtpReceive

Level: Error

Description: The address is already in use. Binding: 0.0.0.0:25.

Source: MSExchangeTransport

Event ID: 1036

Task Category: SmtpReceive

Level: Error

Description: Inbound direct trust authentication failed for certificate %1. The source IP address of the server that tried to authenticate to Microsoft Exchange is [%2]. Make sure EdgeSync is running properly.

Source: MSExchange Common

Event ID: 4999

Task Category: General

Level: Error

Description: Watson report about to be sent for process id: 19848, with parameters: E12IIS, c-RTL-AMD64, 15.00.0847.032, MSExchangeTransport, M.Exchange.Net, M.E.P.WorkerProcessManager.HandleWorkerExited, M.E.ProcessManager.WorkerProcessRequestedAbnormalTerminationException, 5e2b, 15.00.0847.030. ErrorReportingEnabled: False

Source: MSExchange TransportService

Event ID: 1046

Task Category: ProcessManager

Level: Warning

Description: Worker process with process ID 18420 requested the service to terminate with an unhandled exception.

This issue occurs if there is a receive connector of Transport type HubTransport that has the binding set to port 25 on the affected Exchange 2013 server. On an Exchange 2013 server that has both back-end and front-end roles, only the FrontendTransport server-type receive connector should have the binding set to port 25.

To fix this issue, run the following cmdlet to change the connector type from HubTransport to FrontendTransport:

Set-ReceiveConnector –Identity "Receive connector name" –TransportRole FrontendTransport

Alternatively, delete and re-create the receive connector and set its role to FrontendTransport.

Troubleshooting Outlook Calendar Problems

If your users are experiencing issues with their Outlook Calendar, Microsoft released back in May a tool to help administrators in these situations, the Calendar Checking Tool for Outlook (also known as CalCheck).

This is a command-line tool that opens an Outlook profile on the local machine, opens the Outlook Calendar and checks permissions, free/busy information and auto booking, for example. The tool then checks each item in the calendar for any problems.

The calendar to by analyzed must reside on an Exchange Server and this tool does not work with IMAP, POP3 or any other non-Exchange mail servers.

To download it go to this link.

Exchange 2010 ECP Performance Console

A hidden feature that most Exchange Administrators don't know about is the Exchange Control Panel [ECP] Performance Console.

This console, which is not visible by default, provides numerous counters regarding the performance of the ECP. We can use it to check how long it takes to authenticate a user, how many PowerShell cmdlets have been invoked and even how long the server took to process requests.

To learn more about this console and how to enable it, please check the Exchange 2010 ECPPerformance Console article on MSExchange.org