Thursday, July 16, 2015

Fatal error TooManyMissingItemsPermanentException has occurred

During a recent transition from Exchange 2010 to 2013, a user’s mailbox failed to get migrated:

[PS] C:\>Get-MoveRequestStatistics “nuno mota”

DisplayName          StatusDetail
-----------          ------------
Nuno Mota            FailedOther


Ok, “FailedOther” does not provide much information... Let us get a report of the actual move request to try to find out exactly what happened:

[PS] C:\>Get-MoveRequestStatistics “nuno mota” -IncludeReport | FL


7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"Tania   resume", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"Netting Off entry 2011 March  28", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"Data for weekly report -  11/March/2011", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"FW: 2011 JAN ANALYSIS  revised based on ultimate parent name", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"PAYMENT RECONCILIATION  as on 16/2/2011", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"PORTFOLIO CAL  ACC _EOP  LIST CALCULATION   as on 07/01/2011", Folder:"Outbox"
7/13/2015 10:40:24 AM [server1] Mailbox contents verification complete: 139 folders, 254510 items, 2.829 GB (3,037,189,613 bytes).
7/13/2015 10:40:25 AM [server1] Stage: FinalIncrementalSync. Percent complete: 95.
7/13/2015 10:40:25 AM [server1] Fatal error TooManyMissingItemsPermanentException has occurred.
Ok, so from the output above we can have a clearer view of what is happening. At the end of the report we see a TooManyMissingItemsPermanentException error is preventing Exchange from completing the move request, and the reason for that is all the EnterpriveVault stubs that seem to be missing from the mailbox’s Outbox folder.
When we create a mailbox move request, or batch, we can use the BadItemLimit parameter to specify the maximum number of bad items that are allowed before the request fails. A bad item is a corrupt item in the source mailbox that cannot be copied to the target mailbox. However, also included in the bad item limit are missing items. Missing items are items in the source mailbox that cannot be found in the target mailbox when the request is ready to complete.
So, if we are comfortable ignoring this error and possibly leaving a few bad items behind, we can either increase the BadItemLimit by running:

Set-MoveRequest “nuno mota” –BadItemLimit 50
Resume-MoveRequest “nuno mota”

Valid input for BadItemLimit is an integer or the value unlimited. The default value is 0, which means the request will fail if any bad items are detected. If you set this value to 51 or higher, you also have to use the AcceptLargeDataLoss switch, otherwise the cmdlet will fail.

If too many bad items are detected, consider using the New-MailboxRepairRequest cmdlet to attempt to fix corrupted items in the source mailbox, and try the request again.

Alternatively, we can update the ContentVerificationMissingItemThreshold setting in the MsExchangeMailboxReplication.exe.config file which in Exchange 2013 is located by default at C:\Program Files\Microsoft\Exchange Server\V15\Bin.

If you opt for changing this setting, you have to restart the restart Microsoft Exchange Mailbox Replication service and then resume the move request as above.

Monday, July 13, 2015

Exchange ActiveSync v16 is coming!

The last major version of Exchange ActiveSync (EAS) was v14 which came as part of Exchange 2010 (EAS v14.1 came with Exchange 2010 SP1). Almost 6 years after it is now time for an update to EAS and the release of a new version, v16.
At the time of writing this tip, it is unclear if EAS v16 will be part of Exchange 2016... Like with other features, Microsoft will be releasing it first to Office 365 and then eventually enable it in the on-premises Exchange 2016 builds...
This new version of EAS is planned to contain three new major capabilities:
  1. Improved calendar reliability by reworking the calendar workflow. This will most likely go unnoticed by most end users but will, hopefully, help with those appointment nightmares some Exchange admins experience with 3rd-party mail clients;
  2. Calendar attachments. While currently calendar items synchronized with EAS cannot include attachments such as agendas, presentations or spreadsheets, in version 16 these will also sync;
  3. The drafts folder cannot be synced with the current version of EAS, but it will be in EAS v16. This means we can start composing an email on our EAS device and continue editing it on our desktop, for example, or vice-versa.
To check which ActiveSync protocol versions your Exchange Online mailbox supports, go to and run the Exchange ActiveSync test from the Office 365 tab. From the test’s output, look for MS-ASProtocolVersions:
Like with previous EAS version changes, v16 will require the client to support it. It seems that iOS 9 will at least support the calendar features when it is released this fall.
As always, please note that the information provided here might change when EAS v16 is released.

Tuesday, June 9, 2015

Exchange Management Shell and Active Directory

Working in multi-domain / multi-site environments can sometime be tricky if we don’t know how the Exchange Management Shell (EMS) queries Active Directory (AD) in these scenarios.

In Exchange 2013 and 2010, we can use the Set-AdServerSettings cmdlet to manage the Active Directory Domain Services (ADDS) environment in the current EMS session. This cmdlet cmdlet replaces the AdminSessionADSettings session variable that was used in Exchange 2007 (which we will look at in a minute).

The following example specifies that all recipients in the entire forest can be viewed and managed (by default, only those in the local domain are used):
Set-AdServerSettings -ViewEntireForest $True

The following example sets the recipient scope to the IT Users OU in the domain for the current session:
Set-AdServerSettings -RecipientViewRoot “ Users”

The following example sets the scope of the current session to the entire forest and designates as the preferred global catalog server.
Set-AdServerSettings -ViewEntireForest $True -PreferredGlobalCatalog

The following are the most common parameters that administrators change:
PreferredGlobalCatalog: specifies the FQDN of the global catalog server to be used for reading recipient information in this session;
PreferredServer: specifies the FQDN of the domain controller to be used for this session;
RecipientViewRoot: specifies the OU to include in the recipient scope for this session. When we specify a recipient scope with this parameter, only the recipients included in the scope are returned;
ViewEntireForest: when we specify a value of $true, the value stored in the RecipientViewRoot parameter is removed and all of the recipients in the forest can be viewed and managed.

As already mentioned, in Exchange 2007 we had a variable named $AdminSessionADSettings for this purpose. To achieve the same as the examples above, all we have to do is update this variable as follows.

The following example specifies that all recipients in the entire forest can be viewed and managed (by default, only those in the local domain are used):
$AdminSessionADSettings.ViewEntireForest = $True

The following example sets the recipient scope to the IT Users OU in the domain for the current session:
$AdminSessionADSettings.DefaultScope = “ Users”

To set the recipient scope to the domain and use as the recipient domain controller, run the following commands:
$AdminSessionADSettings.DefaultScope = “”
$AdminSessionADSettings.PreferredDomainControllers = “”

Changing the recipient scope in the EMS changes the set of recipients that are returned for the Get- cmdlets of the recipient. The fields that are stored in the $AdminSessionADSettings variable are retained until the EMS is closed and is reset to its default settings the next time that the EMS is opened.

To make the changes permanent, we have to manually edit the Bin\Exchange.ps1 file in the Exchange Server installation folder and update lines such as:
$global:AdminSessionADSettings.ViewEntireForest = $false

Thursday, May 28, 2015

How to Convert a Distribution List into a Mailbox

Sometimes, an organization/administrator might have the need to convert a Distribution Group (DG) (or Distribution List) into a normal mailbox or a shared mailbox. Reasons for this vary, but the most common one is when an organization has the need to start sending emails as that DG.
Unfortunately, there is no native way of achieving this. But it is completely achievable and straightforward. The best way is to:
  1. Write down the DG’s LegacyExchangeDN. To do so, you can run the cmdlet: Get-DistributionGroup “DG_name” | Select LegacyExchangeDN;
  2. Delete the DG;
  3. Create a (shared) mailbox with the same SMTP address;
  4. Add the DG’s legacyExchangeDN as an X500 address to the new mailbox.

I am not going into detail of what the legacyExchangeDN is and how or why it is used as this is already well documented all over the Internet. As a quick overview, the auto-complete cache in Outlook and in OWA uses the value of the legacyExchangeDN attribute to route email messages internally. If the value changes, the delivery of email messages may fail with a 5.1.1 NDR.
If you already deleted the DG and have no way of retrieving its legacyExchangeDN, you have two options:
  1. Clear the auto-complete cache (straightforward but most of the times not the best approach);
  2. Manually create an X500 proxy address for the old legacyExchangeDN attribute for the DG.

To create an X500 proxy address, you need to use an NDR you have received when emailing the DG, which should contain the recipient’s address such as:

From here, make the following changes based on the recipient address in the NDR:
  • Replace any underscore character (_) with a slash character (/);
  • Replace “+20” with a blank space;
  • Replace “+28” with an opening parenthesis character;
  • Replace “+29" with a closing parenthesis character;
  • Delete the “IMCEAEX-“ string;
  • Delete the “” string;
  • Add “X500:” at the beginning.

After you make these changes, the proxy address will look similar to:
X500:/O=LetsExchange/OU=Exchange Administrative Group (FYDIBOHF23SPDLT)/CN=Recipients/CN=f6a32c0ab0e64f33b2a7b3f9a48c2da6

Friday, May 15, 2015

Create Folder on Users’ Mailboxes

One could think that the Exchange Online and Exchange 2013 New-MailboxFolder cmdlet would allow administrators to create folders on other users’ mailboxes. Unfortunately this is not the case...
Basically RBAC (Role Based Access Control) only allows the administrator to run this cmdlet on the mailbox it owns. As we can see below, RBAC has an implicit recipient read and write scope set to Self:
So can we create a new role based on MyBaseOptions and update the ImplicitRecipientReadScope to OrganizationConfig? Once again, unfortunately no... You see, if you read the Understanding management role scopes TechNet article, it states that:
You can't change the implicit scopes defined on management roles. You can, however, override the implicit write scope and configuration scope on a management role. When a predefined relative scope or custom scope is used on a role assignment, the implicit write scope of the role is overridden, and the new scope takes precedence. The implicit read scope of a role can't be overridden and always applies.
By the way, the exact same thing applies to the Get-MailboxFolder cmdlet... The good news for this cmdlet is that we can simply use the Get-MailboxFolderStatistics cmdlet to list all folders in any mailbox we want.
So, as far as I know, there is nothing we can do to make this cmdlet work for other mailboxes the administrator does not own. So is there a way to create folders for other users? Yes, using Exchange Web Services (EWS) script!    :)
I have written a few EWS scripts to perform certain actions on mailboxes that are not possible using the native Exchange cmdlets. To achieve this, I am not going to re-invent the wheel as there is already a great script by David Barrett to do exactly what we want. For more information on his script, please check his blog article PowerShell: Create folders in users' mailboxes.

Wednesday, May 6, 2015

Exchange 2013 EAC Performance Console

Almost 3 years ago, on my Exchange 2010 ECP Performance Console article on, I explored the Performance Console of the Exchange Control Panel in Exchange 2010. Did you know that this console is still present in Exchange 2013?
This console, which is not visible by default, provides numerous counters regarding the performance of the EAC. We can use it to check how long it takes to authenticate a user, how many PowerShell cmdlets have been invoked and even how long the server took to process requests, and much more.
To enable it, we have to manually edit the web.config file located at:
Open the file with Notepad and look for the "appSettings" section, right in the first few lines. In there we will find the following key:
<!-- Set ShowPerformanceConsole to "true" to show ECP's Perf Console: -->
<add key="ShowPerformanceConsole" value="false" />
As the comment explains, all we have to do to enable the console is update the value of the ShowPerformanceConsole key from false to true. Save the file, run the usual IISRESET /NOFORCE to restart IIS and we are good to go!
If we now log in to the EAC, we will have a Performance console link:
Clicking on this link opens the console itself:
To learn more about this console, check my Exchange 2010 ECP Performance Console article at

Updating AADSync Scheduled Task

If you ever need to update AADSync’s scheduled task, it is likely that you will need to update the credentials it runs under. When you do that, and if you do not assign the correct permissions, you might get the following error message:
This is because the account used does not have the required permissions. Make sure you add it to the local Admins group and to the local ADSyncAdmins group.

Tuesday, April 28, 2015

Exchange 2013 Performance Health Checker Script

Microsoft has just published the Exchange 2013 Performance Health Checker script, which checks various configuration items on Exchange 2013 servers to make sure they match the recommendations published in the “Exchange 2013 Sizing and Configuration Recommendations” guidance on TechNet. It also reports on OS, system and hardware information. It can be ran remotely, against a single server or a group of servers.
The script takes some of the most common configuration causes of Exchange 2013 performance cases that Microsoft has encountered in support and allows administrators to rule them out quickly without having to check each server or read through the entire TechNet guidance.
The current list of items the script reports on is:
  • Operation System version
  • Exchange Build
  • Physical/Virtual Machine
  • Server Manufacturer and Model (physical hardware only)
  • VM host processor/memory configuration recommendations
  • Exchange server roles
  • Pagefile Size
  • Power Settings
  • .NET Framework version
  • Network card name and speed
  • Network card driver date and version (Windows 2012 and Windows 2012 R2 only)
  • RSS enabled (Windows 2012 and Windows 2012 R2 only)
  • Physical Memory amount
  • Processor Model
  • Number of processors, cores, and core speed
  • Hyper-threading enabled/disabled
  • Processor speed being throttled
  • Current list of active/passive databases and mailboxes (optional)
Let us look at some examples. First, we run the script without any parameters, meaning it will check the local server. We could use the –Server parameter to run it against a remote server.

By including the –MailboxReport parameter, the script presents some statistics around mailboxes and databases:

Another useful parameter is –LoadBalancingReport which looks at all CAS servers to determine how user connections are being load balanced across these servers. We can use the -CasServerList to specify which CAS servers we want to check.

Every time the script runs, it outputs the results to a log file as mentioned at the end of every output. In the next screenshot we can see the report from the cmdlet above:

Monday, April 20, 2015

Office 365 Compliance Center

The new Compliance Center contains the key compliance-related features for an Office 365 administrator to manage compliance across Office 365, Exchange Online and SharePoint Online.
Currently many of the compliance features are still accessible through service-specific management interfaces, such as the Exchange Admin Center. However, this will change in the future as more service independent compliance features are added to the Compliance Center.
Consolidating compliance functionality across services into this single area will make compliance features easier to access and enhance your end-to-end task-based experience.
To access the Compliance Center you can either go directly to its URL at or, if you are an Office 365 administrator:

1. Sign in to Office 365;
2. Select the app launcher icon in the upper-left and choose Admin:
3. In the lower-left navigation, expand Admin and choose Compliance:
4. You are then taken into the new Compliance Center:
As mentioned, this is currently being rolled out so it is possible that you will not see all the features in here just yet. For example, there will be an Auditing section as well which is not yet present in my tenant.
Stay tuned for a full article on the new Office 365 Compliance Center!

Tuesday, April 14, 2015

AADSync Performance Counters Error

While working on a project recently, I came across the following error on my AADSync server:

Log Name:      Application
Source:        ADSync
Date:          1/12/2015 12:47:11 PM
Event ID:      6313
Task Category: Server
Level:         Error
Keywords:      Classic
User:          N/A
Description: The server encountered an unexpected error creating performance counters for management agent “ – AAD”.

Performance counters will not be available for this management agent.

To fix this issue in AADSync, you can either perform a clean install (often out of the question) or run the following commands to reload the performance counters:

  1. Stop AADSync’s service;
  2. Delete the following registry key: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ADSync\Performance];
  3. Recreate the Performance key;
  4. Run the following two commands from an elevated command prompt:
    • unlodctr.exe ADSync
    • lodctr.exe “C:\Program Files\Microsoft Azure AD Sync\Bin\mmsperf.ini”
  5. Start AADSync’s service.

Wednesday, April 1, 2015

Exchange Online Protection Quarantine

A decade ago, Bill Gates predicted a spam-free world by 2006. Although we are seeing a small decline in spam, this is unfortunately far from coming true... Exchange Online Protection (EOP) does a great job, in my opinion, at filtering out obvious spam. According to the latest figures from Microsoft, ten million spam messages are blocked every single minute on average by EOP, 10 million! That is an impressive number. However, every day attackers around the world come up with new techniques to fool spam detection engines. Threats take different forms, such as an unidentified spam campaign, unknown malware or a completely new virus. This means that a small percentage (around 3%) of email that is likely to be spam still comes through and are sent to users’ Junk E-mail folder. Users obviously do not want spam in their inboxes, but they often have to review this folder to make sure no good messages (false positives) are mixed in with the bad.
EOP provides two main methods of handling spam detected by its content filters. Administrators can configure it so that spam is sent to the Junk E-mail folder in Outlook and Outlook Web App (OWA), which is the default option, or to direct it into a web-based quarantine.
Sending spam to the Junk folder is the most common choice as that is what users have been using for many years. But from experience I also noticed that this is the case as not everyone is aware of the quarantine feature. On the other hand, some customers have non-Exchange email systems that do not support the Junk E-mail folder approach, have a 3rd party filtering system that sends spam reports to users, or simply prefer the spam quarantine.
Since EOP was launched it has supported spam quarantine, but initially administrators were the only ones who had access to this quarantine, through the Exchange Admin Center, and only they were able to release spam messages... But for some time now administrators can configure EOP to give users self-service management of spam-quarantined messages. So let us have a look at how this works and how we can configure it.
In this article, we will explore the Quarantine feature of EOP, including how to enable, configure and manage it both from the administrator and end user perspectives. To continue reading, please go to the Exchange Online Protection Quarantine article at

Friday, March 27, 2015

Speeding up the Exchange Hybrid wizard in global deployments

If you ever ran the Exchange Hybrid wizard in an environment with servers all over the world, it is likely that it took a few hours to run. But why?
If we look at the wizard’s logs ($exinstall\Logging\Update-HybridConfiguration), we will see that most of the changes are fairly quick. However, it eventually goes on to run a Get-WebServicesVirtualDirectory to analyse the EWS virtual directories (VDs) across all Exchange servers in the environment to determine if any need to be configured. If this comes back true, then the wizard runs the same cmdlet again followed by a Set-WebServicesVirtualDirectory to enable the MRS Proxy for VDs that currently have it disabled. After all the necessary EWS VDs are configured, the wizard runs a Get-WebServicesVirtualDirectory for a third time to validate the configuration/changes made.
The problem here is running the *et-WebServicesVirtualDirectory cmdlet between servers in different countries or even continents. How long does it take for you? Usually it should be a few minutes for each server, but I have seen cases where it takes 30 minutes or more. Now multiply that by the total number of Exchange servers and it can quickly turn into hours and hours...
If, for example, your environment also has Exchange 2007 servers, although these do not use or have the MRS Proxy service, because the wizard simply runs the Get-WebServicesVirtualDirectory cmdlet, this returns 2007 servers (instead of filtering them out...), which contributes to delaying the process.
So, to speed things up a bit, you can manually login to all the servers, enable the MRS Proxy and only then run the Hybrid Wizard. Typically I was only enabling it on the Hybrid servers or servers that I was planning to use for mailbox migration, but the wizard enables it across estate anyway...
To recap, the Mailbox Replication Service Proxy (MRS Proxy) facilitates cross-forest mailbox moves and remote move migrations between an on-premises Exchange organization and Exchange Online. During cross-forest and remote move migrations (aka hybrid migrations), a Client Access server acts as a proxy for incoming move requests for the Mailbox server. The ability of a Client Access server to accept these requests is disabled by default. To allow the Client Access server to accept incoming move requests, we have to enable the MRS Proxy endpoint.

Monday, March 23, 2015

Azure Active Directory Connect Public Preview

The latest version of the Azure AD Connect has been released – the March 2015 Public Preview update.

The Azure AD Connect wizard Public Preview provides a guided experience for integrating one or multiple AD forests with Microsoft Azure AD. Optionally you can configure Exchange Hybrid deployment, password change write-back, ADFS and Web Application Proxy.

Azure AD Connect encompasses functionality that was previously released as DirSync and AADSync. These tools will eventually stop being released individually and all future improvements will be included in updates to Azure AD Connect.

This latest version has been updated with new capabilities, support for additional sync options, Additional Tasks and “Pilot Mode”.

You can download it from the Connect website.

Friday, March 20, 2015

How to change AADSync credentials

When it comes to changing the credentials AADSync uses to connect to the on-premises Active Directory (AD) or to Azure AD, one might think that re-running the wizard and updating the credentials there would do the trick:

However, if you re-run the wizard again, you will see that the old credentials are still being used... So how can we change these credentials?! To do this, we need to use the miisclient.

First, navigate to "install dir"\Microsoft Azure AD Sync\UIShell and run missclient.exe. Then, click in Connectors. Here you will have two connectors, one is used to connect to the local AD and the other to connect to Azure AD:
To update the credentials used to connect to the local AD, double-click the respective connector and then go to Configure Directory Partitions. Here, select Alternate credentials for this directory partition, enter the new credentials and click OK:
To update the credentials used to connect to Azure AD, double-click the respective connector and then go to Connectivity. Here, enter the new credentials and click OK:
Job done!

Thursday, March 19, 2015

Dynamic Distribution Lists in a Hybrid Environment

In a hybrid deployment environment between Exchange Online and on-premises Exchange organizations, neither DirSync nor AADSync can be used to synchronize dynamic distribution lists (DDL) to Exchange Online. Therefore, mailboxes that have been migrated to Exchange Online cannot see DDLs in their GAL or email them.
To work around this issue, create a MailContact in Exchange Online for the DDL, and then grant permissions so that only authenticated senders can submit messages to the new contact. This object should have the following mappings:
On-Premises DDL
Cloud MailContact
You should also consider the scope of the DDL before mailboxes are moved to Exchange Online. If the scope included only mailboxes, the scope must be expanded to include mail users and mail contacts. To do this, open the properties of the DDL and on the membership tab (in Exchange 2013), also select Mail users with external email addresses. If using the Shell, add MailUsers to the IncludedRecipients property of the DDL.
Exchange Online users can now select the DDL from the GAL. When they do, messages will be delivered to the members of the group as defined by the settings for the group.

Friday, March 13, 2015

Exchange Online Protection Conditional Mail Routing

Simply put, Conditional Mail Routing, also known as Criteria Based Routing, is a way of configuring Exchange Online Protection [EOP] connectors in order to send or receive mail a certain way based on the condition of the individual email. For example, we can force TLS for a specific sender or route email based on recipients’ properties to different email server locations.
I was recently working on a global Exchange migration to Office 365. This particular client had an Office 365 tenant hosted in Europe with a Hybrid deployment in India. On top of that, there was a separate Exchange organization deployed across the United Kingdom and the United States of America.
For us to be able to update the MX records to point to the client’s Office 365 tenant, two requirements had to be met because of security and legal reasons:
  • Requirement 1: emails sent from partners to certain business application mailboxes (hosted on-premises in the UK) had to go directly from Office 365 to the Exchange servers in the UK, i.e., without being routed through India;
  • Requirement 2: emails addressed to US users had to go directly from Office 365 to the Exchange server on-premises in the US, i.e., without being routed through India.

To see how we can meet these 2 requirements using Conditional Mail Routing, please check my Exchange Online Protection Conditional Mail Routing article at

Sunday, February 22, 2015

Exchange Online Accept mail for all subdomains feature

In Exchange Online (not on-premises), under your Accepted Domains, you might have seen an option to accept mail for all subdomains:
When this feature is enabled for a domain, emails can be sent and received for subdomains of this domain. For example, if is a provisioned domain and match subdomains support is enabled, users can send emails to or receive emails from,,, and other subdomains.
This feature is usually for customers who have a hybrid environment with mailboxes that reside on-premises as it is only applicable for the Internal Relay domain type.
Following the example above, once the feature is enabled for the domain, Office 365 will be able to deliver emails that are sent to addresses by automatically forwarding them to my on-premises environment (assuming all connectors are in place).
But there is a small catch! Spam. Emails will not get blocked by EOP, meaning spammers can send millions of invalid emails to random addresses and their subdomains in order to try to overwhelm on-premises servers. Having said that, this is the case with most relay scenarios anyway.