Exchange Hybrid #550 5.1.1 RESOLVER.ADR.ExRecipNotFound; not found ##

There might be cases where in Exchange hybrid deployments where Exchange Online and on-premises users have the same e-mail namespace (such as alias@domain.com), Office 365 users are unable to send e-mails to on-premises users and receive a nondelivery report (NDR) error message similar to:

 

#550 5.1.1 RESOLVER.ADR.ExRecipNotFound; not found ##

 

Amongst other possible reasons, this can be caused if the domain set up in the hybrid deployment is not configured as a shared domain in Office 365. To correct this problem, follow these steps using the Exchange Admin Center in the new Office 365:

1.       Sign in to the Office 365 portal as a global admin;

2.       In the header, click Admin, and then click Exchange;

3.       In the left navigation pane of the Exchange Admin Center, click mail flow and then click accepted domains;

4.       Select the domain that is set up for the hybrid deployment, and then click Details;

5.       Select Shared, and then click save.

 

The EAC the domains that you added to your account through the Microsoft Office 365 portal. It lets you manage how messages are delivered. In a hybrid scenario, Exchange Online must be set up correctly so that when a cloud-based user sends an e-mail to an on-premises user, Exchange Online routes the e-mail to the on-premises messaging environment.

Office 365 Service Comparison

The following page can be used to filter and compare features of Office 365 plans, Office 365 services and on-premises products, such as comparing every feature of Exchange Server 2013 on-premises vs. Exchange online: Office 365 service comparison.

This is extremely useful when exploring Exchange Online (or any Office 365 service for that matter) and trying to decide if Exchange Online meets all the organization’s requirements.

 

Exchange 2013 StalledDueToCI Error

When running some tests on Exchange 2013 Cumulative Update 1, namely migrating Public Folders from Exchange 2010 to 2013, I noticed the migration was taking too long. When querying more details regarding the migration request using the cmdlet Get-PublicFolderMigrationRequest | Get-PublicFolderMigrationRequestStatistics -IncludeReport, I noticed the following error:

The job is currently stalled due to ‘Content Indexing’ lagging behind on resource ‘CiAgeOfLastNotification(Mailbox Database 1415171211)’

When checking the ContentIndexState for this database, it was in a FailedAndSuspended state...

It seems there is a bug in Exchange 2013 RTM and CU1 which causes Exchange to not setup all the groups necessary in Active Directory. Specifically for this case is the group “ContentSubmitters”. As such, if you are experiencing this problem, you need to:

    1. Manually create the group in AD under the existing exchange groups OU;

    2. Change the security on the group and allow Administrators and NetworkService accounts Full Control;

    3. Restart the Microsoft Exchange Search and the Microsoft Exchange Search Host Controller services on every Mailbox server.

Now, you should notice the mailbox databases content index status changing from Failed to Healthy, with the migrations starting to transfer at a much better speed.

Exchange 2013 DAG with Dynamic Quorum

Windows Server 2012 introduced a new quorum model called Failover Clustering Dynamic Quorum, which we can use with Exchange. When using Dynamic Quorum, the cluster dynamically manages the vote assignment to nodes based on the state of each node. When a node shuts down or crashes, it loses its quorum vote. When a node successfully re-joins the cluster, it regains its quorum vote. By dynamically adjusting the assignment of quorum votes, the cluster can increase or decrease the number of quorum votes that are required to keep it running. This enables the cluster to maintain availability during sequential node failures or shutdowns.

 

With a dynamic quorum, the cluster quorum majority is determined by the set of nodes that are active members of the cluster at any time. This is an important distinction from the cluster quorum in Windows Server 2008 R2 where the quorum majority is fixed, based on the initial cluster configuration.

 

The advantage this brings, is that it is now possible for a cluster to run even if the number of nodes remaining in the cluster is less than 50%! By dynamically adjusting the quorum majority requirement, the cluster can sustain sequential node shutdowns down to a single node and still keep running. It does not allow the cluster to sustain a simultaneous failure of a majority of voting members though. To continue running, the cluster must always have a quorum majority at the time of a node shutdown or failure.

 

 

The following picture shows a DAG still operational even though two out of three servers are offline:

 

 

To read more about this feature, including tests with an Exchange 2013 DAG, please check my Exchange 2013 DAGwith Dynamic Quorum article on MSExchange.org.

Last Logon Information in Exchange 2013

If we want to check when a user last logged on to their mailbox in Exchange 2007 and 2010 we have to use the Exchange Management Shell [EMS] and the following cmdlet:

Get-MailboxStatistics "User" | Select LastLoggedOnUserAccount, LastLogonTime

Note that the LastLoggedOnUserAccount property indicates the account last used to log on to the mailbox. This could be a user with FullAccess permissions to the mailbox, a delegate or even someone simply checking the user’s Calendar!

 

With Exchange 2013 part of this information is now available through the Exchange Administration Centre as well. If you:

    1.       Navigate to Recipients and then Mailboxes;

    2.       Double-click on the user you want to check this information for;

    3.       Select the Mailbox Usage tab and in there you will see the Last Logon date and time.

However, to check who actually logged on, we still need to use the EMS...

Office 365 Network Analysis Tool

With BPOS and the first iteration of Office 365, Microsoft provided the Speed Test web tool to assist in determining if an organization’s internet connection could support Microsoft Online services, mainly focusing on Exchange Online and Lync Online. This tool was available from www.microsoftspeedtest.com.

However, for some reason, this tool has not been available for many months now, leaving organizations and IT professionals without a tool to evaluate the impact of Microsoft Online services on their internet connection.

Fortunately, the Office 365 Network Analysis Tool is now available! Similar to Speed Test, this new tool is also web-based and runs several checks such as port connectivity, route paths between the client and tenant, bandwidth speed and capacity tests, and VoIP readiness. At the end of the test the tool generates a nice report.

 

This tool can be found at:

North America
   http://na.deployoffice365.com (URL not yet available)
   http://na1-fasttrack.cloudapp.net

EMEA
   http://emea.deployoffice365.com (URL not yet available)
   http://em1-fasttrack.cloudapp.net

APAC
   http://apac.deployoffice365.com (URL not yet available)
   http://ap1-fasttrack.cloudapp.net

How to Trigger a Full Password Sync in DirSync

In Windows Azure Active Directory Synchronization Tool (or DirSync), a full Password Sync and a full Directory Sync are two different events. A full Password Sync synchronizes password hashes for all DirSync users, while a full Directory Sync does not trigger a full password sync. By default, the only activity that triggers a full password sync is completing the DirSync’s Configuration Wizard.

But there is a way around this. In order to trigger a full password sync, perform the following steps (you must be using DirSync v6438.0003 or above):

1. On the DirSync server, run the following .psc1: C:\Program Files\Windows Azure Active Directory Sync\DirSyncConfigShell.psc1;
2. In the new Powershell console, run Set-FullPasswordSync;
3. Now load the services console by running Services.msc;
4. Restart the Forefront Identity Manager Synchronization Service Service.

Once this is complete, you should see a series of 656 EventIDs (Password Sync Requests) and 657 EventIDs (Password Sync Results) indicating that a full password sync was triggered.

Exchange 2013 Message Headers

As you might have seen in the Exchange 2013 Mail Flow article, the Transport Pipeline is now made of three main services:

1. Front End Transport service, which runs on all Client Access servers (CAS) and acts as a stateless proxy for all inbound and outbound external SMTP traffic;
2. Transport service which runs on all Mailbox servers and is almost identical to the Hub Transport server role in previous versions of Exchange;
3. Mailbox Transport service which also runs on all Mailbox servers and is made of two separate services, the Mailbox Transport Delivery service and the Mailbox Transport Submission service.

Because of this change in architecture, the message headers in Exchange 2013 have been updated to include information regarding which service(s) dealt with the message, making it easier to troubleshoot any possible issues.

 

For example, if we look at an e-mail received from the Internet, into a CAS server and then delivered to a user’s mailbox, we can see that both the Front End Transport service and the Mailbox Transport service dealt with the message:

 

 

On the other hand, if we look at an internal e-mail, we do not see the Front End Transport service as the e-mail does not go through this service if it is from an internal user to another internal user:

Exchange 2013 E-mail Traffic From inboundproxy@inboundproxy.com and HealthMailbox

In Exchange 2013, native, built-in monitoring and recovery actions are included in a feature called Managed Availability, which is made up of two processes: the Microsoft Exchange Health Manager Service (MSExchangeHMHost.exe) and the Microsoft Exchange Health Manager Worker process (MSExchangeHMWorker.exe), and the following components:

• The Probe Engine takes measurements on the server;
• The Monitoring Probe Engine stores the business logic about what constitutes a healthy state. It functions like a pattern recognition engine, looking for patterns and measurements that differ from a healthy state, and then evaluating whether a component or feature is unhealthy;
• The Responder Engine, when alerted about an unhealthy component, first tries to recover that component. The first attempt may be to restart the application pool, the second attempt may be to restart the corresponding service and the third attempt may be to restart the server. The final attempt may be to put the server offline, so that it no longer accepts traffic.

Exchange 2013 automatically creates several HealthMailbox"guid" objects in Active Directory which are used by Managed Availability to send e-mails through Exchange to verify mail flow every few minutes. These e-mails are used to do health checks for resources from FrontEnd Transport service to Transport Service and health checks on mailbox database resources. The same Microsoft Exchange Health Manager Service is responsible for both these health checks.

Two health mailboxes are created for each mailbox database: one for mailboxes and one for Public Folders (if these are deployed). You can view these hidden mailboxes using Active Directory Users and Computers. You need to enable Advanced Features and then navigate to Microsoft Exchange System Objects and then Monitoring Mailboxes:

Or you can use the Exchange Management Shell and run the following cmdlet:

Get-Mailbox -Monitoring

 

 

This is why in an Exchange 2013 environment you will see lots of traffic from the e-mail addresses inboundproxy@inboundproxy.com, MailDeliveryProbe@MailDeliveryProbe.com and HealthMailbox(...)@domain.com with the subjects of “MBTSubmission/StoreDriverSubmission/(...)-MapiSubmitLAMProve”, “Inbound proxy probe” or “Client submission probe”.

This is by design!

Windows Azure Active Directory Synchronization Tool Version Release History

If you are finding it hard to keep up with all the improvements made to DirSync as well as all its versions, this Wiki keeps track of the versions that have been released and lists the main changes introduced.

Outlook Web App Views

As with previous versions of Exchange, Outlook Web App [OWA] in Exchange 2013 and the new Office 365 comes in two flavors:

• OWA Premium: available in computers, tablets and phone optimized UI. Desktops must be using Internet Explorer 8+ and newer versions of Safari, Chrome and Firefox. As for tablets and phones, the supported versions are (for now) Windows 8 tablets, iOS6 on iPad2+ and iPhone 4+. Although not supported, the new OWA UI works just fine on recent Android mobile phones and tablets;
• OWA Light: this is what users get on any browser not supported with OWA Premium. It uses a very simple HTML4 based UI which works in pretty much any browser in existence and remains the same as with the previous release of Exchange/OWA. A special note to say that IE7 now gets OWA Light as well...

OWA Premium has three different views:

• The traditional Desktop view with a 3-columns mouse-based UI;
• The Table view with a 2-columns touch UI also known as twide;
• The Phone view with a 1-column touch UI also known as tnarrow.

Even if you do not have a tablet or a phone to test how these two different views look like, you can easily use your desktop browser to emulate them. To do so, use the following URLs:

• https://<FQDN>/owa/?layout=twide
• https://<FQDN>/owa/?layout=tnarrow

Exchange Online Mailbox Size Now 50GB!

The size of user mailboxes in all Exchange Online and Office 365 service plans just doubled!

 

Since last week (end of August), the 25GB of mailbox storage increased to 50GB. There is no price increase associated with this change as the doubling of mailbox storage is simply part of Microsoft's promise to continuously deliver value to Office 365 customers. 

 

Customers don't need to do anything to take advantage of this new mailbox size as mailbox sizes will automatically be increased. This increase has already started and continues through November.

 

This increase to a 50GB mailbox per user benefits Exchange Online and Office 365 service plans, including: Exchange Online Plan 1, Office 365 Small Business, Midsize Business, Enterprise E1, Government G1, and Education A1.

Customers with a premium service plan (Exchange Online Plan 2, Office 365 Enterprise E3 and E4, Government G3 and G4, Education A3 and A4), already enjoy unlimited e-mail storage through personal archive, but now the default primary mailbox size is increasing to 50GB. Kiosk user mailboxes are doubling in size too, from 1 to 2GB.

 

In addition to doubling the storage of users' primary mailboxes, Microsoft has also increased the size of other mailboxes: shared mailboxes and resource mailboxes are both increasing to 10GB (more than double), but the size of Site mailboxes remains unchanged.

 

For complete details about mailbox types and sizes, read the Office 365 Service Descriptions.

 

Source: Office 365 Technology

MS Exchange CON 2013 Virtual Conference

Get your top Microsoft Exchange questions answered!

- Hear from a top analyst from Osterman Research with the latest survey research on Exchange top trends and challenges;
- Watch how vendors are solving some of the biggest Exchange Management problems;
- Get answers to your top Exchange and Exchange 2013 questions with an Exchange MVP.

All from the convenience of your home/office, on September 12, 2013!

Discover answers to questions like:
- What are the key features of Exchange 2013?
- How can we secure and better control our Exchange environment?
- What are 5 strategies to better manage Exchange for 2013 and beyond?

This unique, online conference is limited to 1,000 participants, so register now if you have not already done so, register here.

How to Configure Public and Private Computer Settings in OWA 2013

The new Exchange 2013 Outlook Web App (OWA) logon page no longer allows users to select whether they are using a public or a private computer. By default, OWA 2013 assumes users are using a private computer the default timeout of 8 hours is used. This timeout specifies how long a user can be inactive before requiring him/her to sign in again.

The LogonPagePublicPrivateSelectionEnabled parameter in the Set-OWAVirtualDirectory cmdlet specifies whether the OWA sign-in page includes this private/public computer sign-in option. While by default this parameter is set to True in Exchange 2010, in 2013 it is set to False. To change this on server CAS1, simply run the following cmdlets:

Set-OwaVirtualDirectory “CAS1\owa (Default Web Site)” -LogonPagePublicPrivateSelectionEnabled $True
IISreset /noforce

Similarly to previous versions of Exchange, the default timeout for private computers is still 8 hours for public computers 15 minutes. You can change this by running the following cmdlets to create the necessary registry keys:

Set-ItemProperty “HKLM:\SYSTEM\CurrentControlSet\Services\MSExchange OWA” -Name PrivateTimeout -Value "timeout_minutes" -Type DWORD

Set-ItemProperty “HKLM:\SYSTEM\CurrentControlSet\Services\MSExchange OWA” -Name PublicTimeout -Value "timeout_minutes" -Type DWORD

IISreset /noforce

Error Deleting Database "Failed to remove monitoring mailbox object"

When removing databases from Exchange 2013, you might get the following error if the correct procedures are not followed:

Failed to remove monitoring mailbox object of database “database_name”. Exception: Active directory operation failed on “server_name”. This error is not retrievable. Additional information: Access is denied. Active directory response: 000000005: SecErr: DSID-031520B2, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0.

In this case, the database was removed an Active Directory [AD] error (with a not very useful description) complaining about insufficient permissions is thrown. If you run:

Get-Mailbox -Monitoring

You will most likely see a warning regarding a corrupted Health Mailbox:

WARNING: The object “domain_name”/Microsoft Exchange System Objects/Monitoring Mailboxes/”Health_Mailbox_GUID” has been corrupted, and it's in an inconsistent state. The following validation errors happened: WARNING: Database is mandatory or UserMailbox.

Because Exchange 2013 did not have sufficient permissions to the domainname/Microsoft Exchange System Objects/Monitoring Mailboxes Organizational Unit [OU], it could not delete the AD objects related to the database’s health mailboxes. In this case, the database attribute is null because the database the health mailbox references no longer exists.

To fix this issue, simply delete the health mailboxes referenced by the error(s) from that OU by using Active Directory Users and Computers. After removing these, the warning should be gone.

Deleting health mailboxes is a low risk procedure because they will be automatically re-created by the Microsoft Exchange Health Manager service on the Exchange 2013 server hosting the database when this service is restarted.

Exchange Cmdlet Extension Agents

Cmdlet Extension Agents are Exchange components that are invoked by cmdlets when the cmdlets run. These agents are available on any server role and, as the name suggests, they extend the capabilities of cmdlets that invoke them by assisting in processing data or performing additional actions based on the requirements of the cmdlet. They can be used to modify, replace or extend functionality of any Exchange cmdlet. For example, they can override a value provided by a user, perform actions outside of the cmdlet workflow, provide a value for a required parameter that was not provided, etc.

An example of such functionality is when creating a new mailbox. If an administrator does not specify a mailbox database when creating a new mailbox with Exchange 2007, the process fails. However, in Exchange 2010 and 2013, the New-Mailbox cmdlet invokes the Mailbox Resources Management agent whenever it is run. If a database is not specified, this agent automatically determines a suitable database on which to create the new mailbox and inserts that value into the Database parameter.

Note that cmdlet extension agents can only be invoked by Exchange 2010 and 2013 cmdlets as they are a feature introduced in Exchange 2010. Scripts cannot invoke these agents directly but if they contain Exchange 2010/2013 cmdlets, those cmdlets continue to call the cmdlet extension agents.

Unfortunately, cmdlet extension agents are not a well-known feature and, therefore, are often overlooked or ignored. A good example is the requirement that many organizations have to customize things immediately after creating a mailbox. They often develop custom scripts or even buy 3rd-party applications to achieve this, when cmdlet extension agents can very well be used to achieve most, if not all, these requirements.

In this article I use a particular cmdlet extension agent, called Scripting agent, in order to customize the mailbox provisioning process. Specifically, when a new mailbox is created, the agent will automatically configure archiving, permissions, Single Item Recovery, ActiveSync, other user attributes such as country and city, and send a welcome e-mail to the user.

To continue reading, please check the Cmdlet Extension Agents article on MSExchange.org.

 

Hyper-V Replica with Exchange Supportability

If you are considering whether to use Hyper-V Replica with Exchange, remember that Exchange has its own disaster recovery and site awareness capabilities that should be used instead.

 

Hyper-V Replica is not needed and is not supported because of problems with Exchange roles being forced back in time in the event of an unplanned failover.

DirSync Error "Microsoft.Online.PasswordSynchronization.Cryptography.dll Could Not Be Loaded"

The Directory Synchronization Tool (DirSync) is supported on Windows Server 2008 SP1 and higher. However, the Password Sync feature introduced in the latest version will not function correctly if installed on an OS older than Windows Server 2008 R2 SP1 (Windows Server 2008 R2 SP1 itself is supported).

When this is the case, you will receive the following error:

Password synchronization failed for domain: "your domain".
Details: System.IO.FileLoadException: A procedure imported by 'Microsoft.Online.PasswordSynchronization.Cryptography.dll' could not be loaded.

To resolve this issue, install the Directory Sync tool on a supported Windows Server OS.

eDiscovery During Exchange 2010/2013 Coexistence

When transitioning from Exchange 2010 to 2013, you should be aware of the following in regarding to the eDiscovery feature during the coexistence period.

 

Administrator audit logs and metadata for eDiscovery searches (not search results) are stored in a system (arbitration) mailbox name SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9} with the display name of Microsoft Exchange.

 

For eDiscovery to work during the period while Exchange 2010 and 2013 coexist (and after the transition), you need to move this system mailbox to an Exchange 2013 server. You can do this using the EAC or through PowerShell:

Get-Mailbox -Arbitration "SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}" | New-MoveRequest -TargetDatabase “Exchange_2013_database”

When the move finishes, to check if everything went OK, run the following cmdlet and ensure the Server and Database values refer to Exchange 2013:

Get-Mailbox -Arbitration “SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}” | FL ServerName, Database

If you do not move this system mailbox to Exchange 2013, the following issues will occur when Exchange 2010 and Exchange 2013 coexist in your Exchange organization:

• Exchange 2013 tasks are not saved to the administrator audit log. When you run the Search-AdminAuditLog cmdlet or try to export the administrator audit log in the EAC, you will receive an error that says you cannot create an administrator audit log search because this system mailbox is located on a server that is not running Exchange 2013. A Microsoft Exchange error with an Event ID of 5000 is also logged in the Windows Application log each time a command is run;
• You cannot run eDiscovery searches using the EAC or the Shell in Exchange 2013. Mailbox searches can be created and queued, but they cannot be started. An error with an Event ID of 6 is logged in the MsExchange Management log, stating that the Start-MailboxSearch cmdlet failed. However, you can search mailboxes using the Shell and the Exchange Control Panel in Exchange 2010.

Exchange 2013 “Please restart the Microsoft Information Store service”

With Exchange 2013 Cumulative Update 1, administrators will notice that every time a new database is added, they are prompted to restart the Information Store service. Although this was also true in Exchange 2013 RTM, the difference was that before administrators did not receive this restart warning.

 

As we saw previously, Exchange 2013 introduced the new Managed Store, which uses a different memory management model than previous editions of Exchange. With a Store Worker Process for each active and passive database present on a server, it is now required to restart the Information Store service because the Store only determines the amount of memory that it will use to manage each database when it starts (which happens when the server starts or when the Information Store service is restarted).

 

In previous editions, Exchange typically seizes as much memory as it is available on a server and uses that memory to cache Store data. In Exchange 2013 this approach was revised and it now calculates how much memory it should use and makes it available to worker processes (obviously active databases are assigned more memory than passive databases). However, this new memory management approach depends on knowing how many worker processes are in use. When databases are added from a server, the Store does not re-calculate the amount of RAM for each worker process dynamically. This does not mean that you cannot mount the new database – it means caching will not be as efficient as it should be until the next time the Store process restarts and memory use is adjusted.

This might be seen as a big drawback of Exchange 2013, and it might even change in the future, but adding or removing databases should not happen that often, so the impact should not be that massive.