Tuesday, April 10, 2018

Exchange Online Mailbox Audit Improvements: UpdateFolderPermissions

A few weeks ago, Microsoft added a new action to the Exchange mailbox auditing: UpdateFolderPermissions. As the name suggests, when this action is being logged, it records changes to folder permissions, be that Owner, Delegates, or Admin.

Microsoft has updated the default mailbox audit configuration to include the UpdateFolderPermissions action. In the following screenshot, we can see that auditing has not been enabled for the ServiceDesk shared mailbox, but UpdateFolderPermissions is part of the default auditing configuration for all 3 access types:

Existing mailboxes that have not deviated from the default configuration will be automatically updated to include the UpdateFolderPermissions action. My own mailbox has had the default auditing settings enabled for a long time, and now UpdateFolderPermissions was automatically added as well:

With mailbox audit configured with this action, we will find records in the audit logs when permissions of folders are added, deleted or modified. These records can be found both in the unified audit log:

Or in the mailbox audit log through the Search-MailboxAuditlog cmdlet.

In the following screenshot, we are searching the mailbox audit logs for the AddFolderPermissions action. Using these logs, we can see that I (LogonType: Owner) used OWA to give ServiceDesk Owner rights to my Clutter folder, and that I used Outlook to give them Reviewer rights to my Archive folder:

Besides AddFolderPermissions, we can also track ModifyFolderPermissions and RemoveFolderPermissions actions:

Friday, March 16, 2018

The Microsoft Exchange Mailbox Replication service was unable to process jobs in a mailbox database - EventID 1006

After deleting a mailbox database from Exchange 2013 in one of my test environment, I was getting the following warning in the Application log of the server hosting that database:
Log Name:      Application
Source:        MSExchange Mailbox Replication
Date:          3/13/2018 7:16:44 AM
Event ID:      1006
Task Category: Service
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      server.domain.com
Description:   The Microsoft Exchange Mailbox Replication service was unable to process jobs in a mailbox database.
Database: Missing database (a435e9be-b010-400d-9d56-659065d6c9df)
Error: Database 'a435e9be-b010-400d-9d56-659065d6c9df' doesn't exist.

After running the cmdlet below, I confirmed that the database mentioned in the warning didn’t exist (so it was safe to assume it was the one I deleted):
Get-MailboxDatabase | FT Name, GUID -Auto

Most of the times, you can resolve this by simply restarting the Microsoft Exchange Mailbox Replication service or the Information Store service. Alternatively, rebooting the server will stop the warning since all the required services will be restarted.

Friday, March 9, 2018

Understanding Office 365 Exchange Online migration endpoints

For a few years now, Microsoft has provided pretty much the same types of migration for organizations to use to migrate email data to Exchange Online in Office 365 from on-premises messaging systems, be that Exchange or not. Additionally, end users themselves can also import their own email, contacts, and other mailbox information to an Office 365 mailbox created for them, but that’s typically only used in very small migrations.

Part of the migration process, excluding some third-party migrations, is the configuration and use of one or more Exchange Online migration endpoints. These are simply a connector from Office 365 to the on-premises email system, and that’s what this article focuses on.

To read the full article, please head over to TechGenix or click on this link: Understanding Office 365 Exchange Online migration endpoints.

Friday, December 15, 2017

Search-MessageTrackingReport Error: You aren't authorized to perform the search

We can use the Search-MessageTrackingReport cmdlet to retrieve the message tracking report for a particular email. With this report ID, we can then pass it on to the Get-MessageTrackingReport cmdlet to get the full tracking information for that email.
However, if you are not used to these cmdlets, you might come across the following error when running your search:
Search-MessageTrackingReport nuno.mota@domain.com -Recipients jonathan.sinasac@domain.com

You aren't authorized to perform the search. You need to have permissions to search for the sender of the message or for every recipient specified.

This is because, by default, users can only track messages that they send or receive from their own mailbox. To overcome this error, simply use the BypassDelegateChecking switch which allows us to track messages for any user.

Sunday, October 15, 2017

Exchange Mailbox Move History

A common task amongst many Exchange administrators around the world is moving users’ mailboxes between databases. This could be to move the user to a database with a higher quota limit, because the user moved to a different location so we want to move his mailbox to an Exchange server closer to his new location, simply because of an Exchange transition/migration, or for many other reasons.

Whatever the case might be, Exchange keeps track of these mailbox moves in case we need to find out, for example, where a mailbox was located before. To access this information, we need to run the Get-MailboxStatistics cmdlet against the mailbox we want to check and use the IncludeMoveHistory switch, such as follows (some output has been removed for brevity):
Get-MailboxStatistics nuno –IncludeMoveHistory | FL
(Get-MailboxStatistics nuno –IncludeMoveHistory).MoveHistory

Status                           : Completed
Flags                            : IntraOrg, Pull
SourceDatabase                   : MDB2
SourceVersion                    : Version 14.3 (Build 224.0)
SourceServer                     : server2.nunomota.pt
SourceArchiveDatabase            :
SourceArchiveVersion             : Version 0.0 (Build 0.0)
SourceArchiveServer              :
TargetDatabase                   : MDB01
TargetVersion                    : Version 15.0 (Build 1076.0)
TargetServer                     : server1.nunomota.pt
TargetArchiveDatabase            :
TargetArchiveVersion             : Version 0.0 (Build 0.0)
TargetArchiveServer              :
BadItemLimit                     : 0
BadItemsEncountered              : 0
LargeItemLimit                   : 0
LargeItemsEncountered            : 0
QueuedTimestamp                  : 6/18/2017 12:51:20 PM
StartTimestamp                   : 6/18/2017 12:51:26 PM
FinalSyncTimestamp               : 6/18/2017 12:52:17 PM
CompletionTimestamp              : 6/18/2017 12:55:16 PM
OverallDuration                  : 00:03:55.2744307
TotalFinalizationDuration        : 00:02:57.7932767
TotalSuspendedDuration           : 00:00:00
TotalFailedDuration              : 00:00:00
TotalQueuedDuration              : 00:00:00.7499765
TotalInProgressDuration          : 00:03:54.5244542
TotalStalledDueToHADuration      : 00:00:00
TotalTransientFailureDuration    : 00:00:00
MRSServerName                    : server2.nunomota.pt
TotalMailboxSize                 : 51.25 MB (53,742,458 bytes)
TotalMailboxItemCount            : 669
Message                          :
FailureTimestamp                 :
Report                           :

“The IncludeMoveHistory switch specifies whether to return additional information about the mailbox that includes the history of a completed move request, such as status, flags, target database, bad items, start times, end times, duration that the move request was in various stages, and failure codes.”

The number of moves that are kept in Exchange will depend what version of Exchange you are running. For Exchange 2010 the default value is 2 while for 2013 it is 5. However, this number can be customized by editing the Mailbox Replication Service (MRS) configuration. To do this, open the MsExchangeMailboxReplication.exe.config file, which by default is located at C:\Program Files\Microsoft\Exchange Server\V15\Bin, using Notepad with Admin rights. Then, go to the MRSConfiguration section and update the MaxMoveHistoryLength setting with a value between 0 and 100.

To get a higher level of detail, we can instead use the IncludeMoveReport switch. This switch specifies whether to return a verbose detailed move report for a completed move request, such as server connections and move stages.

Sunday, September 17, 2017

Outlook Calendar Sharing Error Message

When you use Outlook to share a calendar with another user, you might get one of the following errors:
  • Calendar sharing is not available with the following entries because of permission settings on your network;
  • You cannot request to share calendars with the following people because of permission settings on your network.

This typically happens when users type the recipients’ email address or use the address from their cached addresses. When this happens, clear the cached entry for the user, click the TO button and select the intended recipient(s) from the address book.

Emails disappear from Outlook Inbox after being read

It is not the first time that users or clients come to me complaining that as soon as they read an email in their Inbox folder in Outlook, the email disappears when they select another email or folder.

Every time this is caused by the user (somehow!) creating a view filter to only display unread emails:

Once the filter is removed, everything is back to normal!

Wednesday, July 19, 2017

The program used to create this object is Outlook

Many Outlook users are facing a number of problems after applying the recent June 2017 security updates.

In one of these issues, when you open an attachment in an email, contact, or task formatted as Rich Text you get the following error:
The program used to create this object is Outlook. That program is either not installed on your computer or it is not responding. To edit this object, install Outlook or ensure that any dialog boxes in Outlook are closed

Microsoft has acknowledged the problems and suggested that users make use of the following workarounds until the issue is resolved.

1. Forward the email to yourself and then open the attachments from the forwarded email;
2. Change the email format to HTML, or Text format;
3. Save the attachments to your computer, using one of the following methods, then open them from the saved location:
   - Drag and drop the attachments to your desktop;
   - Go to File -> Save Attachments;
   - Copy and paste the attachment to your computer.

This and other issues caused by the June 2017 updates are detailed in the Outlook known issues in the June 2017 security updates page. Microsoft said they will update this page when fixes are available, so keep an eye on it!

Wednesday, May 17, 2017

An error occurred while using SSL configuration for endpoint

The other day, one of my Exchange 2016 lab servers stopped working. Well, I say "stopped working" but most things seemed to be working except for the Exchange Management Shell where I would get the following error when opening it:
OWA and ECP were also not working: I would simply get a blank page after signing in...
In the event log, there were hundreds of 15021 event errors complaining about SSL configuration:
Log Name:      System
Source:        Microsoft-Windows-HttpEvent
Date:          17/05/2017 03:51:17
Event ID:      15021
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX1.nunomota.pt
Description:   An error occurred while using SSL configuration for endpoint The error status code is contained within the returned data.
As you can imagine, problems with SSL connections can affect multiple Exchange components such as the shell, ECP and OWA in this case (and many others if I had checked I'm sure). These problems can be the result of certificates not installed or incorrectly installed, and should be deleted from the system and reinstalled with the appropriate information.
So, I checked the installed certificates on this server by running netsh http show sslcert:
Nothing suspicious at first sight, but then again, I can't remember the last time I ran this command so I wasn't exactly sure what to look for. Until I ran the same command on a healthy server and noticed that the certificates used for 443 and 444 were the same:
So I went back to the affected server, deleted the certificate assigned to by running netsh http delete sslcert ipport= and assigned the same certificate as the one used on port 443 by running netsh httpadd sslcert ipport= certhash="certificate_hash" appid="application id":
Restarted the server and everything was back to normal! :)

Monday, May 15, 2017

Exchange Meeting Room Statistics

A while back I wrote an article named Exchange Meeting Room Statistics about a script to gather statistics regarding Exchange meeting room usage for MSExchange.org. For this script to work, we have to give ourselves FullAccess to the meeting rooms’ mailbox, add them into our Outlook profile, and then use an Outlook COM Object to connect to Outlook and gather this information. Far from ideal, especially when trying to analyse dozens of rooms!

I have finally written a new version that uses Exchange Web Services to gather the same information, plus some further stats. All the script requires is for AutoDiscover to be working, the EWS Managed API to be installed, and for the user running the script to have Reviewer permissions to the meeting rooms’ calendar (FullAccess permissions to the room’s mailbox will also work).

UPDATE (15/12/2017): I have updated the script to also work with Exchange Online (Office 365). If you want to analyse meeting rooms in EXO, simply add the -ExchangeOnline switch when running the script.

This new script, available in the TechNet Gallery, will gather statistics such as the number of meetings during the specified times, the total and average meeting duration (in minutes), the total and average number of attendees, how many meetings started in the morning and afternoon, how many recurring meetings, and the 5 five organizers and attendees. It will export all the stats to a CSV file and also print in on the screen:
PS C:\Scripts\> .\Get-MeetingRoomStats.ps1 -RoomListSMTP "room.1@domain.com, room.2@domain.com" -From "03/01/2017" -To "04/01/2017"

From         : 01/Mar/17 0:00:00
To           : 01/Apr/17 0:00:00
RoomEmail    : room.1@domain.com
RoomName     : IT - 16 Floor - Room 16.23
Meetings     : 104
Duration     : 4920
AvgDuration  : 47
TotAttendees : 442
AvgAttendees : 4
RecAttendees : 383
OptAttendees : 59
AMtotal      : 46
AMperc       : 44
PMtotal      : 58
PMperc       : 56
RecTotal     : 38
RecPerc      : 37
TopOrg       : user.1@domain.com (12), user.2@domain.com (9), user.3@domain.com (9), user.4@domain.com (7), user.5@domain.com (5), user.6@domain.com
(4), user.7@domain.com (4), user.8@domain.com (4), user.9@domain.com (4), user.10@domain.com (3),
TopAtt       : user.2@domain.com (25), user.4@domain.com (23), user.1@domain.com (19), user.3@domain.com (16), user.11@domain.com (16),
user.12@domain.com (15), user.9@domain.com (12), user.13@domain.com (11), user.14@domain.com (9), user.15@domain.com (9),

From         : 01/Mar/17 0:00:00
To           : 01/Apr/17 0:00:00
RoomEmail    : room.2@domain.com
RoomName     : IT - 16 Floor - Room 16.24
Meetings     : 121
Duration     : 6178
AvgDuration  : 51
TotAttendees : 570
AvgAttendees : 5
RecAttendees : 537
OptAttendees : 33
AMtotal      : 45
AMperc       : 37
PMtotal      : 76
PMperc       : 63
RecTotal     : 42
RecPerc      : 35
TopOrg       : user.16@domain.com (9), user.17@domain.com (8), user.10@domain.com (8), user.18@domain.com (7), user.19@domain.com (6),
user.20@domain.com (5), user.21@domain.com (5), user.22@domain.com (4), user.6@domain.com (4), user.23@domain.com (4),
TopAtt       : user.24@domain.com (22), user.4@domain.com (20), user.17@domain.com (17), user.25@domain.com (16), user.16@domain.com (15),
user.11@domain.com (12), user.26@domain.com (12), user.21@domain.com (12), user.27@domain.com (11), user.28@domain.com (11),

You can download the script from here.

Tuesday, April 18, 2017

Block all Outlook versions earlier than X version

For one reason or another, some organizations have the requirement to block older versions of Outlook from connecting to their Exchange environment. This can easily be done on a per-mailbox or on a per-mailbox server basis.

Let’s say we want to block user nuno from accessing his mailbox using all Outlook versions earlier than 11.8010.8036. To achieve this, we run the following cmdlet:
Set-CASMailbox nuno -MAPIBlockOutlookVersions "-11.8010.8036"

When the user tries to use an older version of Outlook, he will get the following message (in this case I blocked all versions of Outlook):

Followed by:

To restore access to the mailbox to any version of Outlook, we simply clear the MAPIBlockOutlookVersion parameter:
Set-CASMailbox nuno -MAPIBlockOutlookVersion $null

To achieve the same but on a per-server basis, we need to use a Registry Key on all servers. In the next example, we create the Disable MAPI Clients registry value to block access to all mailboxes for Outlook clients prior to version 14.0.0:
New-ItemProperty "HKLM:\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem" -Name "Disable MAPI Clients" -PropertyType String -Value "0.0.0-5.9.9, 14.0.0-"

Important: be careful when restricting client access because server-side Exchange components must also use MAPI to log on. Some components report their client version as the component name (such as SMTP or OLE DB), while others report the Exchange build number (such as 6.0.4712.0). For this reason, we must avoid restricting clients that have version numbers that start with 6.x.x.

Outlook Calendar Sharing Error Message

When you use Outlook to share a calendar with another user, you might get one of the following errors:
  • Calendar sharing is not available with the following entries because of permission settings on your network
  • You Cannot request to share calendars with the following people because of permission settings on your network

This typically happens when users type the recipients’ email address or use the address from their cached addresses. When this happens, clear the cached entry for the user, click the TO button and select the intended recipient(s) from the address book.

Sunday, April 9, 2017

How to Check Exchange Prepare AD Values

Before installing a new Exchange environment, and sometimes when updating to a newer Cumulative Update, we need to prepare our Active Directory (AD) forest and its domains. Exchange needs to prepare AD so that it can store information about users' mailboxes and the configuration of Exchange servers in the organization.

Once we have extended the AD Schema, prepared AD, and prepared all its domains, several properties are updated to show that preparation is complete. At this stage, it is strongly recommended to make sure everything has gone smoothly. To do so, typically a tool called Active Directory Service Interfaces Editor (ADSI Edit) is used to check these properties have been updated to the value they should have. Each property needs to match the value for the release of Exchange that we are installing:
  • “In the Schema naming context, verify that the rangeUpper property on ms-Exch-Schema-Verision-Pt is set to the value shown for your version of Exchange 2016 in the Exchange 2016 Active Directory versions table;
  • In the Default naming context, verify that the objectVersion property in the Microsoft Exchange System Objects container under DC=root domain is set to the value shown for your version of Exchange 2016 in the Exchange 2016 Active Directory versions table;
  • In the Configuration naming context, verify that the objectVersion property in the CN=your organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain container is set to the value shown for your version of Exchange 2016 in the Exchange 2016 Active Directory versions table.”
But there are other methods. For example, Michel de Rooij has blogged about checking these properties using LDAP and PowerShell in his Exchange Schema Versions blog post. Additionally, we can use the LDP.exe tool (an LDAP client GUI), or we can use DSQuery.
Dsquery is a command-line tool that is built into Windows Server when the Active Directory Domain Services (AD DS) server role or the AD DS admin tools are installed. Using dsquery we can query AD (shocker!) by using search criteria that we specify.
So, let’s say we want to check the value of the rangeUpper property using dsquery. We can easily do this by running the following command:
DSQUERY.exe * “CN=ms-Exch-Schema-Version-Pt,CN=schema,CN=configuration,DC=domain,DC=com” -Scope base -Attr rangeUpper

In my lab, where I have a nunomota.pt domain, I would run:
DSQUERY.exe * “CN=ms-Exch-Schema-Version-Pt,CN=schema,CN=configuration,DC=nunomota,DC=pt” -Scope base -Attr rangeUpper

To check for the other two properties, we would use the following two queries:
DSQUERY.exe * “CN=Exchange_Org_Name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com” -Scope base -Attr objectVersion

DSQUERY.exe * “CN=Exchange_Org_Name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com” -Scope base -Attr msExchProductId

Now that we know how to easily check these properties, we can make it even easier by automating the check with the script below. In order to automate this, I use the Get-OrganizationConfig Exchange cmdlet to determine the Exchange Organization Name (when it is not specified), so the Exchange Management Shell (EMS) should be used. This means that if we run the script from a normal PowerShell console and do not specify the Exchange Org Name, the script throws an error:
Obviously, if this is a brand-new Exchange environment, the EMS will not yet be available at this stage. In these cases, we must specify the Exchange Org Name and the script will not use the Get-OrganizationConfig cmdlet:
From the values presented by the script, and by looking at the Exchange 2016 Active Directory versions table, we can see that this Exchange Environment is running Exchange 2016 CU3.
The following is the script that produced the output above:
Param (
 [Parameter(Position = 0, Mandatory = $True)]
 [String] $Domain,

 [Parameter(Position = 1, Mandatory = $False)]
 [String] $ExOrgName = (Get-OrganizationConfig).ID

If ($Domain -notmatch "\.") {Write-Host "Please enter a correct Domain name." -ForegroundColor Red; Exit}
ForEach ($name in $Domain.Split(".")) {$tempDomain += ",DC=$($name)"}

$rangeUpper = (DSQUERY.exe * “CN=ms-Exch-Schema-Version-Pt,CN=schema,CN=configuration$($tempDomain)” -Scope base -Attr rangeUpper)[1].Trim()
$objectVersionDNC = (DSQUERY.exe * “CN=Microsoft Exchange System Objects$($tempDomain)” -Scope base -Attr objectVersion)[1].Trim()
$objectVersionCNC = (DSQUERY.exe * “CN=$ExOrgName,CN=Microsoft Exchange,CN=Services,CN=Configuration$($tempDomain)” -Scope base -Attr objectVersion)[1].Trim()
$msExchProductId = (DSQUERY.exe * “CN=$ExOrgName,CN=Microsoft Exchange,CN=Services,CN=Configuration$($tempDomain)” -Scope base -Attr msExchProductId)[1].Trim()

Write-Host "rangeUpper (Schema)           :", $rangeUpper
Write-Host "objectVersion (Default)       :", $objectVersionDNC
Write-Host "objectVersion (Configuration) :", $objectVersionCNC
Write-Host "msExchProductId               :", $msExchProductId

We could take this script one step further and build a list of all these properties and their values for each Exchange CU, and then print which ones match. Because some CUs do not update all these properties, sometimes they will match more than one Exchange version:
Param (
 [Parameter(Position = 0, Mandatory = $True)]
 [String] $Domain,

 [Parameter(Position = 1, Mandatory = $False)]
 [String] $ExOrgName = (Get-OrganizationConfig).ID

If ($Domain -notmatch "\.") {Write-Host "Please enter a correct Domain name." -ForegroundColor Red; Exit}
ForEach ($name in $Domain.Split(".")) {$tempDomain += ",DC=$($name)"}

$rangeUpperHash = @{}
$rangeUpperHash.Add("Exchange 2016 CU3 and CU4", "15326")
$rangeUpperHash.Add("Exchange 2016 CU2", "15325")
$rangeUpperHash.Add("Exchange 2016 CU1", "15323")
$rangeUpperHash.Add("Exchange 2016 Beta and RTM", "15317")
$rangeUpperHash.Add("Exchange 2013 CU7 and later", "15312")
$rangeUpperHash.Add("Exchange 2013 CU6", "15303")
$rangeUpperHash.Add("Exchange 2013 CU5", "15300")
$rangeUpperHash.Add("Exchange 2013 SP1", "15292")
$rangeUpperHash.Add("Exchange 2013 CU3", "15283")
$rangeUpperHash.Add("Exchange 2013 CU2", "15254")
$rangeUpperHash.Add("Exchange 2013 CU1", "15254")
$rangeUpperHash.Add("Exchange 2013 RTM", "15137")

$objVerDNCHash = @{}
$objVerDNCHash.Add("Exchange 2016 Beta to CU4", "13236")
$objVerDNCHash.Add("Exchange 2013 RTM to CU10", "13236")

$objVerCNCHash = @{}
$objVerCNCHash.Add("Exchange 2016 CU4", "16213")
$objVerCNCHash.Add("Exchange 2016 CU2 and CU3", "16212")
$objVerCNCHash.Add("Exchange 2016 CU1", "16211")
$objVerCNCHash.Add("Exchange 2016 RTM", "16210")
$objVerCNCHash.Add("Exchange 2016 Beta", "16041")
$objVerCNCHash.Add("Exchange 2013 CU10 and later", "16130")
$objVerCNCHash.Add("Exchange 2013 CU6 to CU9", "15965")
$objVerCNCHash.Add("Exchange 2013 CU5", "15870")
$objVerCNCHash.Add("Exchange 2013 SP1", "15844")
$objVerCNCHash.Add("Exchange 2013 CU3", "15763")
$objVerCNCHash.Add("Exchange 2013 CU2", "15688")
$objVerCNCHash.Add("Exchange 2013 CU1", "15614")
$objVerCNCHash.Add("Exchange 2013 RTM", "15449")

$msExchProductIdHash = @{}
$msExchProductIdHash.Add("Exchange 2016 CU5", "15.01.0845.032")
$msExchProductIdHash.Add("Exchange 2016 CU4", "15.01.0669.032")
$msExchProductIdHash.Add("Exchange 2016 CU3", "15.01.0544.027")
$msExchProductIdHash.Add("Exchange 2016 CU2", "15.01.0466.034")
$msExchProductIdHash.Add("Exchange 2016 CU1", "15.01.0396.030")
$msExchProductIdHash.Add("Exchange 2016 RTM", "15.01.0225.042")
$msExchProductIdHash.Add("Exchange 2016 Beta", "15.01.0225.017")
$msExchProductIdHash.Add("Exchange 2013 CU15", "15.00.1263.005")
$msExchProductIdHash.Add("Exchange 2013 CU14", "15.00.1236.003")
$msExchProductIdHash.Add("Exchange 2013 CU13", "15.00.1210.003")
$msExchProductIdHash.Add("Exchange 2013 CU12", "15.00.1178.004")
$msExchProductIdHash.Add("Exchange 2013 CU11", "15.00.1156.006")
$msExchProductIdHash.Add("Exchange 2013 CU10", "15.00.1130.007")
$msExchProductIdHash.Add("Exchange 2013 CU9", "15.00.1104.005")
$msExchProductIdHash.Add("Exchange 2013 CU8", "15.00.1076.009")
$msExchProductIdHash.Add("Exchange 2013 CU7", "15.00.1044.025")
$msExchProductIdHash.Add("Exchange 2013 CU6", "15.00.0995.029")
$msExchProductIdHash.Add("Exchange 2013 CU5", "15.00.0913.022")
$msExchProductIdHash.Add("Exchange 2013 SP1", "15.00.0847.032")
$msExchProductIdHash.Add("Exchange 2013 CU3", "15.00.0775.038")
$msExchProductIdHash.Add("Exchange 2013 CU2", "15.00.0712.024")
$msExchProductIdHash.Add("Exchange 2013 CU1", "15.00.0620.029")
$msExchProductIdHash.Add("Exchange 2013 RTM", "15.00.0516.032")

$rangeUpper = (DSQUERY.exe * "CN=ms-Exch-Schema-Version-Pt,CN=schema,CN=configuration$($tempDomain)" -Scope base -Attr rangeUpper)[1].Trim()
$objVerDNC = (DSQUERY.exe * "CN=Microsoft Exchange System Objects$($tempDomain)" -Scope base -Attr objectVersion)[1].Trim()
$objVerCNC = (DSQUERY.exe * "CN=$ExOrgName,CN=Microsoft Exchange,CN=Services,CN=Configuration$($tempDomain)" -Scope base -Attr objectVersion)[1].Trim()
$msExchProductId = (DSQUERY.exe * "CN=$ExOrgName,CN=Microsoft Exchange,CN=Services,CN=Configuration$($tempDomain)" -Scope base -Attr msExchProductId)[1].Trim()

Write-Host "rangeUpper (Schema)           : $rangeUpper ($(($rangeUpperHash.GetEnumerator() | ? {$_.Value -eq $rangeUpper}).Name -join ' | '))"
Write-Host "objectVersion (Default)       : $objVerDNC ($(($objVerDNCHash.GetEnumerator() | ? {$_.Value -eq $objVerDNC}).Name -join ' | '))"
Write-Host "objectVersion (Configuration) : $objVerCNC ($(($objVerCNCHash.GetEnumerator() | ? {$_.Value -eq $objVerCNC}).Name -join ' | '))"
Write-Host "msExchProductId               : $msExchProductId ($(($msExchProductIdHash.GetEnumerator() | ? {$_.Value -eq $msExchProductId}).Name -join ' | '))"

Saturday, March 11, 2017

Exchange DAG Replication Port

Have you ever wondered what TCP port Exchange 2010/2013/2016 uses for database replication (log shipping and seeding)? That would be 64327 by default.

This can be checked using the Get-DatabaseAvailabilityGroup cmdlet:

Administrators can also change this default port is they so desire by using the Set-DatabaseAvailabilityGroup cmdlet with the -ReplicationPort parameter.
If you decide to do so, it is recommended to create a new Windows Firewall rule for the new port on all DAG members before the actual change to avoid any disruption to database replication. After the change, the existing firewall rule can then be deleted or updated (depending on the approach taken):

Thursday, February 16, 2017

Office Online Server Installation Error - Microsoft Setup Bootstrapper has stopped working

I have been facing this same problem for ages now: every time I try to install Office Online Server (the same thing was happening to me when installing Office Web Apps) on a fully patched Windows Server 2012 R2 Standard virtual machine on Hyper-V, with all the requirements specified here, I was getting the following error within a couple of seconds of starting the installation process:
Followed by:

On the Windows Application Event Log, I could see the following error:

Log Name: Application
Source: Application Error
Date: 26/01/2017 17:06:51
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: “server.domain.com”
Description: Faulting application name: MsiExec.exe, version: 5.0.9600.18333, time stamp: 0x572b8067
Faulting module name: KERNELBASE.dll, version: 6.3.9600.18340, time stamp: 0x57366075
Exception code: 0xe06d7363
Fault offset: 0x0000000000008a5c
Faulting process id: 0x404
Faulting application start time: 0x01d277f6952f3bec
Faulting application path: C:\Windows\System32\MsiExec.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: d3dae5ea-e3e9-11e6-80c1-00155d01040e
Faulting package full name:
Faulting package-relative application ID:

I could only find one website with the exact same problem and it said to change the Power Options from High Performance to Balanced. However, that didn’t work for me... I’ve been having this same problem on multiple VMs, in Hyper-V on Windows 8 and Windows 10, both with Office Web Apps and Office Online Server...

It was only thanks to the great MVP community, more precisely fellow MVP Vasil Michev, that I found a solution for this problem. It turns out that there is a missing registry key at HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer. Vasil told me he was experiencing the same issue and that creating the following keys solved the problem for him:
If (!(Test-Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer)) {
 New-Item -Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer | Out-Null

$regProps = Get-ItemProperty -Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer
If (!$regProps.logging) {
 New-ItemProperty -Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer -Name logging -Value voicewarmup -PropertyType String | Out-Null

If (!$regProps.debug) {
 New-ItemProperty -Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer -Name debug -Value 3 -PropertyType DWord | Out-Null

Vasil mentioned he found the answer in this StackExchange.com forum topic, which was in turn answered by Dragan Radovic with a link to his blog post at DevFacto.com.

After creating these keys, I was able to successfully install Office Web Apps:

I still don’t understand why I am having this issue since Office Web Apps on different Hyper-V environments but it seems that it works fine for most people though...

Thank you Vasil!

Sunday, February 12, 2017

Delete IIS Logs Remotely using PowerShell

There are many scripts out there using a variety of methods to delete IIS logs from servers using PowerShell. These scripts are usually written to run locally on servers, which can have two drawbacks:
  1. In environments with a large number of servers, scheduling and maintaining these scripts can require a significant amount of work;
  2. Depending on the security restrictions on production servers, administrators might not be able to schedule this script to run whether they are logged on to the server or not:

This was the scenario I found recently. In order to overcome both obstacles, I decided to run the script from a “script server” where security settings did not prevent cached credentials. Additionally, by deleting these logs remotely I could have a single script targeting multiple servers, making it much easier to manage!

As I have mentioned, there are many ways of deleting IIS logs from a server. The method I have been using lately uses the WebAdministration module and the Get-WebSite cmdlet to get a list of all websites on the local server:

For each website, we can easily check where its logs are being saved to:

So both websites are saving their logs to the same location?! No :) If we use IIS Manager, this path is what we see in the config of the website:

But remember that IIS then creates a subfolder named W3SVC1 (for example) and saves the logs there. This way, each website has a unique log folder. The W3SVCx number refers to the website’s ID. For example, the Default Web Site is usually ID 1, so the log directory would be W3SVC1. The Exchange Back End site will be ID 2 (W3SVC2) and so on. This can be verified in the previous screenshot.

So now, all we have to do is append “\W3SVC” plus the website ID to construct the file path:

It’s also worth adding “.Replace("%SystemDrive%", $env:SystemDrive)” in case the logs are stored in the default location. This way the full path will be “C:\inetpub\logs\LogFiles\W3SVC1” instead of “%SystemDrive%\inetpub\logs\LogFiles\W3SVC1”.

Once we know the location of the files, we can easily delete all that are older than 7 days using the following code:
Get-ChildItem -Path  -Recurse | Where {$_.LastWriteTime -lt (Get-Date).addDays(-7)} | ForEach {del $_.FullName -Confirm:$False}

At the end, the basic script looks like this:
Import-Module WebAdministration

ForEach($webSite in $(Get-WebSite)) {
$dir = "$($webSite.logFile.directory)\W3SVC$($webSite.ID)".Replace("%SystemDrive%", $env:SystemDrive)

 Write-Host "Deleting IIS logs from $dir" -ForegroundColor Green
 Get-ChildItem -Path $dir -Recurse | Where {$_.LastWriteTime -lt (Get-Date).addDays(-7)} | ForEach {del $_.FullName -Confirm:$False}

This is how we would delete IIS logs for all websites on a local server. But what about if we want to delete those logs remotely? Easy! We use Invoke-Command.

The Invoke-Command cmdlet runs commands on a local or remote computer and returns all output from the commands, including errors. By using a single Invoke-Command command, we can run commands on multiple computers.

Using this cmdlet, we easily run the code above against multiple servers using the following code:
Invoke-Command -ComputerName “server1”, “server2”, “server3” -ScriptBlock {
  Import-Module WebAdministration

  ForEach($webSite in $(Get-WebSite)) {
  $dir = "$($webSite.logFile.directory)\W3SVC$($webSite.ID)".Replace("%SystemDrive%", $env:SystemDrive)

    Write-Host "Deleting IIS logs from $dir" -ForegroundColor Green
    Get-ChildItem -Path $dir -Recurse | Where {$_.LastWriteTime -lt (Get-Date).addDays(-7)} | ForEach {del $_.FullName -Confirm:$False}

Simple as that! :)

This code can be significantly improved by passing credentials to the Invoke-Command, by checking if a server is reachable before trying to run a cmdlet against it, and by adding some logging and error handling.

In my case, instead of passing a list of servers using the -ComputerName parameter, I chose to create an array with all my servers, and then process them one by one so I could more easily test connectivity to the server and deal with any errors.
We could also make this part of user input to allow users to specify which servers to action on more easily. Modules would be the next step :)

The final script, which is also available in TechNet Gallery, looks like this:
Delete IIS log files from remote server

The script retrieves the location of IIS logs for all websites on a remote server and deletes those older than $Days days.

Specifies the number of days’ worth of IIS logs to keep on the server

Deletes IIS logs older than 14 days from all servers manually specified within the script's $excServers array
.\Delete-IISlogs.ps1 -Days 14

Name:     Delete-IISlogs.ps1
Author:   Nuno Mota


Param (
 [Parameter(Position = 0, Mandatory = $False)]
 [Int] $Days = 7

Function Write-Log {
 Param ([String] $Type, [String] $Message)

 # Create a log file in the same location as the script containing all the actions taken
 $Logfile = $PSScriptRoot + "\Delete-IISlogs_Log_$(Get-Date -f 'yyyyMMdd').txt"
 If (!(Test-Path $Logfile)) {New-Item $Logfile -Force -ItemType File | Out-Null}

 $timeStamp = (Get-Date).toString("yyyy/MM/dd HH:mm:ss")
 "$timeStamp $Type $Message" | Out-File -FilePath $Logfile -Append
 Write-Verbose $Message

# Script Start

[Array] $excServers = @("server1", "server2", "server3", "server4")
ForEach ($server in $excServers) {
 Write-Log -Type "INF" -Message "Processing $server"
 If (Test-Connection -ComputerName $server -BufferSize 16 -Count 1 -ErrorAction 0 -Quiet) {
  Try {
   $countDel = Invoke-Command -ComputerName $server -ArgumentList $Days, $server -ScriptBlock {
    param($Days, $server)
    [Int] $countDel = 0
    Import-Module WebAdministration
    ForEach($webSite in $(Get-WebSite)) {
        $dir = "$($webSite.logFile.directory)\W3SVC$($webSite.ID)".Replace("%SystemDrive%", $env:SystemDrive)
     Write-Host "Checking IIS logs in $dir on $server" -ForegroundColor Green
     Get-ChildItem -Path $dir -Recurse | ? {$_.LastWriteTime -lt (Get-Date).addDays(-$Days)} | ForEach {
      Write-Host "Deleting", $_.FullName
      del $_.FullName -Confirm:$False
    Return $countDel
   Write-Log -Type "INF" -Message "Deleted $countDel logs from server $server"
  } Catch {
   Write-Log -Type "ERR" -Message "Unable to connect to $($server): $($_.Exception.Message)"
   Send-MailMessage -From "ExchangeAdmin@domain.com” -To "user@domain.com" -Subject "ERROR – Delete IIS Logs" -Body "Unable to connect to $($server): $($_.Exception.Message)" -SmtpServer smtp.domain.com -Priority "High"
 } Else {
  Write-Log -Type "ERR" -Message "Unable to connect to $server"
  Send-MailMessage -From "ExchangeAdmin@domain.com” -To "user@domain.com" -Subject "ERROR – Delete IIS Logs" -Body "Unable to connect to $server" -SmtpServer smtp.domain.com -Priority "High"

Please be aware that we could add even more error handling for cases where we are unable to load the WebAdministration module or delete the files for example.

Friday, February 3, 2017

Disk space missing from Exchange LUN

I was recently troubleshooting an issue where the LUN disk space for one particular Exchange database kept reducing by around 3GB a day, even though the database had plenty of whitespace for use:
Looking at the properties of the mount point, I could see there was indeed 80GB left of free space, so the previous report was accurate:
However, looking at how much the Exchange database and log files were taking, there was supposed to be over 200GB free space!
After some digging around, it turns out this space was being used by Volume Shadow Copies. Using vssadmin tool, I could see 122GB being used by VSS (Volume Shadow Copy Service) for MDB11:

By listing all the shadows, we can check when this shadow copy was created. In my case, it was over a month’s old:
We can also get details regarding shadow copies on Windows servers by using a hidden utility named vssuirun.exe:
It turns out that this particular server was rebooted mid-backup, causing this orphaned shadow copy. Since all the backups were working, I could safely delete this shadow copy. To do this, I tried using the "vssadmin delete shadows /all" command to delete it, but received the following error:
Error: Snapshots were found, but they were outside of your allowed context. Try removing them with the backup application which created them.”
Despite being logged in as an admin, Windows won’t let me touch the shadow copy. Or better put, VSSadmin doesn’t like messing with snapshots taken by other applications. Enter DiskShadow, a “tool that exposes the functionality offered by the Volume Shadow Copy Service (VSS).” Using diskshadow we can double-check the shadow copy details we got with vssadmin:
To delete all shadow copies using diskshadow, we can run "delete shadows all" or, if we want to delete only a particular one (not relevant in this case as there was only one copy), we can specify the ID of the shadow copy we want to delete:
Once it has been deleted, we can confirm there are no more shadow copies lying around using DiskShadow:
Or using VSSadmin:
As expected, the space was then recovered :)