Tuesday, April 18, 2017

Block all Outlook versions earlier than X version

For one reason or another, some organizations have the requirement to block older versions of Outlook from connecting to their Exchange environment. This can easily be done on a per-mailbox or on a per-mailbox server basis.

Let’s say we want to block user nuno from accessing his mailbox using all Outlook versions earlier than 11.8010.8036. To achieve this, we run the following cmdlet:
Set-CASMailbox nuno -MAPIBlockOutlookVersions "-11.8010.8036"

When the user tries to use an older version of Outlook, he will get the following message (in this case I blocked all versions of Outlook):

Followed by:

To restore access to the mailbox to any version of Outlook, we simply clear the MAPIBlockOutlookVersion parameter:
Set-CASMailbox nuno -MAPIBlockOutlookVersion $null

To achieve the same but on a per-server basis, we need to use a Registry Key on all servers. In the next example, we create the Disable MAPI Clients registry value to block access to all mailboxes for Outlook clients prior to version 14.0.0:
New-ItemProperty "HKLM:\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem" -Name "Disable MAPI Clients" -PropertyType String -Value "0.0.0-5.9.9, 14.0.0-"

Important: be careful when restricting client access because server-side Exchange components must also use MAPI to log on. Some components report their client version as the component name (such as SMTP or OLE DB), while others report the Exchange build number (such as 6.0.4712.0). For this reason, we must avoid restricting clients that have version numbers that start with 6.x.x.

Outlook Calendar Sharing Error Message

When you use Outlook to share a calendar with another user, you might get one of the following errors:
  • Calendar sharing is not available with the following entries because of permission settings on your network
  • You Cannot request to share calendars with the following people because of permission settings on your network

This typically happens when users type the recipients’ email address or use the address from their cached addresses. When this happens, clear the cached entry for the user, click the TO button and select the intended recipient(s) from the address book.

Sunday, April 9, 2017

How to Check Exchange Prepare AD Values

Before installing a new Exchange environment, and sometimes when updating to a newer Cumulative Update, we need to prepare our Active Directory (AD) forest and its domains. Exchange needs to prepare AD so that it can store information about users' mailboxes and the configuration of Exchange servers in the organization.

Once we have extended the AD Schema, prepared AD, and prepared all its domains, several properties are updated to show that preparation is complete. At this stage, it is strongly recommended to make sure everything has gone smoothly. To do so, typically a tool called Active Directory Service Interfaces Editor (ADSI Edit) is used to check these properties have been updated to the value they should have. Each property needs to match the value for the release of Exchange that we are installing:
  • “In the Schema naming context, verify that the rangeUpper property on ms-Exch-Schema-Verision-Pt is set to the value shown for your version of Exchange 2016 in the Exchange 2016 Active Directory versions table;
  • In the Default naming context, verify that the objectVersion property in the Microsoft Exchange System Objects container under DC=root domain is set to the value shown for your version of Exchange 2016 in the Exchange 2016 Active Directory versions table;
  • In the Configuration naming context, verify that the objectVersion property in the CN=your organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain container is set to the value shown for your version of Exchange 2016 in the Exchange 2016 Active Directory versions table.”
But there are other methods. For example, Michel de Rooij has blogged about checking these properties using LDAP and PowerShell in his Exchange Schema Versions blog post. Additionally, we can use the LDP.exe tool (an LDAP client GUI), or we can use DSQuery.
Dsquery is a command-line tool that is built into Windows Server when the Active Directory Domain Services (AD DS) server role or the AD DS admin tools are installed. Using dsquery we can query AD (shocker!) by using search criteria that we specify.
So, let’s say we want to check the value of the rangeUpper property using dsquery. We can easily do this by running the following command:
DSQUERY.exe * “CN=ms-Exch-Schema-Version-Pt,CN=schema,CN=configuration,DC=domain,DC=com” -Scope base -Attr rangeUpper

In my lab, where I have a nunomota.pt domain, I would run:
DSQUERY.exe * “CN=ms-Exch-Schema-Version-Pt,CN=schema,CN=configuration,DC=nunomota,DC=pt” -Scope base -Attr rangeUpper

To check for the other two properties, we would use the following two queries:
DSQUERY.exe * “CN=Exchange_Org_Name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com” -Scope base -Attr objectVersion

DSQUERY.exe * “CN=Exchange_Org_Name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com” -Scope base -Attr msExchProductId

Now that we know how to easily check these properties, we can make it even easier by automating the check with the script below. In order to automate this, I use the Get-OrganizationConfig Exchange cmdlet to determine the Exchange Organization Name (when it is not specified), so the Exchange Management Shell (EMS) should be used. This means that if we run the script from a normal PowerShell console and do not specify the Exchange Org Name, the script throws an error:
Obviously, if this is a brand-new Exchange environment, the EMS will not yet be available at this stage. In these cases, we must specify the Exchange Org Name and the script will not use the Get-OrganizationConfig cmdlet:
From the values presented by the script, and by looking at the Exchange 2016 Active Directory versions table, we can see that this Exchange Environment is running Exchange 2016 CU3.
The following is the script that produced the output above:
Param (
 [Parameter(Position = 0, Mandatory = $True)]
 [String] $Domain,

 [Parameter(Position = 1, Mandatory = $False)]
 [String] $ExOrgName = (Get-OrganizationConfig).ID

If ($Domain -notmatch "\.") {Write-Host "Please enter a correct Domain name." -ForegroundColor Red; Exit}
ForEach ($name in $Domain.Split(".")) {$tempDomain += ",DC=$($name)"}

$rangeUpper = (DSQUERY.exe * “CN=ms-Exch-Schema-Version-Pt,CN=schema,CN=configuration$($tempDomain)” -Scope base -Attr rangeUpper)[1].Trim()
$objectVersionDNC = (DSQUERY.exe * “CN=Microsoft Exchange System Objects$($tempDomain)” -Scope base -Attr objectVersion)[1].Trim()
$objectVersionCNC = (DSQUERY.exe * “CN=$ExOrgName,CN=Microsoft Exchange,CN=Services,CN=Configuration$($tempDomain)” -Scope base -Attr objectVersion)[1].Trim()
$msExchProductId = (DSQUERY.exe * “CN=$ExOrgName,CN=Microsoft Exchange,CN=Services,CN=Configuration$($tempDomain)” -Scope base -Attr msExchProductId)[1].Trim()

Write-Host "rangeUpper (Schema)           :", $rangeUpper
Write-Host "objectVersion (Default)       :", $objectVersionDNC
Write-Host "objectVersion (Configuration) :", $objectVersionCNC
Write-Host "msExchProductId               :", $msExchProductId

We could take this script one step further and build a list of all these properties and their values for each Exchange CU, and then print which ones match. Because some CUs do not update all these properties, sometimes they will match more than one Exchange version:
Param (
 [Parameter(Position = 0, Mandatory = $True)]
 [String] $Domain,

 [Parameter(Position = 1, Mandatory = $False)]
 [String] $ExOrgName = (Get-OrganizationConfig).ID

If ($Domain -notmatch "\.") {Write-Host "Please enter a correct Domain name." -ForegroundColor Red; Exit}
ForEach ($name in $Domain.Split(".")) {$tempDomain += ",DC=$($name)"}

$rangeUpperHash = @{}
$rangeUpperHash.Add("Exchange 2016 CU3 and CU4", "15326")
$rangeUpperHash.Add("Exchange 2016 CU2", "15325")
$rangeUpperHash.Add("Exchange 2016 CU1", "15323")
$rangeUpperHash.Add("Exchange 2016 Beta and RTM", "15317")
$rangeUpperHash.Add("Exchange 2013 CU7 and later", "15312")
$rangeUpperHash.Add("Exchange 2013 CU6", "15303")
$rangeUpperHash.Add("Exchange 2013 CU5", "15300")
$rangeUpperHash.Add("Exchange 2013 SP1", "15292")
$rangeUpperHash.Add("Exchange 2013 CU3", "15283")
$rangeUpperHash.Add("Exchange 2013 CU2", "15254")
$rangeUpperHash.Add("Exchange 2013 CU1", "15254")
$rangeUpperHash.Add("Exchange 2013 RTM", "15137")

$objVerDNCHash = @{}
$objVerDNCHash.Add("Exchange 2016 Beta to CU4", "13236")
$objVerDNCHash.Add("Exchange 2013 RTM to CU10", "13236")

$objVerCNCHash = @{}
$objVerCNCHash.Add("Exchange 2016 CU4", "16213")
$objVerCNCHash.Add("Exchange 2016 CU2 and CU3", "16212")
$objVerCNCHash.Add("Exchange 2016 CU1", "16211")
$objVerCNCHash.Add("Exchange 2016 RTM", "16210")
$objVerCNCHash.Add("Exchange 2016 Beta", "16041")
$objVerCNCHash.Add("Exchange 2013 CU10 and later", "16130")
$objVerCNCHash.Add("Exchange 2013 CU6 to CU9", "15965")
$objVerCNCHash.Add("Exchange 2013 CU5", "15870")
$objVerCNCHash.Add("Exchange 2013 SP1", "15844")
$objVerCNCHash.Add("Exchange 2013 CU3", "15763")
$objVerCNCHash.Add("Exchange 2013 CU2", "15688")
$objVerCNCHash.Add("Exchange 2013 CU1", "15614")
$objVerCNCHash.Add("Exchange 2013 RTM", "15449")

$msExchProductIdHash = @{}
$msExchProductIdHash.Add("Exchange 2016 CU5", "15.01.0845.032")
$msExchProductIdHash.Add("Exchange 2016 CU4", "15.01.0669.032")
$msExchProductIdHash.Add("Exchange 2016 CU3", "15.01.0544.027")
$msExchProductIdHash.Add("Exchange 2016 CU2", "15.01.0466.034")
$msExchProductIdHash.Add("Exchange 2016 CU1", "15.01.0396.030")
$msExchProductIdHash.Add("Exchange 2016 RTM", "15.01.0225.042")
$msExchProductIdHash.Add("Exchange 2016 Beta", "15.01.0225.017")
$msExchProductIdHash.Add("Exchange 2013 CU15", "15.00.1263.005")
$msExchProductIdHash.Add("Exchange 2013 CU14", "15.00.1236.003")
$msExchProductIdHash.Add("Exchange 2013 CU13", "15.00.1210.003")
$msExchProductIdHash.Add("Exchange 2013 CU12", "15.00.1178.004")
$msExchProductIdHash.Add("Exchange 2013 CU11", "15.00.1156.006")
$msExchProductIdHash.Add("Exchange 2013 CU10", "15.00.1130.007")
$msExchProductIdHash.Add("Exchange 2013 CU9", "15.00.1104.005")
$msExchProductIdHash.Add("Exchange 2013 CU8", "15.00.1076.009")
$msExchProductIdHash.Add("Exchange 2013 CU7", "15.00.1044.025")
$msExchProductIdHash.Add("Exchange 2013 CU6", "15.00.0995.029")
$msExchProductIdHash.Add("Exchange 2013 CU5", "15.00.0913.022")
$msExchProductIdHash.Add("Exchange 2013 SP1", "15.00.0847.032")
$msExchProductIdHash.Add("Exchange 2013 CU3", "15.00.0775.038")
$msExchProductIdHash.Add("Exchange 2013 CU2", "15.00.0712.024")
$msExchProductIdHash.Add("Exchange 2013 CU1", "15.00.0620.029")
$msExchProductIdHash.Add("Exchange 2013 RTM", "15.00.0516.032")

$rangeUpper = (DSQUERY.exe * "CN=ms-Exch-Schema-Version-Pt,CN=schema,CN=configuration$($tempDomain)" -Scope base -Attr rangeUpper)[1].Trim()
$objVerDNC = (DSQUERY.exe * "CN=Microsoft Exchange System Objects$($tempDomain)" -Scope base -Attr objectVersion)[1].Trim()
$objVerCNC = (DSQUERY.exe * "CN=$ExOrgName,CN=Microsoft Exchange,CN=Services,CN=Configuration$($tempDomain)" -Scope base -Attr objectVersion)[1].Trim()
$msExchProductId = (DSQUERY.exe * "CN=$ExOrgName,CN=Microsoft Exchange,CN=Services,CN=Configuration$($tempDomain)" -Scope base -Attr msExchProductId)[1].Trim()

Write-Host "rangeUpper (Schema)           : $rangeUpper ($(($rangeUpperHash.GetEnumerator() | ? {$_.Value -eq $rangeUpper}).Name -join ' | '))"
Write-Host "objectVersion (Default)       : $objVerDNC ($(($objVerDNCHash.GetEnumerator() | ? {$_.Value -eq $objVerDNC}).Name -join ' | '))"
Write-Host "objectVersion (Configuration) : $objVerCNC ($(($objVerCNCHash.GetEnumerator() | ? {$_.Value -eq $objVerCNC}).Name -join ' | '))"
Write-Host "msExchProductId               : $msExchProductId ($(($msExchProductIdHash.GetEnumerator() | ? {$_.Value -eq $msExchProductId}).Name -join ' | '))"

Saturday, March 11, 2017

Exchange DAG Replication Port

Have you ever wondered what TCP port Exchange 2010/2013/2016 uses for database replication (log shipping and seeding)? That would be 64327 by default.

This can be checked using the Get-DatabaseAvailabilityGroup cmdlet:

Administrators can also change this default port is they so desire by using the Set-DatabaseAvailabilityGroup cmdlet with the -ReplicationPort parameter.
If you decide to do so, it is recommended to create a new Windows Firewall rule for the new port on all DAG members before the actual change to avoid any disruption to database replication. After the change, the existing firewall rule can then be deleted or updated (depending on the approach taken):

Thursday, February 16, 2017

Office Online Server Installation Error - Microsoft Setup Bootstrapper has stopped working

I have been facing this same problem for ages now: every time I try to install Office Online Server (the same thing was happening to me when installing Office Web Apps) on a fully patched Windows Server 2012 R2 Standard virtual machine on Hyper-V, with all the requirements specified here, I was getting the following error within a couple of seconds of starting the installation process:
Followed by:

On the Windows Application Event Log, I could see the following error:

Log Name: Application
Source: Application Error
Date: 26/01/2017 17:06:51
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: “server.domain.com”
Description: Faulting application name: MsiExec.exe, version: 5.0.9600.18333, time stamp: 0x572b8067
Faulting module name: KERNELBASE.dll, version: 6.3.9600.18340, time stamp: 0x57366075
Exception code: 0xe06d7363
Fault offset: 0x0000000000008a5c
Faulting process id: 0x404
Faulting application start time: 0x01d277f6952f3bec
Faulting application path: C:\Windows\System32\MsiExec.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: d3dae5ea-e3e9-11e6-80c1-00155d01040e
Faulting package full name:
Faulting package-relative application ID:

I could only find one website with the exact same problem and it said to change the Power Options from High Performance to Balanced. However, that didn’t work for me... I’ve been having this same problem on multiple VMs, in Hyper-V on Windows 8 and Windows 10, both with Office Web Apps and Office Online Server...

It was only thanks to the great MVP community, more precisely fellow MVP Vasil Michev, that I found a solution for this problem. It turns out that there is a missing registry key at HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer. Vasil told me he was experiencing the same issue and that creating the following keys solved the problem for him:
If (!(Test-Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer)) {
 New-Item -Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer | Out-Null

$regProps = Get-ItemProperty -Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer
If (!$regProps.logging) {
 New-ItemProperty -Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer -Name logging -Value voicewarmup -PropertyType String | Out-Null

If (!$regProps.debug) {
 New-ItemProperty -Path HKLM:\SOFTWARE\Policies\Microsoft\Windows\Installer -Name debug -Value 3 -PropertyType DWord | Out-Null

Vasil mentioned he found the answer in this StackExchange.com forum topic, which was in turn answered by Dragan Radovic with a link to his blog post at DevFacto.com.

After creating these keys, I was able to successfully install Office Web Apps:

I still don’t understand why I am having this issue since Office Web Apps on different Hyper-V environments but it seems that it works fine for most people though...

Thank you Vasil!

Sunday, February 12, 2017

Delete IIS Logs Remotely using PowerShell

There are many scripts out there using a variety of methods to delete IIS logs from servers using PowerShell. These scripts are usually written to run locally on servers, which can have two drawbacks:
  1. In environments with a large number of servers, scheduling and maintaining these scripts can require a significant amount of work;
  2. Depending on the security restrictions on production servers, administrators might not be able to schedule this script to run whether they are logged on to the server or not:

This was the scenario I found recently. In order to overcome both obstacles, I decided to run the script from a “script server” where security settings did not prevent cached credentials. Additionally, by deleting these logs remotely I could have a single script targeting multiple servers, making it much easier to manage!

As I have mentioned, there are many ways of deleting IIS logs from a server. The method I have been using lately uses the WebAdministration module and the Get-WebSite cmdlet to get a list of all websites on the local server:

For each website, we can easily check where its logs are being saved to:

So both websites are saving their logs to the same location?! No :) If we use IIS Manager, this path is what we see in the config of the website:

But remember that IIS then creates a subfolder named W3SVC1 (for example) and saves the logs there. This way, each website has a unique log folder. The W3SVCx number refers to the website’s ID. For example, the Default Web Site is usually ID 1, so the log directory would be W3SVC1. The Exchange Back End site will be ID 2 (W3SVC2) and so on. This can be verified in the previous screenshot.

So now, all we have to do is append “\W3SVC” plus the website ID to construct the file path:

It’s also worth adding “.Replace("%SystemDrive%", $env:SystemDrive)” in case the logs are stored in the default location. This way the full path will be “C:\inetpub\logs\LogFiles\W3SVC1” instead of “%SystemDrive%\inetpub\logs\LogFiles\W3SVC1”.

Once we know the location of the files, we can easily delete all that are older than 7 days using the following code:
Get-ChildItem -Path  -Recurse | Where {$_.LastWriteTime -lt (Get-Date).addDays(-7)} | ForEach {del $_.FullName -Confirm:$False}

At the end, the basic script looks like this:
Import-Module WebAdministration

ForEach($webSite in $(Get-WebSite)) {
$dir = "$($webSite.logFile.directory)\W3SVC$($webSite.ID)".Replace("%SystemDrive%", $env:SystemDrive)

 Write-Host "Deleting IIS logs from $dir" -ForegroundColor Green
 Get-ChildItem -Path $dir -Recurse | Where {$_.LastWriteTime -lt (Get-Date).addDays(-7)} | ForEach {del $_.FullName -Confirm:$False}

This is how we would delete IIS logs for all websites on a local server. But what about if we want to delete those logs remotely? Easy! We use Invoke-Command.

The Invoke-Command cmdlet runs commands on a local or remote computer and returns all output from the commands, including errors. By using a single Invoke-Command command, we can run commands on multiple computers.

Using this cmdlet, we easily run the code above against multiple servers using the following code:
Invoke-Command -ComputerName “server1”, “server2”, “server3” -ScriptBlock {
  Import-Module WebAdministration

  ForEach($webSite in $(Get-WebSite)) {
  $dir = "$($webSite.logFile.directory)\W3SVC$($webSite.ID)".Replace("%SystemDrive%", $env:SystemDrive)

    Write-Host "Deleting IIS logs from $dir" -ForegroundColor Green
    Get-ChildItem -Path $dir -Recurse | Where {$_.LastWriteTime -lt (Get-Date).addDays(-7)} | ForEach {del $_.FullName -Confirm:$False}

Simple as that! :)

This code can be significantly improved by passing credentials to the Invoke-Command, by checking if a server is reachable before trying to run a cmdlet against it, and by adding some logging and error handling.

In my case, instead of passing a list of servers using the -ComputerName parameter, I chose to create an array with all my servers, and then process them one by one so I could more easily test connectivity to the server and deal with any errors.
We could also make this part of user input to allow users to specify which servers to action on more easily. Modules would be the next step :)

The final script, which is also available in TechNet Gallery, looks like this:
Delete IIS log files from remote server

The script retrieves the location of IIS logs for all websites on a remote server and deletes those older than $Days days.

Specifies the number of days’ worth of IIS logs to keep on the server

Deletes IIS logs older than 14 days from all servers manually specified within the script's $excServers array
.\Delete-IISlogs.ps1 -Days 14

Name:     Delete-IISlogs.ps1
Author:   Nuno Mota


Param (
 [Parameter(Position = 0, Mandatory = $False)]
 [Int] $Days = 7

Function Write-Log {
 Param ([String] $Type, [String] $Message)

 # Create a log file in the same location as the script containing all the actions taken
 $Logfile = $PSScriptRoot + "\Delete-IISlogs_Log_$(Get-Date -f 'yyyyMMdd').txt"
 If (!(Test-Path $Logfile)) {New-Item $Logfile -Force -ItemType File | Out-Null}

 $timeStamp = (Get-Date).toString("yyyy/MM/dd HH:mm:ss")
 "$timeStamp $Type $Message" | Out-File -FilePath $Logfile -Append
 Write-Verbose $Message

# Script Start

[Array] $excServers = @("server1", "server2", "server3", "server4")
ForEach ($server in $excServers) {
 Write-Log -Type "INF" -Message "Processing $server"
 If (Test-Connection -ComputerName $server -BufferSize 16 -Count 1 -ErrorAction 0 -Quiet) {
  Try {
   $countDel = Invoke-Command -ComputerName $server -ArgumentList $Days, $server -ScriptBlock {
    param($Days, $server)
    [Int] $countDel = 0
    Import-Module WebAdministration
    ForEach($webSite in $(Get-WebSite)) {
        $dir = "$($webSite.logFile.directory)\W3SVC$($webSite.ID)".Replace("%SystemDrive%", $env:SystemDrive)
     Write-Host "Checking IIS logs in $dir on $server" -ForegroundColor Green
     Get-ChildItem -Path $dir -Recurse | ? {$_.LastWriteTime -lt (Get-Date).addDays(-$Days)} | ForEach {
      Write-Host "Deleting", $_.FullName
      del $_.FullName -Confirm:$False
    Return $countDel
   Write-Log -Type "INF" -Message "Deleted $countDel logs from server $server"
  } Catch {
   Write-Log -Type "ERR" -Message "Unable to connect to $($server): $($_.Exception.Message)"
   Send-MailMessage -From "ExchangeAdmin@domain.com” -To "user@domain.com" -Subject "ERROR – Delete IIS Logs" -Body "Unable to connect to $($server): $($_.Exception.Message)" -SmtpServer smtp.domain.com -Priority "High"
 } Else {
  Write-Log -Type "ERR" -Message "Unable to connect to $server"
  Send-MailMessage -From "ExchangeAdmin@domain.com” -To "user@domain.com" -Subject "ERROR – Delete IIS Logs" -Body "Unable to connect to $server" -SmtpServer smtp.domain.com -Priority "High"

Please be aware that we could add even more error handling for cases where we are unable to load the WebAdministration module or delete the files for example.

Friday, February 3, 2017

Disk space missing from Exchange LUN

I was recently troubleshooting an issue where the LUN disk space for one particular Exchange database kept reducing by around 3GB a day, even though the database had plenty of whitespace for use:
Looking at the properties of the mount point, I could see there was indeed 80GB left of free space, so the previous report was accurate:
However, looking at how much the Exchange database and log files were taking, there was supposed to be over 200GB free space!
After some digging around, it turns out this space was being used by Volume Shadow Copies. Using vssadmin tool, I could see 122GB being used by VSS (Volume Shadow Copy Service) for MDB11:

By listing all the shadows, we can check when this shadow copy was created. In my case, it was over a month’s old:
We can also get details regarding shadow copies on Windows servers by using a hidden utility named vssuirun.exe:
It turns out that this particular server was rebooted mid-backup, causing this orphaned shadow copy. Since all the backups were working, I could safely delete this shadow copy. To do this, I tried using the "vssadmin delete shadows /all" command to delete it, but received the following error:
Error: Snapshots were found, but they were outside of your allowed context. Try removing them with the backup application which created them.”
Despite being logged in as an admin, Windows won’t let me touch the shadow copy. Or better put, VSSadmin doesn’t like messing with snapshots taken by other applications. Enter DiskShadow, a “tool that exposes the functionality offered by the Volume Shadow Copy Service (VSS).” Using diskshadow we can double-check the shadow copy details we got with vssadmin:
To delete all shadow copies using diskshadow, we can run "delete shadows all" or, if we want to delete only a particular one (not relevant in this case as there was only one copy), we can specify the ID of the shadow copy we want to delete:
Once it has been deleted, we can confirm there are no more shadow copies lying around using DiskShadow:
Or using VSSadmin:
As expected, the space was then recovered :)

Thursday, January 26, 2017

Unable to connect to POP3

The other day I was troubleshooting an issue where a couple of business applications could not connect to their mailbox using POP3.

First, I made sure that both POP3 services were running across all servers., Then, I enabled POP3 logging by running the following cmdlet:
Set-PopSettings -Server “server_name” -ProtocolLogEnabled $True
And restarted the POP3 service. However, I couldn’t find anything in the logs...

So, I turned to telnet. Using this great tool and by connecting to one particular server, everything seems to be working fine. However, when connecting to another server, I would not get the usual banner, but only a blank screen! OK, definitely something not right with POP3...

It turns out this issue was occurring because the PopProxy component was in Inactive state for this particular server! I found this by running the following cmdlet:
Get-ServerComponentState “server_name” -Component PopProxy

We can use the following cmdlet to determine which requester made PopProxy inactive:
Get-ServerComponentState “server_name” -Component PopProxy).LocalStates

In my case, the requester was HealthAPI that changed the state of PopProxy to Inactive. As such, in order to bring it back to an Active state, all we have to do is run:
Set-ServerComponentState “server_name” -Component PopProxy -State Active -Requester HealthAPI

And all is back to normal! :)

Monday, January 23, 2017

Get-MoveRequest Queued "Job is waiting for resource reservation"

The other day I was trying to move a 5 MB mailbox from one server to another (both running Exchange 2013) when I noticed the move request was stuck on Queued for a long time. The first thing I did was checking the statistics of the move request by running the following cmdlet:
Get-MoveRequest | Get-MoveRequestStatistics | FL

In the stats I could see the following warning/error:
Job is waiting for resource reservation. MRS will continue trying to pick up this request. Details: Resource reservation failed for 'LocalServer/ServerRead' (Processor): load ratio 4.2, load state 'Overloaded', metric 64. This resource is currently unhealthy.

This would change intermittently to the following warning/error:
Message: Resource 'Processor' is unhealthy and shouldn't be accessed.

This could also be seen in the Application Event Log of the server:
Log Name:      Application
Source:        MSExchange Mailbox Replication
Date:          1/19/2017 10:25:09 AM
Event ID:      1121
Task Category: Request
Level:         Error
Computer:      server.domain.com
Description:   The Microsoft Exchange Mailbox Replication service was unable to process a request due to an unexpected error. Request GUID: 'e2f8d856-f258-4cab-a1d1-dde19df2a000' Database GUID: '3479f71f-df65-48ff-a80d-9379495b6aac' Error: Resource 'Processor' is unhealthy and shouldn't be accessed.

The cause is self-explanatory: high CPU usage. When I checked the server’s CPU, this was indeed the case:

The possible workarounds are to stop any CPU-intensive processes (if there are any that can be stopped), investigate what is causing the high CPU and fix it, or wait for the CPU usage to come down at which point the move request will resume automatically.

You may also see similar warnings for other resources such as disk IOPS for example:
Resource reservation failed for 'MdbWrite(“database_name”)' (MdbLatency(“database_name”)): load ratio -1, load state 'Unknown', metric (null).
Resource reservation failed for 'Mailbox Database/MdbWrite' (CiAgeOfLastNotification(“database_name”)): load ratio X.XXXXXXX, load state 'Critical', metric 2147483647. This resource is currently unhealthy.

Thursday, December 1, 2016

Exchange Alerts using Microsoft Teams

Back in September I wrote the Exchange Monitoring Concerns? Pick Up the Slack article for TechGenix (later posted in my blog as Exchange alerting using Slack) about monitoring Exchange and sending mobile alerts to administrators using Slack, a messaging app for teams. At the time, Slack was one of the best apps I could find to easily generate alerts on mobile devices.

A few weeks ago, Microsoft announced Microsoft Teams, a competitor to Slack. From my short experience with it, Microsoft Teams seems to work great, especially since it’s fully integrated with the rest of the Office 365 suite. To learn more about Microsoft Teams, please watch this Microsoft Mechanics video.

The question now is: can we use Microsoft Teams to alert administrators on their mobile devices when something is wrong with their systems or application (such as Exchange)? Let’s find out!

Signing Up to Microsoft Teams
At the time of writing this article, Microsoft Teams is available in preview (since November 2, 2016) to eligible Office 365 commercial customers (Business Essentials, Business Premium, and Enterprise E1, E3, E4 and E5). It is expected the service will become generally available in the first quarter of calendar year 2017.

To turn on Microsoft Teams, IT admins should go to their Office 365 admin center, click Settings, click Organization profile and scroll down to Release preferences. In here, ensure preview features are enabled:
Now click on Apps:

On the list of available apps, search for Microsoft Teams and enable the service, plus all the required features you want to use:

Accessing Microsoft Teams
For some reason, after enabling Microsoft Teams, its icon is still not available in the app launcher:

However, if we navigate to https://teams.microsoft.com we will be able to login to the service just fine.

Similar to Slack, and many other Office 365 applications, Microsoft Team is available in three versions: web app, desktop app, and mobile app.

The purpose of this blog post is not to explain how to use Microsoft Teams (the Microsoft Mechanics video is a great place to start on that), but to see if and how we can use the service to programmatically send alerts to administrators on their mobile devices. But before we do so, we need to use the web or desktop apps to do some initial configuration. So let’s get to it.

Setting Up Microsoft Teams
The first step to configuring Microsoft Teams is to login to https://teams.microsoft.com, select Teams and create our first team by clicking on Create team:
Next we give our new team a name and a description (optional). If we are currently the owner of an Office 365 Group, we get the option to add Teams functionality to that group:
The final step (optional) is to add one or more members to our new team:
If we add users, each will receive an email notifying them they have been added to our new Messaging Team:

We now have our first Team created :)

Each Team can have multiple channels. Channels are how Microsoft Teams organizes conversations. We can set up our channels however we like: by topic, discipline, project, and so on. Channels are open to everyone on the team and contain their own files, OneNote, etc...

So let’s create one channel just for alerts by clicking on ... next to our team’s name and then Add channel:
Give the channel a name and click Add:

We now have our Alerts channel dedicated to Exchange alerts:

Configuring WebHook Connector
Office 365 Connectors are used to get information and content into Microsoft Teams. Any user can connect their team to services like Trello, GitHub, Bing News, Twitter, etc., and get notified of the team's activity in that service. Connectors also provide a way for developers to integrate with Microsoft Teams by building custom incoming WebHook Connectors to generate rich cards within channels.
To generate our alerts to administrators, we will create these cards (messages) by sending an HTTP request with a simple JavaScript Object Notation (JSON) payload to a Microsoft Teams webhook address.

First, we need to create a webhook address for our Alerts channel:
1. From within Microsoft Teams, click ... next to the channel name and then select Connectors:
2. Scroll through the list of connectors to Incoming Webhook, and click Add:
3. Enter a name for the webhook, upload an image to associate with data from the webhook (optional), and select Create:
4. Copy the webhook URL to the clipboard and save it. We will need this URL for sending information to our channel:
 5. Click Done and a message will be visible in the Conversations tab informing all members that an incoming webhook has been configured for the channel:

We now have our webhook configured which we will use to post messages to our Alert channel. If we go into Connectors one more time, we are informed that an Incoming Webhook is already configured and by whom. If we click on Manage we get the options to change its name (which I have changed to AlertsBot) and to remove the webhook.
Please be aware that any member of the team can get the webhook URL and use it to send messages. On top of that, any member can also remove the webhook...

Sending Messages to Microsoft Teams
Now that we have our webhook configured, we need a method to send an HTTP request with a JSON payload to the webhook address. To achieve this, we have two options. The first option is to use cURL, a tool used in command lines or scripts to transfer data with URLs. Since my workstation is 64-bit, I downloaded the Win64 - Generic version (at the time of writing this blog, v7.51.0).
From the command line (not PowerShell), we can use the following command to send a basic “Hello World!” message to our channel:
curl.exe -H “Content-Type: application/json” -d “{\”text\”: \”Hello World!\”}”

If the POST succeeds, we will get a simple 1 returned by cURL:
If we go back to our channel’s Conversation window, we can see the new card posted to the team:
Our first programmatic alert/message to Microsoft Teams! :-D

Doing the same using PowerShell and cURL is a bit more tricky because of the “ (quotes) within the code. In the example above we used \” to escape the quotes, which will not work with PowerShell. The easiest method I found was to put the whole payload in a file (let’s call it alert.json, but we can also use alert.txt for example) and then pass the file into cURL. The file will look like this:
And the code used will be the following:
$webHook = “https://outlook.office365.com/webhook/bcbc68a4-606f-4ebf-8d78-4bbeac2c0c96@ed835685-e329-4799-9a9e-7ec941c92287/IncomingWebhook/(...)"

.\curl.exe -H “Content-Type: application/json” -d “@alert.json” $webHook

The second option to send messages to Microsoft Teams (and a much easier one!), is to simply use PowerShell’s native capabilities with the following two cmdlets:
Invoke-RestMethod: sends HTTP/HTTPS requests to Representational State Transfer (REST) web services;
ConvertTo-Json: converts any object to a string in JSON format. The properties are converted to field names, the field values are converted to property values, and the methods are removed.

Using these two cmdlets, we don’t need cURL anymore. Our previous “Hello World!” example becomes simply the following:

$webHook = “https://outlook.office365.com/webhook/bcbc68a4-606f-4ebf-8d78-4bbeac2c0c96@ed835685-e329-4799-9a9e-7ec941c92287/IncomingWebhook/(...)”

$alert = ConvertTo-JSON @{
text = “Hello World!”

Invoke-RestMethod -ContentType “application/json” -Method Post -body $alert -Uri $webHook

Simple as that! :)

If we manually run the code just to see what the variable $alert contains, we will see that it is in the exact same (JSON) format as our alert.json file:
In our next example, we start to get live data from Exchange and report on it. This simple example just sends a message containing certain details about a mailbox database named MDB01:
$webHook = “https://outlook.office365.com/webhook/bcbc68a4-606f-4ebf-8d78-4bbeac2c0c96@ed835685-e329-4799-9a9e-7ec941c92287/IncomingWebhook/(...)”

$exchDB = Get-MailboxDatabase “MDB01” -Status | Select Name, LastFullBackup, DatabaseSize, Mounted, ServerName
$userCount = (Get-Mailbox -Database $exchDB.Name -ResultSize Unlimited).Count
$size = $($exchDB.DatabaseSize.Split(“(“)[0])

$alert = ConvertTo-Json -Depth 4 @{
  text = “**$($exchDB.Name) Information:**”
  sections = @(
      facts = @(
        name = "Database:"
        value = $exchDB.Name
        name = "Last Bck:"
        value = $exchDB.LastFullBackup
        name = "Size (GB):"
        value = $size
        name = "Mounted?"
        value = $($exchDB.Mounted)
        name = "On Server:"
        value = $exchDB.ServerName
        name = "User Count:"
        value = $userCount

Invoke-RestMethod -ContentType "application/json" -Method Post -body $alert -Uri $webHook

The result will be all the details for MDB01 nicely formatted:
Let’s now look at how we could monitor Exchange’s transport queues and issue an alert if the total number of queued emails goes beyond a certain limit. To achieve this, we get the queues across all servers, exclude any Shadow Redundancy emails, and count the total number of emails across the queues. Then, if that number is above our limit, we send an alert. Obviously, this is a basic script just for demonstration purposes. In a production environment, a few tweaks would likely be required, such as the threshold limit, any queues or servers to include/exclude, use a scheduled task instead perhaps, and so on.
$webHook = “https://outlook.office365.com/webhook/bcbc68a4-606f-4ebf-8d78-4bbeac2c0c96@ed835685-e329-4799-9a9e-7ec941c92287/IncomingWebhook/(...)”

While ($True) {
[Int] $msgCount = 0
Get-TransportService | Get-Queue | Where {$_.MessageCount -gt 0 -and $_.DeliveryType -notlike "Shadow*"} | ForEach {$msgCount += $_.MessageCount}

If ($msgCount -gt 50) {
$alert = ConvertTo-Json -Depth 1 @{
text = “**High Mail Queues!**`nTotal queued emails: $msgCount”

Invoke-RestMethod -ContentType "application/json" -Method Post -body $alert -Uri $webHook
Start-Sleep –Seconds 1800

The result will be the following alert:
Microsoft Teams Mobile Client
The purpose of this post was not to test Microsoft Teams itself, but to test if it can be used to reliably alert administrators on their mobile devices with any potential issues with Exchange.

We already established that we can easily generate alerts to a Teams’ channel, but what we now need to test is Microsoft Teams’ mobile app. To show how platform-independent Microsoft Teams is, I am going to use an iPhone to test the mobile app (trust me, I am not an Apple fan) :)

We start by searching and downloading the app from the Apple Store:
Once we open the app for the first time, we are asked to sign in:
After signing in we are presented with four tips, the last one being about notifications, which for the purpose of this article, we should obviously enable:

Once we sign in we can easily see the message we previously sent:
According to Microsoft, the best way to make sure people see our message is to @mention them. We do this by typing @ before a name and choosing the person we want to mention from the picker. They will generate a notification and a number will appear next to the channel we mentioned them in. When we mention someone, no one else will receive a notification, but everyone in the team will be able to see that we @mentioned someone though. Alternatively, we can also mention the entire channel, which will make everyone receive a notification.

At this stage it seems that I only get an alert on my mobile phone when someone mentions me:
Notice my name highlighted in red and the “@” at the top right hand corner of the message:
So, I had a look at the Notification configuration in the app and found the following:
I was hoping that by enabling everything I would start getting a notification on my phone for everything, but this was not the case. I still don’t get an alert unless someone mentions me, or the entire channel, sends me a direct message, replies to a message of mine or likes something I posted.

The problem is that I haven’t been able to find a way of mentioning someone or a channel using cURL or PowerShell... :( This means that users will get the messages on their mobile devices but not a notification, making this method not suitable for what I am trying to achieve... This, of course, until I find a way of mentioning someone using JSON!

I found an article on Bot Builder for .NET that has a section about Activities and Mention Entities. This article states that we can mention someone using a JSON like this:

  "entities": [{ 
    "mentioned": { 
      "id": "UV341235", "name":"Color Bot"
    "text": "@ColorBot"

But I still haven’t been able to make it work using PowerShell (or any other method for that matter!)...

So using both the web and desktop Team apps, I sent a few messages where I mentioned someone and captured the JSON payload to see exactly how these are constructed and sent. For example, the following message:
Translates into the following JSON:
So it should just be a matter of building an identical (except maybe for the ClientMessageID property) JSON, right? Nope... Unfortunately, even sending an identical JSON using PowerShell results in the exact same message but without the mention...

Surely I am missing something here because of my lack of JSON knowledge and experience... As such, it might be that adding or changing something really simple will make it work! As is, I can’t yet use Teams to reliably alert me until I figure how to mention someone...

Nonetheless, Microsoft Teams is an awesome product and I am looking forward to explore it even further!