Thursday, December 1, 2016

Exchange Alerts using Microsoft Teams

Back in September I wrote the Exchange Monitoring Concerns? Pick Up the Slack article for TechGenix (later posted in my blog as Exchange alerting using Slack) about monitoring Exchange and sending mobile alerts to administrators using Slack, a messaging app for teams. At the time, Slack was one of the best apps I could find to easily generate alerts on mobile devices.

A few weeks ago, Microsoft announced Microsoft Teams, a competitor to Slack. From my short experience with it, Microsoft Teams seems to work great, especially since it’s fully integrated with the rest of the Office 365 suite. To learn more about Microsoft Teams, please watch this Microsoft Mechanics video.

The question now is: can we use Microsoft Teams to alert administrators on their mobile devices when something is wrong with their systems or application (such as Exchange)? Let’s find out!

Signing Up to Microsoft Teams
At the time of writing this article, Microsoft Teams is available in preview (since November 2, 2016) to eligible Office 365 commercial customers (Business Essentials, Business Premium, and Enterprise E1, E3, E4 and E5). It is expected the service will become generally available in the first quarter of calendar year 2017.

To turn on Microsoft Teams, IT admins should go to their Office 365 admin center, click Settings, click Organization profile and scroll down to Release preferences. In here, ensure preview features are enabled:
 
Now click on Apps:

On the list of available apps, search for Microsoft Teams and enable the service, plus all the required features you want to use:
 

Accessing Microsoft Teams
For some reason, after enabling Microsoft Teams, its icon is still not available in the app launcher:

However, if we navigate to https://teams.microsoft.com we will be able to login to the service just fine.

Similar to Slack, and many other Office 365 applications, Microsoft Team is available in three versions: web app, desktop app, and mobile app.

The purpose of this blog post is not to explain how to use Microsoft Teams (the Microsoft Mechanics video is a great place to start on that), but to see if and how we can use the service to programmatically send alerts to administrators on their mobile devices. But before we do so, we need to use the web or desktop apps to do some initial configuration. So let’s get to it.


Setting Up Microsoft Teams
The first step to configuring Microsoft Teams is to login to https://teams.microsoft.com, select Teams and create our first team by clicking on Create team:
 
Next we give our new team a name and a description (optional). If we are currently the owner of an Office 365 Group, we get the option to add Teams functionality to that group:
 
The final step (optional) is to add one or more members to our new team:
 
If we add users, each will receive an email notifying them they have been added to our new Messaging Team:

 
We now have our first Team created :)

 
Each Team can have multiple channels. Channels are how Microsoft Teams organizes conversations. We can set up our channels however we like: by topic, discipline, project, and so on. Channels are open to everyone on the team and contain their own files, OneNote, etc...

So let’s create one channel just for alerts by clicking on ... next to our team’s name and then Add channel:
 
Give the channel a name and click Add:

We now have our Alerts channel dedicated to Exchange alerts:
 

Configuring WebHook Connector
Office 365 Connectors are used to get information and content into Microsoft Teams. Any user can connect their team to services like Trello, GitHub, Bing News, Twitter, etc., and get notified of the team's activity in that service. Connectors also provide a way for developers to integrate with Microsoft Teams by building custom incoming WebHook Connectors to generate rich cards within channels.
To generate our alerts to administrators, we will create these cards (messages) by sending an HTTP request with a simple JavaScript Object Notation (JSON) payload to a Microsoft Teams webhook address.

First, we need to create a webhook address for our Alerts channel:
1. From within Microsoft Teams, click ... next to the channel name and then select Connectors:
 
2. Scroll through the list of connectors to Incoming Webhook, and click Add:
  
3. Enter a name for the webhook, upload an image to associate with data from the webhook (optional), and select Create:
  
4. Copy the webhook URL to the clipboard and save it. We will need this URL for sending information to our channel:
 
 5. Click Done and a message will be visible in the Conversations tab informing all members that an incoming webhook has been configured for the channel:

 
We now have our webhook configured which we will use to post messages to our Alert channel. If we go into Connectors one more time, we are informed that an Incoming Webhook is already configured and by whom. If we click on Manage we get the options to change its name (which I have changed to AlertsBot) and to remove the webhook.
 
Please be aware that any member of the team can get the webhook URL and use it to send messages. On top of that, any member can also remove the webhook...

 
Sending Messages to Microsoft Teams
Now that we have our webhook configured, we need a method to send an HTTP request with a JSON payload to the webhook address. To achieve this, we have two options. The first option is to use cURL, a tool used in command lines or scripts to transfer data with URLs. Since my workstation is 64-bit, I downloaded the Win64 - Generic version (at the time of writing this blog, v7.51.0).
 
From the command line (not PowerShell), we can use the following command to send a basic “Hello World!” message to our channel:
curl.exe -H “Content-Type: application/json” -d “{\”text\”: \”Hello World!\”}”

If the POST succeeds, we will get a simple 1 returned by cURL:
 
If we go back to our channel’s Conversation window, we can see the new card posted to the team:
 
Our first programmatic alert/message to Microsoft Teams! :-D


Doing the same using PowerShell and cURL is a bit more tricky because of the “ (quotes) within the code. In the example above we used \” to escape the quotes, which will not work with PowerShell. The easiest method I found was to put the whole payload in a file (let’s call it alert.json, but we can also use alert.txt for example) and then pass the file into cURL. The file will look like this:
 
And the code used will be the following:
$webHook = “https://outlook.office365.com/webhook/bcbc68a4-606f-4ebf-8d78-4bbeac2c0c96@ed835685-e329-4799-9a9e-7ec941c92287/IncomingWebhook/(...)"

.\curl.exe -H “Content-Type: application/json” -d “@alert.json” $webHook


  
 
The second option to send messages to Microsoft Teams (and a much easier one!), is to simply use PowerShell’s native capabilities with the following two cmdlets:
Invoke-RestMethod: sends HTTP/HTTPS requests to Representational State Transfer (REST) web services;
ConvertTo-Json: converts any object to a string in JSON format. The properties are converted to field names, the field values are converted to property values, and the methods are removed.


Using these two cmdlets, we don’t need cURL anymore. Our previous “Hello World!” example becomes simply the following:

$webHook = “https://outlook.office365.com/webhook/bcbc68a4-606f-4ebf-8d78-4bbeac2c0c96@ed835685-e329-4799-9a9e-7ec941c92287/IncomingWebhook/(...)”

$alert = ConvertTo-JSON @{
text = “Hello World!”
}

Invoke-RestMethod -ContentType “application/json” -Method Post -body $alert -Uri $webHook

Simple as that! :)

If we manually run the code just to see what the variable $alert contains, we will see that it is in the exact same (JSON) format as our alert.json file:
  
 
In our next example, we start to get live data from Exchange and report on it. This simple example just sends a message containing certain details about a mailbox database named MDB01:
$webHook = “https://outlook.office365.com/webhook/bcbc68a4-606f-4ebf-8d78-4bbeac2c0c96@ed835685-e329-4799-9a9e-7ec941c92287/IncomingWebhook/(...)”

$exchDB = Get-MailboxDatabase “MDB01” -Status | Select Name, LastFullBackup, DatabaseSize, Mounted, ServerName
$userCount = (Get-Mailbox -Database $exchDB.Name -ResultSize Unlimited).Count
$size = $($exchDB.DatabaseSize.Split(“(“)[0])

$alert = ConvertTo-Json -Depth 4 @{
  text = “**$($exchDB.Name) Information:**”
  sections = @(
    @{
      facts = @(
        @{
        name = "Database:"
        value = $exchDB.Name
        },
        @{
        name = "Last Bck:"
        value = $exchDB.LastFullBackup
        },
        @{
        name = "Size (GB):"
        value = $size
        },
        @{
        name = "Mounted?"
        value = $($exchDB.Mounted)
        },
        @{
        name = "On Server:"
        value = $exchDB.ServerName
        },
        @{
        name = "User Count:"
        value = $userCount
        }
      )
    }
  )
}

Invoke-RestMethod -ContentType "application/json" -Method Post -body $alert -Uri $webHook

The result will be all the details for MDB01 nicely formatted:
 
 
Let’s now look at how we could monitor Exchange’s transport queues and issue an alert if the total number of queued emails goes beyond a certain limit. To achieve this, we get the queues across all servers, exclude any Shadow Redundancy emails, and count the total number of emails across the queues. Then, if that number is above our limit, we send an alert. Obviously, this is a basic script just for demonstration purposes. In a production environment, a few tweaks would likely be required, such as the threshold limit, any queues or servers to include/exclude, use a scheduled task instead perhaps, and so on.
$webHook = “https://outlook.office365.com/webhook/bcbc68a4-606f-4ebf-8d78-4bbeac2c0c96@ed835685-e329-4799-9a9e-7ec941c92287/IncomingWebhook/(...)”

While ($True) {
[Int] $msgCount = 0
Get-TransportService | Get-Queue | Where {$_.MessageCount -gt 0 -and $_.DeliveryType -notlike "Shadow*"} | ForEach {$msgCount += $_.MessageCount}

If ($msgCount -gt 50) {
$alert = ConvertTo-Json -Depth 1 @{
text = “**High Mail Queues!**`nTotal queued emails: $msgCount”
}

Invoke-RestMethod -ContentType "application/json" -Method Post -body $alert -Uri $webHook
}
Start-Sleep –Seconds 1800
}

The result will be the following alert:
  
 
Microsoft Teams Mobile Client
The purpose of this post was not to test Microsoft Teams itself, but to test if it can be used to reliably alert administrators on their mobile devices with any potential issues with Exchange.

We already established that we can easily generate alerts to a Teams’ channel, but what we now need to test is Microsoft Teams’ mobile app. To show how platform-independent Microsoft Teams is, I am going to use an iPhone to test the mobile app (trust me, I am not an Apple fan) :)

We start by searching and downloading the app from the Apple Store:
  
Once we open the app for the first time, we are asked to sign in:
  
After signing in we are presented with four tips, the last one being about notifications, which for the purpose of this article, we should obviously enable:

  
Once we sign in we can easily see the message we previously sent:
 
  
According to Microsoft, the best way to make sure people see our message is to @mention them. We do this by typing @ before a name and choosing the person we want to mention from the picker. They will generate a notification and a number will appear next to the channel we mentioned them in. When we mention someone, no one else will receive a notification, but everyone in the team will be able to see that we @mentioned someone though. Alternatively, we can also mention the entire channel, which will make everyone receive a notification.

At this stage it seems that I only get an alert on my mobile phone when someone mentions me:
  
Notice my name highlighted in red and the “@” at the top right hand corner of the message:
 
 
So, I had a look at the Notification configuration in the app and found the following:
 
 
I was hoping that by enabling everything I would start getting a notification on my phone for everything, but this was not the case. I still don’t get an alert unless someone mentions me, or the entire channel, sends me a direct message, replies to a message of mine or likes something I posted.

The problem is that I haven’t been able to find a way of mentioning someone or a channel using cURL or PowerShell... :( This means that users will get the messages on their mobile devices but not a notification, making this method not suitable for what I am trying to achieve... This, of course, until I find a way of mentioning someone using JSON!

I found an article on Bot Builder for .NET that has a section about Activities and Mention Entities. This article states that we can mention someone using a JSON like this:

{   
  ...
  "entities": [{ 
    "type":"mention",
    "mentioned": { 
      "id": "UV341235", "name":"Color Bot"
    },
    "text": "@ColorBot"
  }]
  ...
}


But I still haven’t been able to make it work using PowerShell (or any other method for that matter!)...

So using both the web and desktop Team apps, I sent a few messages where I mentioned someone and captured the JSON payload to see exactly how these are constructed and sent. For example, the following message:
  
Translates into the following JSON:
 
So it should just be a matter of building an identical (except maybe for the ClientMessageID property) JSON, right? Nope... Unfortunately, even sending an identical JSON using PowerShell results in the exact same message but without the mention...


Conclusion
Surely I am missing something here because of my lack of JSON knowledge and experience... As such, it might be that adding or changing something really simple will make it work! As is, I can’t yet use Teams to reliably alert me until I figure how to mention someone...

Nonetheless, Microsoft Teams is an awesome product and I am looking forward to explore it even further!

Wednesday, November 9, 2016

What is a Scoped Send Connector?

After so many years of being available, the Scoped property of Send Connectors is still confuses some Exchange administrators, or simply goes by unnoticed.

So what exactly is a Scoped Send Connector? Simply put, when we configure a send connector as scoped, we are stating that only Exchange 2007/2010 Hub Transport or Exchange 2013/2016 Mailbox servers in the same Active Directory (AD) site as the send connector can use it. If this option is not selected, then the connector can be used by any Exchange server in any AD site in the same Exchange organization.

But what is the point of this option if we need to specify the Source Servers for the send connector? Doesn’t this option stipulate which Exchange servers can use the connector?! Well, not exactly...

The source servers for a send connector determine the destination Exchange server for mail that needs to be routed through the send connector. The send connector scope controls the visibility of the connector within the Exchange organization.

Once again, send connectors are by default visible to all the Exchange servers in the entire AD forest, and are used in routing decisions. However, we can limit a send connector’s scope so that it is only visible to other Exchange servers in the same AD site. The send connector is invisible to Exchange servers in other AD sites, and is not used in their routing decisions. A send connector that is restricted in this way is said to be scoped.

To configure scoped send connectors in the EAC (assuming Exchange 2013/2016), we select Scoped send connector in the Address space section of the new send connector wizard, or on the Scoping tab in the properties of existing send connectors (as previously shown).
When using the Exchange Management Shell, we use the IsScopedConnector parameter on the New-SendConnector and Set-SendConnector cmdlets.

Tuesday, November 1, 2016

Cannot expand the folder. The set of folders cannot be opened.

The other day a user was having problems accessing a meeting room’s mailbox in Outlook. Whenever she tried to expand its folder list, she would get the following error:

Cannot expand the folder. The set of folders cannot be opened. Microsoft Exchange is not available. Either there are network problems or the Exchange server is down for maintenance.


First thing I tried was to remove and re-add the permissions, but to no vail:
Remove-MailboxPermission meeting.room@domain.com -User Linda -AccessRights FullAccess -InheritanceType All

Add-MailboxPermission meeting.room@domain.com -User Linda -AccessRights FullAccess -InheritanceType All


Second thing I tried was removing the permissions once again, but this time re-adding them without auto mapping:
Add-MailboxPermission meeting.room@domain.com -User Linda -AccessRights FullAccess -InheritanceType All –AutoMapping:$False

Then I manually added the mailbox via Account Settings. This seemed to have worked as the user could now expand the folders! However, when I closed and reopened Outlook the issue came back...


Third thing I tried was to use OWA to access the mailbox and everything worked fine! OK, seems to be a problem with Outlook itself. However, after creating a new profile, the issue was still there...
So, I decided to also give same access to a couple of other users and they all had the same problem (independently if they were in Online or Cached mode).

Time for some digging! Since this issue was all about Outlook, the first place I decided to look was in the RPC Client Access logs (or RCA logs), located by default at C:\Program Files\Microsoft\Exchange Server\V15\Logging\RPC Client Access


In these logs I found the following entry (multiple times) in the Failures column for the user:
RopHandler: Logon: [RopExecutionException] Couldn't find a mailbox for MailboxId: Dn[/o=EXCHANGE/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Meeting Room] in the current forest. The client should attempt to re-discover it.. Error code = UnknownUser (0x000003EB) -> [ObjectNotFoundException] The Active Directory user wasn't found.


Next, I used the Get-Mailbox and Get-ADUser cmdlets to search for the Distinguished Name of “/o=EXCHANGE/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Meeting Room” but couldn’t find it anywhere. There were a couple of X500 addresses for this mailbox, but none with this exact DN.

This meeting room was created years ago on Exchange 2010 and migrated to 2013. Like many other resource mailboxes in this particular environment, their naming convention is just awful and inconsistence, which made me think that I wouldn’t be surprised if somehow someone changed/deleted something they shouldn’t have...

Once I added a X500 address to this room with a value of “/o=EXCHANGE/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Meeting Room” the issue got fixed!

Thursday, October 13, 2016

Exchange alerting using Slack

In the Exchange monitoring concerns? Pick up the Slack article at TechGenix, I explore how to use Slack to send alerts to administrators’ mobile devices when something is wrong with Exchange.
 
 
A company’s messaging infrastructure is typically one of its most critical systems. Every administrator knows the importance of continuously monitoring Exchange. This is not only to prevent downtime and quickly fix problems, but also to be aware of the health of the infrastructure, to help identify potential problems, and spot performance degradations before they turn into actual problems and cause downtime.

Monitoring solutions such as Microsoft’s System Center Operations Manager (SCOM), SolarWinds, Nagios, and MailScape are some examples of monitoring tools for Exchange. However, not every organization has the means and capabilities to acquire and use them. On top of that, many monitoring solutions alert administrators exclusively by email, which can be problematic if the mail flow itself is affected, or through a dashboard that administrators might not have access to outside business hours.

Throughout my career, I have developed several scripts that run continuously every x minutes, and if they detect something is wrong with Exchange, they send an email alert. There are, however, two issues with this approach:
  1. If mail flow is affected, as already mentioned, then administrators will not receive the email alert.
  2. When out of the office, not everyone is constantly checking their emails, meaning it might be a while until an administrator is made aware something is wrong.
 
 
For these reasons, I researched the best methods to send notifications to my phone when something was wrong with Exchange. I tried Mobile Device Management (MDM) solution we were using at the time, but it seemed there was no way to interact with its API in order to send alerts programmatically. An alternative I also considered was WhatsApp. Although it is possible to use PowerShell to send WhatsApp notifications, it has a few drawbacks, such as the need for the recipients to register with a service like WhatsMate WA Gateway and the fact that I would need to maintain a list of users who would like to receive the notifications on my scripts. Users wouldn’t be able to opt-in or opt-out from receiving notifications themselves.

Slack to the rescue
Slack is a messaging app for teams that is even used by the NASA team behind the Mars Curiosity Rover. Slack is becoming extremely popular, and I am starting to see why. Just have a quick look at the video below and judge for yourself.

To continue reading, please check the Exchange monitoring concerns? Pick up the Slack article at TechGenix where I explore how to use Slack to send alerts to administrators’ mobile devices when something is wrong with Exchange.
 

Tuesday, September 20, 2016

RoutingMasterDN is pointing to the Deleted Objects container in Active Directory

The other a client noticed the following errors in the Application Event Log in their Exchange 2013 server:

Log Name: Application
Source: MSExchange ADAccess
Date: 3/8/2016 9:44:27 AM
Event ID: 2937
Task Category: Validation
Level: Warning
Keywords: Classic
User: N/A
Computer: xxxxxxxxxxx
Description:
Process edgetransport.exe (Transport) (PID=32144). Object [CN=Exchange Routing Group (DWBGZMFD01QNBJR),CN=Routing Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=xxxxxx,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=xxxxx,DC=xxxxx]. Property [RoutingMasterDN] is set to value [xxxxxxxx
DEL:67729122-b25b-4d4c-b5b0-75ad9e9dacd4], it is pointing to the Deleted Objects container in Active Directory. This property should be fixed as soon as possible.


To correct this problem, we need to use ADSI Edit and navigate to CN=Connections:

In here, right click CN=Exchange Routing Group (DWBGZMFD01QNBJR), go to its properties and find the msExchRoutingMasterDN:

In this particular case, this attribute was pointing to an old Exchange 2010 server that had already been decommissioned. Once I corrected this, and restarted the Transport services, the error went away.

Thursday, September 15, 2016

What are Arbitration Mailboxes used for?

Arbitration Mailboxes were first introduced in Exchange 2010, but many administrators still do not fully understand what each arbitration mailbox is used for. In Exchange 2010 there were three arbitration mailboxes and in Exchange 2013 two new ones were introduced (2016 has the same as 2013):
 
SystemMailbox{1f05a927-xxxx-xxxx-xxxx-xxxxxxxxxxxx} moderates messages, i.e., it is used for managing approval workflow. For example, an arbitration mailbox is used for handling moderated recipients and distribution group membership approval. The display name for this account is Microsoft Exchange Approval Assistant and is available since Exchange 2010.

SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c} is used in the Offline Address Book (OAB) generation process. This arbitration mailbox, with persisted capability of OrganizationCapabilityOABGen, is called an Organization Mailbox. Administrators can create additional Organization Mailboxes for fault tolerance or for serving users in a geographically disbursed Exchange deployment. As such, to list the arbitration mailboxes with persisted capability of OABGen, user the following cmdlet: Get-Mailbox -Arbitration | Where {$_.PersistedCapabilities -match “oab”}. This mailbox is new in Exchange 2013.

SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9} holds administrator audit log reports and stores in-place e-discovery search metadata. The display name for this account is Microsoft Exchange. This mailbox is available since Exchange 2010.

FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95fa1e042 is used for federation between different Exchange organizations and is available since Exchange 2010. Its display name is Microsoft Exchange Federation Mailbox.

Migration.8f3e7716-2011-43e4-96b1-aba62d229136, new in Exchange 2013, holds details of mailboxes being moved in migration batches.

Saturday, August 27, 2016

Outlook Online Mode Search Limited to 250 Results with Exchange 2013

When running Outlook in Online mode in an Exchange Server 2013 environment, the number of search results is never more than 250, even though there are more results available.
 
 
Cached Mode
When running Outlook in Cached Mode, this does not happen. When we perform a search that returns more than 250 results, we get the following message informing us of the high number of results:
 
And only 200 results are returned:
 
However, if we click on the message itself, which states "(…) click here to view all results" we do get all the results back, as expected.
 
An option to avoid this message and automatically return all the results, is to go to options and disable the Improve search speed by limiting the number of results shown:
 
 
Now we get all the results, no matter how many, whenever we perform a search.
Unfortunately, the same does not happen when in Online Mode...
 
 
Online Mode
If we follow the same steps when in Online Mode, we get the same 200 results back:
 
 As well as the message informing us of the high number of matches found:
 
 However, if we click on the warning message, we only get an additional 50 results:
 
 If we untick the same option under Search settings:
 
 We no longer receive the warning message, but only receive 250 results no matter what:
 
 
To fix this issue, we need to install Cumulative Update (CU) 11, or later, for Exchange 2013.

After installing the latest CU, the default value still remains 250, but we can now edit the Microsoft.Exchange.Store.Worker.exe.CONFIG file to increase this search result limit. To do so, we must add the following entry under the <appSettings> section as follows:
<appSettings>
  <add key="MaxHitsForFullTextIndexSearches" value="2000" />
</appSettings>

If <appSettings> does not exist, simply create it after the </runtime> tag:
 
The above entry will increase the number of search items returned to 2000. The MaxHitsForFullTextIndexSearches value can be between 1 and 1,000,000. However, please be careful when increasing it as it might cause performance impact on the server(s).

Once the above file has been edited, restart the information store service on the server for the change to take effect.