There may be a point in time where you may wish to clear and rebuild the search index catalog on your Microsoft Exchange 2016 Server. This will cause the server to rebuild the search index from scratch.
In my case, for the past month or so Outlook 2019 (Office 365) clients connecting to an on-premise Microsoft Exchange 2016 Server, have been seeing the message “We’re having trouble fetching results from the server…”. The user can click on “Let’s look on your computer instead.” and the search will complete.
When troubleshooting this issue, I tried all of the following:
Clearing and rebuilding the Search index on the client computers
Deleted the OST files to re-create the local cached copy on the client computers
Restarting the Exchange Server
Restarting the Client Computers
Analyzing the Event Log for any errors (none)
None of the above helped in troubleshooting.
Outlook: “We’re having trouble fetching results from the server…”
Because of this, I decided to clear and rebuild the Search Index catalog for the mailbox database on the Exchange Server.
To check the status and to see if your index is corrupt, run the following command:
Get-MailboxDatabaseCopyStatus | ft ContentIndexState
“ContentIndexState” will report as “Corrupt” if it is corrupt, or “Healthy” if it is healthy.
[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus | ft ContentIndexState
ContentIndexState
-----------------
Healthy
My server reported as healthy, but I still chose to run the instructions below to rebuild the index.
Instructions
To do delete and re-create your Exchange Server Mailbox Database Search Index Catalog, please perform the following instructions.
Please Note: This is only for Exchange servers that are not part of a DAG. Do not perform these steps if your server is part of an Exchange cluster. Always make sure you have a complete backup of your server.
Log on to your Exchange server.
From the Start Menu, expand “Microsoft Exchange Server 2016”, and right-click on “Exchange Management Shell”, and select “Run as Administrator”.
Type the following commands to stop required search services.
You’ll see a folder inside of the mailbox folder with a GUID type name with Single at the end. Delete or move this (preferred is move to alternate location). I’ve put an example below.
12854239C-1823-8c32-ODJQ-SSDFK123CSDFG.1.Single
This is the folder you want to move/delete.
Go back to the “Exchange Management Shell”, and run the following commands to start the services.
As mentioned above, you can check the status of the rebuild by running the “Get-MailboxDatabaseCopyStatus” command, and looking at the “ContentIndexState” status.
That’s it! After running the command, you may notice your server will experience heavy CPU usage due to Exchange rebuilding the search index.
After rebuilding the search index, I noticed that my Outlook clients were able to successfully search on the server without having to select “Let’s look on your computer instead.”.
Upgrading to Exchange 2016 CU12 may fail when using Let’s Encrypt SSL Certificates
On a Microsoft Exchange 2016 Server, utilizing Let’s Encrypt SSL Certificates, an upgrade to Cumulative Update 12 may fail. This is due to security permissions on the SSL certificate.
I later noticed that this occurs on all cumulative updates when using the Let’s Encrypt SSL certificates. This includes Exchange 2016 CU13 and CU14.
The CU install will fail, some services may function, but the server will not accept e-mail, or allow connections from Microsoft Outlook, or ActiveSync devices. PowerShell and EAC will not function.
The issue can be identified on this failure log:
[02/18/2019 19:24:28.0862] [2] Beginning processing Install-AuthCertificate [02/18/2019 19:24:28.0867] [2] Ending processing Install-AuthCertificate [02/18/2019 19:24:28.0868] [1] The following 1 error(s) occurred during task execution: [02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. [02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied. at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule) at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule) at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService) --- End of inner exception stack trace --- at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl) at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target) at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services) at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord() at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1() at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed) [02/18/2019 19:24:28.0883] [1] [ERROR] The following error was generated when "$error.Clear(); Install-ExchangeCertificate -services "IIS, POP, IMAP" -DomainController $RoleDomainController if ($RoleIsDatacenter -ne $true -And $RoleIsPartnerHosted -ne $true) { Install-AuthCertificate -DomainController $RoleDomainController } " was run: "Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied. at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule) at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule) at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService) --- End of inner exception stack trace --- at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl) at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target) at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services) at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord() at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1() at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)". [02/18/2019 19:24:28.0883] [1] [ERROR] Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. [02/18/2019 19:24:28.0883] [1] [ERROR] Access is denied. [02/18/2019 19:24:28.0883] [1] [ERROR-REFERENCE] Id=CafeComponent___ece23aa8c6744163B617570021d78090 Component=EXCHANGE14:\Current\Release\Shared\Datacenter\Setup [02/18/2019 19:24:28.0895] [1] Setup is stopping now because of one or more critical errors. [02/18/2019 19:24:28.0895] [1] Finished executing component tasks. [02/18/2019 19:24:28.0925] [1] Ending processing Install-CafeRole [02/18/2019 19:35:09.0688] [0] CurrentResult setupbase.maincore:396: 0 [02/18/2019 19:35:09.0689] [0] End of Setup
The Fix
Unfortunately because Exchange is not working, you won’t be able to use Powershell or the EAC to configure SSL certs.
To resolve this, open up the IIS Manager, right click on the Exchange Web Site, click “Edit Bindings”
IIS Exchange Edit Bindings
Once the “Edit Bindings” windows is open, you’ll want to open BOTH https bindings, and click “Edit”, and then change the SSL Certificate from the Let’s Encrypt SSL cert, to the self-signed Exchange certificate that ships on the brand new install. The self-signed certification most likely will be labelled as the computer name.
Exchange SSL Bindings
If you configured the Let’s Encrypt SSL certificate on the “Exchange Backend” IIS site, you’ll also need to repeat these steps on that as well.
You can now restart the server, run the “setup.exe” on CU12 again, and it will attempt to continue and repair Exchange 2016 Cumulative Update 12.
Final Note
After the update is complete, you’ll want to restart the server. You’ll notice that the acme script, whether run automatically or manually, will not set the Let’s Encrypt certificate up again (because it’s not due for renewal). You’ll need to run the letsencrypt.exe file, and force an auto renewal which will kick off the Exchange configuration scripts (or you can manually set the certificate if you’re comfortable applying Exchange SSL certificates via PowerShell.
I have noticed an issue when after upgrading Microsoft Exchange 2016 CU10 to Exchange 2016 CU11, services may fail to start. This issue can be intermittent, where some restarts are able to start more services, and others restarts fewer. I have observed this on 2 separate Exchange upgrades, both were CU10 to CU11.
The Problem
Recently, a customer had an issue where a Microsoft Exchange security update bricked their entire Exchange CU10 installation. Files were missing and services would not start (even after manually re-configuring all system services to their prior settings, and force starting). To fix this, we weighed our options and decided the best course of action would be to attempt the latest CU (CU11). This is because each Microsoft Exchange Cumulative update is actually a full installer that completely removes the old version, and installs the new version cleanly.
After installing CU11 we were able to rescue the Exchange installation (services could now start, and functioned), however numerous errors and warnings were now present, and we also noticed that there were some new issues with services.
One service in particular called “Net.Tcp Port Sharing Service”, would occasionally not start in time and cause all the Exchange Services not to start (Exchange is dependent on this services). Other times, this service would start, however random Exchange services would timeout.
Some of the errors and warnings included:
Event ID 7000
Source: Service Control Manager
Description:
The MSComplianceAudit service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.
Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSComplianceAudit service to connect.
Event ID 7000
Source: Service Control Manager
Description:
The MSExchangeRepl service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.
Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSExchangeRepl service to connect.
I also observed that on a few restarts, the services that failed would eventually end up restarting 10-15 minutes later (this only occurred 50% of the time).
Originally I was concerned and believed these issues were related to the original issues the customer experienced, however I upgraded my own Exchange 2016 server to CU11 and experienced the same problems (my instance was a clean fully functioning install). I also attempted to upgrade .NET to version 4.7.2 to see if this had any effect, but it did not.
When you go in to services (services.msc) and manually start the services, Exchange functions perfectly and everything works.
The Solution
As of yet, I don’t have a proper solution. I did however notice that with my customer’s environment, after it was left to sit overnight (around 8 hours), that subsequent restarts actually were able to start the majority of the services properly. It almost seemed as if it just needed time to fix itself. I’m not sure if this is because of IO load, or some type of Exchange database maintenance, but I’m waiting to see if it clears up on my instance as well after an amount of time. I’ll be keeping this post updated.
UPDATE – October 29th: I’ve confirmed for the 2nd time that the issue resolves at least 6-8 hours after the upgrade. At the end of the day I restarted my machine and everything was functioning properly.
If you are experiencing this issue, or can make a comment on it, please leave a comment on this post!
In this post, I’ll be going over how to add additional and/or alternative UPN suffixes to your Active Directory. I’ll also be going over why you may require this inside of your environment.
There is a number of reasons why you may want to do this:
You’re migrating to a newer version of Microsoft Outlook 2016, and require the users UPN to match the users e-mail address for auto-configure to function.
Your internal domain is is a “domain.local” domain, however you want users to log in with a “domain.com” domain.
You are implementing a line of business application or other piece of software that requires user’s UPNs to match their e-mail addresses.
You’re performing a migration.
How
Let’s get to it! Here’s how to add an alternative UPN suffix to an Active Directory domain:
Log on to your domain controller.
Open “Active Directory Domains and Trusts”
On the left hand side of the new window, right click on “Active Directory Domains and Trusts”, and select “Properties” (as shown below).
Active Directory Domains and Trusts Window
Type in your new domain suffix in to the “Alternative UPN suffixes” box, and then click “Add”. As shown below.
Add Alternative UPN suffix
Click “Apply” and then close out of the windows.
The new UPN suffix should be available via “Active Directory Users and Computers” and you should be able to set it to users.
You can also configure the user accounts via the Exchange Administration Center (EAC). See below for an example:
In the latest updates and versions of Microsoft Office 2016, I found a bug where when a user adds a new on-premise Microsoft Exchange 2016 account, it will repeatedly prompt for a username and password and ultimately fail if you hit cancel (no matter how many times you enter credentials). This was on the internal LAN on a domain joined workstation.
I did the usual checks:
Check Virtualdirectory configuration on Exchange
Check Virtualdirectory configuration on IIS (Internet Information Services)
Check Autodiscover DNS entries, InternalURL and ExternalURL configuration
Check for SCP inside of domain
All the of the above came back fine and were configured properly.
I have numerous other Outlook 2016 clients configured and working (installed as older versions, but have been updated), so I used those to troubleshoot (same scenario, domain joined on internal LAN and external WAN). After spending 10 hours ripping apart everything, confirming configuration, I noticed that when using the “Test Email Autoconfiguration…” (holding CTRL while right clicking on Outlook tray icon), that the e-mail clients had a skewed order for checking autodiscovery.
The e-mail clients were actually trying to authenticate with Office365 before my own on-premise Exchange Server (domain SCP or autodiscover records). This is absolutely bizarre! After spending 2 hours googling (I couldn’t find anything), I finally stumbled across this document and found an interesting piece of information:
“Outlook uses a set of heuristics to determine whether the user account provided comes from Office 365. If Outlook determines confidently that you are an O365 user, a try is made to retrieve the Autodiscover payload from the known O365 endpoints (typically https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml or https://autodiscover-s.partner.outlook.cn/autodiscover/autodiscover.xml). If this step does not retrieve a payload, Outlook moves to step 5.”
WTF?!?!?
So while this doesn’t explain why this happened, it explains what’s happening. I believe this is what’s happening as my working clients are trying to Autodisocver with Office365 first…
I went ahead an created a registry value to control the policy for “ExcludeExplicitO365Endpoint“. After configuring the registry key, I noticed that Autodiscover was now functioning properly and checking SCP and autodiscover DNS records first. I have no idea why the “heuristics” determined I was an Office365 user, but I’m not (I do have access to Office365 as a partner, but don’t use it and don’t have it configured). This may effect other partners, or users that utilize some O365 services…
The Fix
To fix this issue, create a text file and copy/paste this text below.
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\AutoDiscover]
"ExcludeExplicitO365Endpoint"=dword:00000001
Then save it, and rename it as ExcludeExplicitO365Endpoint.reg and run it (this will import the applicable registry key). ONLY DO THIS if you are using an Exchange On-Premise account, and not a Office365 or hosted exchange account.
Keep in mind that autodiscover also queries the domain root (domain.com), before querying the autodiscover host (autodiscover.domain.com). If you want to stop both the Office365 autodiscover and the root domain autodiscover challenge, use the following below:
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\AutoDiscover]
"ExcludeExplicitO365Endpoint"=dword:00000001
"ExcludeHttpsRootDomain"=dword:00000001
You’ll notice that we also set “ExcludeHttpsRootDomain” to “1” which stops it from checking the root domain.
After this, the issue was completely fixed. If you know what you’re doing, you can also use Outlook GPO settings and deploy this to a vast number of systems using Group Policy.
Additional Note (added November 2nd, 2018)
While reading numerous documents covering autodiscovery, I also came across an article that went in to detail with particulars as to how Mapi over HTTP functions. Even with the above, when accessing Outlook externall from the domain, you may still notice a single password prompt for the first time you log in externally.
After reading through documentation, I found that this is most likely because the first user account login (the very first time the user logged in on the computer), the username format of “DOMAIN\Username” was used, and not the UPN. The documentation mentioned that this may fail the negotiation, which will require a single password prompt on autodiscovery. This issue can be avoided by using the users UPN ([email protected]) to log in for the first time on the system.
Please note that the UPN must match the user’s e-mail address.
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish.
Do you accept the use of cookies and accept our privacy policy? AcceptRejectCookie and Privacy Policy
Privacy & Cookies Policy
Privacy Overview
This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
