Upgrading to Exchange 2016 CU12 may fail when using Let’s Encrypt SSL Certificates
On a Microsoft Exchange 2016 Server, utilizing Let’s Encrypt SSL Certificates, an upgrade to Cumulative Update 12 may fail. This is due to security permissions on the SSL certificate.
I later noticed that this occurs on all cumulative updates when using the Let’s Encrypt SSL certificates. This includes Exchange 2016 CU13 and CU14.
The CU install will fail, some services may function, but the server will not accept e-mail, or allow connections from Microsoft Outlook, or ActiveSync devices. PowerShell and EAC will not function.
The issue can be identified on this failure log:
[02/18/2019 19:24:28.0862] [2] Beginning processing Install-AuthCertificate
[02/18/2019 19:24:28.0867] [2] Ending processing Install-AuthCertificate
[02/18/2019 19:24:28.0868] [1] The following 1 error(s) occurred during task execution:
[02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown.
[02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied.
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)
[02/18/2019 19:24:28.0883] [1] [ERROR] The following error was generated when "$error.Clear();
Install-ExchangeCertificate -services "IIS, POP, IMAP" -DomainController $RoleDomainController
if ($RoleIsDatacenter -ne $true -And $RoleIsPartnerHosted -ne $true)
{
Install-AuthCertificate -DomainController $RoleDomainController
}
" was run: "Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied.
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)".
[02/18/2019 19:24:28.0883] [1] [ERROR] Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown.
[02/18/2019 19:24:28.0883] [1] [ERROR] Access is denied.
[02/18/2019 19:24:28.0883] [1] [ERROR-REFERENCE] Id=CafeComponent___ece23aa8c6744163B617570021d78090 Component=EXCHANGE14:\Current\Release\Shared\Datacenter\Setup
[02/18/2019 19:24:28.0895] [1] Setup is stopping now because of one or more critical errors.
[02/18/2019 19:24:28.0895] [1] Finished executing component tasks.
[02/18/2019 19:24:28.0925] [1] Ending processing Install-CafeRole
[02/18/2019 19:35:09.0688] [0] CurrentResult setupbase.maincore:396: 0
[02/18/2019 19:35:09.0689] [0] End of Setup
The Fix
Unfortunately because Exchange is not working, you won’t be able to use Powershell or the EAC to configure SSL certs.
To resolve this, open up the IIS Manager, right click on the Exchange Web Site, click “Edit Bindings”
Once the “Edit Bindings” windows is open, you’ll want to open BOTH https bindings, and click “Edit”, and then change the SSL Certificate from the Let’s Encrypt SSL cert, to the self-signed Exchange certificate that ships on the brand new install. The self-signed certification most likely will be labelled as the computer name.
If you configured the Let’s Encrypt SSL certificate on the “Exchange Backend” IIS site, you’ll also need to repeat these steps on that as well.
You can now restart the server, run the “setup.exe” on CU12 again, and it will attempt to continue and repair Exchange 2016 Cumulative Update 12.
Final Note
After the update is complete, you’ll want to restart the server. You’ll notice that the acme script, whether run automatically or manually, will not set the Let’s Encrypt certificate up again (because it’s not due for renewal). You’ll need to run the letsencrypt.exe file, and force an auto renewal which will kick off the Exchange configuration scripts (or you can manually set the certificate if you’re comfortable applying Exchange SSL certificates via PowerShell.
Hi Stephen, i was about to upgrade our exchange 2016 environment when i came upon your article.
Do you have any suggestions for what i can do to fix the security permissions on our Lets encrypt certs. or advice on what i can do to not encounter this issue when doing the upgrade.
Thanks
Aarron
Hi Aarron,
To avoid the issue, use IIS Manager to change the Exchange certs to the self signed one’s that ship standard with Exchange. Restart the host, and then proceed with the upgrade.
After the upgrade, use the command prompt and run the acme certificate program with the force renew flag to reconfigure the Let’s Encrypt certificate.
Cheers
Stephen
Thanks for your advice.
Aarron
Is there a fix to prevent this from breaking CU updates when using LetsEncrypt SSL cert? Or do I need to get in the habit of going back to the default SSL cert before every CU update?
Hi Eric,
I actually don’t know. Every CU I’ve done since, I perform these steps now. This morning I actually upgraded to Exchange 2016 CU14 and performed these steps to avoid issues.
I’m hoping someone else can chime in and answer your question.
Stephen
Thank you. Ran into this problem with 2019 CU3 (oddly CU1 didn’t have the same problem when I installed it a few months ago) and your workaround got my upgrade done.
hey, thanks for the heads-up. So, to avoid this issue before applying a cumulative update, would this PS command in EAC suffice?
Enable-ExchangeCertificate -Thumbprint ###SELFSIGNEDCERT### -Services IIS
I haven’t tried it, but it might possibly!
FYI I was able to fix this issue with the ” –acl-fullcontrol” flag with ACME. I believe it was added on version 2.1.2.641.
For example, I used the following command:
wacs.exe –target manual –host mail.psh-lab.gq,webmail.psh-lab.gq,autodiscover.psh-lab.gq –certificatestore My –acl-fullcontrol “network service,administrators” –installation iis,script –installationsiteid 1 –script “./Scripts/ImportExchange.ps1” –scriptparameters “‘{CertThumbprint}’ ‘IIS,SMTP,IMAP’ 1 ‘{CacheFile}’ ‘{CachePassword}’ ‘{CertFriendlyName}'”
Hope this helps someone.
Thanks Edward!
Very helpful, had this Problem when upgrading to CU20 and changing certificate in IIS solved my problem.
[…] 07:26 – Let’s Encrypt issue when Upgrading Microsoft Exchange CU […]