Dec 132019
 
Microsoft Windows Logo

Yesterday, December 12th 2019, I powered on my Windows 10 1909 workstation to see that the start menu wasn’t working, along with the notification tray. I could launch programs from the taskbar, but search, start, and notifications were not functioning.

Attention: If you are experiencing issues with search, please continue reading to the bottom of the blog post and the update marked February 5th, 2020.

Since my workstation is running as a VDI instance, I checked vSphere and noticed the VM was running at extremely high CPU. Inside of the workstation, I opened up the event log and found numerous errors pertaining to the User Shell Experience, as well as multiple Windows 10 apps (UWP apps).

I tried to troubleshoot this using multiple methods found online on google. It sounds like this is a common issue for the past couple months, but no one has been able to find a fix.

Finally, after 14 hours of frusteration, I finally decided to restore the workstaton (VM) from a snapshot backup the night before. Powering it on the start menu was working. I installed some updates and everything is still working great.

If anyone has any information on this, please post it in the comments! I was surprised this isn’t easily fixable and actually required a restore from backup. I’m assuming numerous others are experiencing this issue.

Windows 10 Search on Start Menu not working

Update – February 5th 2020: Today, I noticed my search bar had stopped functioning. I also noticed lots of traffic to my blog for people searching for this. After some research I found this page: https://superuser.com/questions/1522905/windows-10-search-not-loading-showing-blank-window

I’ve condensed the fix. To resolve this issue, perform the following instructions:

  1. Open Notepad
  2. Copy and paste the below text:
    Windows Registry Editor Version 5.00
    
    [HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Search]
    "CortanaConsent"=dword:00000000
    "BingSearchEnabled"=dword:00000000
  3. Save file as “FixSearch.reg”
  4. Run the file (double click) or import the .reg file in to your registry
  5. Restart explorer.exe or restart your computer

The issue should now be resolved.

Oct 072019
 
Microsoft Windows Server Logo Image

Today I’m going to be talking about Read Only Domain Controllers (RODC). An RODC is a Read Only Domain Controller for Active Directory Services inside of Microsoft Windows Server. It has become one of my favorite discoveries in the last 10 years for use with clients in certain situations.

A Read Only Domain Controller is similar to a regular Domain Controller, with the exception that the content is synchronized and available as a read-only copy. You cannot write to an RODC AD database.

Let’s explore RODC’s in more depth and find out what they are, why they are used, and use-case scenario examples.

What is an RODC

Read Only Domain Controllers were originally released with Windows Server 2008, and have been available on all versions since (including Windows Server 2008R2, Windows Server 2012/2012R2, Windows Server 2016, and Windows Server 2019).

A Domain controller that is an RODC contains a read-only cached copy of the Active Directory database. Additional sets of controls are available to control and limit this information and what is stored and cached.

Why an RODC

A Read Only Domain Controller is typically placed in situations and scenarios where a standard writable domain controller cannot be placed. The AD data/information can be filtered so that important items such as passwords, credentials, and other security sensitive information are not cached on that server. This provides a safety mechanism if the RODC is stolen or compromised (either physically, or virtually). You can control it so that only required information is cached, such as credentials for the users in the specific office.

RODC’s are meant to be used at remote offices and/or branch offices (ROBO) to allow services to function that rely on Active Directory such as file/print services and other Active Directory applications. Also, typically at these sites it either wouldn’t make sense or be safe to have a writable domain controller, however the RODC is needed to cache AD information, and enhance performance of these AD applications.

Offloading Active Directory requests to a single cached copy onsite on an RODC significantly reduces bandwidth pipe requirements versus having numerous computers and users authenticating and requesting Active Directory content over a site-to-site VPN between the main office and remote office/branch office.

Also, if you have an office with an unstable internet connection where the site-to-site VPN regularly has issues or isn’t always available, having an RODC available to handle Active Directory requests can keep that office online and functioning.

Scenarios for an RODC

In the past I’ve used Read Only Domain Controllers for a few different types of scenarios. I’ll get in to them below and explain why.

The scenarios:

  • AD Cache for ROBO (Remote Office Branch Office)
    • Unstable internet connection
    • AD Services at remote site (File/Print, LoB)
    • Numerous Users accessing Active Directory
    • Improve login times
  • ROBO with Potential Security issues (theft, lack of survailence, etc.)
    • Office is in remote area with delayed physical security response, risk of theft
    • Server physical security at risk, employees could have access
  • Corporate Infrastructure hosted in the Cloud
    • Domain Controller in the Cloud
    • Need a DC on-premise to handle logins and resource access

AD Cache for ROBO (Remote Office Branch Office)

Implementing an RODC in this situation is an excellent example. In a situation where an office has unreliable (intermittent or slow) internet but is critical to business continuity, an RODC can keep them up and running uninterrupted.

Typically, if you were just using a Site-to-Site VPN, if that connection went down, users wouldn’t be able to authenticate against Active Directory or access resources in Active Directory. Having an RODC on-site, allows them to authenticate (if their credentials are stored) and access resources.

As most IT professionals are aware, having a large number of users authenticating and accessing these resources over a VPN can use up the bandwidth pipe and cause issues. Having an RODC locally virtually eliminates VPN bandwidth usage to only Active Directory synchronization, and synchronization deltas.

Finally, having users authenticate locally instead of a saturated high latency VPN connection, improves their login time and performance.

ROBO with Potential Security issues (theft, lack of survailence, etc.)

If you have a remote site with security concerns, an RODC can help you with your security strategy.

If an RODC is physically stolen, only credentials that are filtered to be cached on that RODC are stored locally, this usually excludes administrative accounts as well as other users and services that aren’t accessed or used at the remote site. Also, because the domain controller isn’t writable, the thief cannot power on, inject data in to Active Directory and have it sync to your other domain controllers if they somehow gained access to your internal network.

The above also holds true for possible malicious employees who may have skills or knowledge, or allow other 3rd parties to have physical or virtual access to the server.

In the event of a disaster, restoring or recreating an RODC is easy and fast. Since it synchronizes from writable DCs on the network, the concerns of traditional writable domain controller restores don’t need to be considered.

Corporate Infrastructure hosted in the Cloud

If by chance most of your corporate infrastructure is hosted in the cloud, you know that you still need some on-premise resources and infrastructure to handle and offload bandwidth requirements between your LAN network and virtual cloud LAN network.

Typically, in most cases you’d have an on-site on-premise domain controller to handle local login and authentication, as well as resource access. But why use a fully writable domain controller, when you can use an easy to manage and maintain RODC?

Using an RODC at your local site allows you to offload services off the pipe, to the RODC, again limiting bandwidth requirements to AD synchronizations and delta synchronizations. This allows your bandwidth to be used for more important things like Line of Business applications, e-mail, etc.

As most IT professionals prefer simple and functional, this keeps simplified and easy to manage.

Conclusion

RODC’s are a perfect tool to compliment your IT infrastructure and help secure it as well. I’ve placed numerous Read Only Domain Controllers at customers branch offices, remote oil and gas sites, and in various other scenarios.

Not only have they kept these customers up and running during outages, but the ease of use and ease of management make it common sense to use this technology.

Oct 062019
 

Today I wanted to do a brief post addressing Microsoft Exchange Backup and Disaster Recovery.

In the past week I’ve had over 30 people reach out to me via chat looking for help and advice in situations where:

  • A Cumulative Update Failed
  • Exchange Services will not start
  • Hardware Failure Occurred

In all of these cases the admins took a snapshot of their Exchange virtual machine (in Hyper-V or ESXi/VMware), and then restored it to the previous point when the failure occurred. This completely broke their Exchange install and possibly made it unrecoverable.

The above example is what you DO NOT want to do.

Microsoft Exchange Aware Backups

As per: https://docs.microsoft.com/en-us/exchange/high-availability/disaster-recovery/disaster-recovery?view=exchserver-2019

Exchange Server supports only Exchange-aware, VSS-based backups. Exchange Server includes a plug-in for Windows Server Backup that enables you to make and restore VSS-based backups of Exchange data. To back up and restore Exchange Server, you must use an Exchange-aware application that supports the VSS writer for Exchange Server, such as Windows Server Backup (with the VSS plug-in), Microsoft System Center 2012 – Data Protection Manager, or a third-party Exchange-aware VSS-based application.

You must use an Exchange-aware backup and/or disaster recovery application/software suite. These applications are aware of Exchange and designed to perform proper backups of Exchange, the mailboxes, and configuration. Not only do they backup the mailbox database and the VM running Exchange, but they also backup the system state and configuration of Microsoft Exchange.

Simply performing a VM snapshot is not supported and can break your Exchange installation.

Note that the configuration for Microsoft Exchange is stored inside of Active Directory, and not on the actual Exchange Server. Restoring the Exchange Server to a previous snapshot will cause a configuration synchronization gap between the Active Directory configuration and the mailbox database on the Exchange Server.

Options for Backup

There are plenty of options to perform Microsoft Exchange-aware backups.

If you’re looking for something free and easy, you could use the built-in Windows Server Backup function on Microsoft Windows Server. It’s perfect for special migration and upgrade jobs, homelabs, and small/micro sized businesses.

For larger organizations, I’ve used, setup, implemented, and managed the following backup applications:

There’s no excuse for not having a backup, especially if you call yourself a professional. You should always have a full working backup, especially before performing any type of maintenance, updates, or upgrades to your environment.

Microsoft Exchange Issues and Failures

Additionally, in the event of an issue, the solution isn’t always to restore from backup.

In most cases when something fails, it’s best practice to troubleshoot and correct the issue, instead of blasting away Exchange and restoring from backups.

Most Exchange installs can be saved simply by following standard troubleshooting procedures. Even if an Exchange Cumulative Update fails, you can fix what caused it to fail, and then re-run the installer/upgrader to attempt to recover! No backup restore needed!

Aug 092019
 
IIS Logo Image

You may find yourself unable to download attachments on an e-mail message you received on your Android or Apple iPhone from your Microsoft Exchange Server. In my case, this presented a “Unable to download.” with a retry option. Retrying would not work.

If the attachment is larger (over 10MB), this is most likely due to a limit enforced on the Activesync site in IIS on your Exchange Server. In this post I’m going to tell you why this happens, and how to fix it!

The Problem

Microsoft Exchange uses IIS (Internet Information Server) for numerous services including ActiveSync. ActiveSync provides the connectivity to your mobile device for your Exchange access.

IIS has numerous limits configured to stop massive bogus requests, reduce DDOS attacks, and other reasons.

The Fix

To resolve this and allow the attachment to download, we need to modify two configuration values inside of the web.config file on IIS.

Below are the values we will be modifying:

  • MaxDocumentDataSize – Maximum file (message) data size for transfer. “Sets the maximum data size that we will fetch (range or othewise)”
  • maxRequestLength – “Specifies the limit for the input stream buffering threshold, in KB. This limit can be used to prevent denial of service attacks that are caused, for example, by users posting large files to the server. The default is 4096 KB.” (as per here)

These settings are configured in the following file:

C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Sync\web.config

Before modifying the variables, please make a copy or backup of the web.config file so you can restore.

After you make a backup, open the file in notepad (right click -> run as administrator), and open the web.config file.

Simply search for the two values listed above, and change them. In my case, I tripled the “MaxDocumentDataSize”, and the “maxRequestLength” values. Examples from my “web.config” file are below:

add key="MaxDocumentDataSize" value="30720000"
httpRuntime maxRequestLength="30720" fcnMode="Disabled"

After changing these, run the following command from an elevated (as administrator) command prompt to restart IIS:

iisreset

And bam, you’re good to go!

May 222019
 
Microsoft Windows Logo

You may find yourself in a situation where an MMC snap-in errors out, and doesn’t allow you to reconfigure, fix, or use it. It becomes unusable.

In my case, this occurred on a system where I was trying to use the WSUS (Windows Server Update Services) snap-in, and it was configured for an old server that didn’t exist anymore.

When opening the WSUS MMC snap-in, it would report an error, give me the option to unload (which didn’t work), and do nothing else. There was no way to use or reconfigure it.

The Fix

To resolve this, you need to clear your local configuration for the snap-in. Your user profile contains all MMC snap-in information and configuration in the following directory:

C:\Users\USERNAME\AppData\Roaming\Microsoft\MMC

When browsing, here’s what it looks like on my system:

Picture of MMC user cache in appdata
User MMC config/cache folder

In my case, deleting the “wsus” file, reset the MMC snap-in, and allowed me to use it again and configure it for a new server.

Let me know if this helped you!