The Tech Journal

VMware ESXi asyncUnmapFile grows massive, cannot delete, errors

ESXi, iSCSI, VMware, vSphere No Responses »

Oct 202018

On an ESXi host when performing a manual unmap on your storage datastore, you may notice a very large (hidden) file on the datastore root called “.asyncUnmapFile”. This file could be taking up terabytes of space, and you aren’t able to delete this file.

asyncUnmapFile

Typically the asyncUnmapFile is used by the UNMAP feature on ESXi hosts to deal with unmapping and unallocating storage blocks on the SAN. When you run a manual UNMAP, this file should be created and should appear to using “0” (no) space (even if it is). When an UNMAP completes, this file should disappear and be automatically removed by the function. If an UNMAP is interupted, this file will not be deleted, allowing you to restart the process and upon a full successful completion, it should then be deleted.

The Problem

Some time ago, I had an issue when performing a manual UNMAP, where the ESXi host became unresponsive (due to memory issues). The command appeared to be completed, however I believe it caused potential issues or corruption on the iSCSI datastore. In subsequent runs, the UNMAP appeared to be functioning and working, however I didn’t realize that the asyncUnmapFile had grown to around 1.5TB.

This was noticed during a SAN storage audit, where we saw that the virtual pool on the SAN was using up way more storage than it should be on the datastore.

When we identified the file was this large and causing issues, we attempted to perform 2 UNMAPs (different reclaim sizes) to see if it would be automatically cleared afterwards. It had no effect and the file was unchanged.

We also tried to modify the permissions on the file, however when trying to delete it, it would report that the file or folder was not found, or that it does not exist. This was concerning as we were worried about potential datastore corruption.

It was also noticed that in the hostd.log and vmkernel.log we saw some errors where the host believed that the blocks on the datastore had already been freed: “on volume labeled ‘iSCSIDatastore01’ already freed by another host: This may be a non-issue”

The Solution

Unfortunately with all the research we did, we couldn’t find a clear-cut solution. With worries that the datastore may be corrupted, we needed to do something.

A decision was ultimately made to Storage vMotion all the VMs (Virtual Machines) to another datastore on a separate storage pool, delete the now empty LUN, and recreate it from scratch. After this, we used Storage vMotion again to move the VMs back.

Instantly I noticed that the VMs on that datastore were running faster (it’s only been 12 hours, so I’ll be adding an update in a few days to confirm). We no longer have the file on any of our datastores.

If anyone has further insight in to this issue, please leave a comment!

How to add an alternative UPN suffix to an Active Directory domain

Active Directory, Exchange Server, Microsoft 29 Responses »

Oct 162018

In this post, I’ll be going over how to add additional and/or alternative UPN suffixes to your Active Directory. I’ll also be going over why you may require this inside of your environment.

This is also a follow up post to the article here: https://www.stephenwagner.com/2016/09/23/outlook-2016-exchange-2013-password-prompts-upn-and-samaccountname-troubles/ as Microsoft has deleted the KB 243629 article which contained the original instructions.

Why

There is a number of reasons why you may want to do this:

You’re migrating to a newer version of Microsoft Outlook 2016, and require the users UPN to match the users e-mail address for auto-configure to function.
Your internal domain is is a “domain.local” domain, however you want users to log in with a “domain.com” domain.
You are implementing a line of business application or other piece of software that requires user’s UPNs to match their e-mail addresses.
You’re performing a migration.

How

Let’s get to it! Here’s how to add an alternative UPN suffix to an Active Directory domain:

Log on to your domain controller.
Open “Active Directory Domains and Trusts”
On the left hand side of the new window, right click on “Active Directory Domains and Trusts”, and select “Properties” (as shown below).
Active Directory Domains and Trusts Window
Type in your new domain suffix in to the “Alternative UPN suffixes” box, and then click “Add”. As shown below.
Add Alternative UPN suffix
Click “Apply” and then close out of the windows.

The new UPN suffix should be available via “Active Directory Users and Computers” and you should be able to set it to users.

You can also configure the user accounts via the Exchange Administration Center (EAC). See below for an example:

Exchange Administration Center UPN Suffix

Slow web browsing on business internet connection due to lack of reverse DNS entry (RDNS)

Internet No Responses »

Oct 122018

In the perfect and properly configured world, every internet user has a reverse DNS entry. This is is the DNS entry which tells people, servers, and services, what any given IP’s hostname is. Also, again in the perfect world, web servers shouldn’t check these, as the DNS query itself usually has to complete before it starts serving website data.

One of the key way’s webmasters and web server administrators increase their web server response times, is to make sure that their server is NOT performing reverse DNS queries when serving the site. However, we aren’t in a perfect world, and many web servers and web sites still perform these queries.

Many web servers do these queries because they are using mis-configured statistic generation software (website stats), default web server configuration files, or other reasons.

The problem

I recently had a discussion with a fellow IT professional where they were having issues with load times when opening websites. They were on a high speed business internet connection, so they figured it had to do with something else. They said they checked absolutely everything, so I decided to see what I could do to help out!

In my own research I noticed that on my own web server (which doesn’t perform reverse DNS queries on users), that numerous visitors both local to North America and abroad, did not actually have properly configured reverse DNS entries. One can deduce that when one of these users visits a website that actually performs an RDNS check during initial connection, it could cause a delay while the server itself waits for the DNS query to be performed (or even worse, timeout).

When further investigating, I also noticed a trend that the larger the company and the more expensive the internet connection, the more IPs that did not have reverse DNS records. I also noticed the IP addresses provided to my colleague did not have RDNS records.

I relayed this information back to my colleague and after they created the proper reverse DNS records, it seemed to help the issue!

Final Note

Since I don’t have direct access to their network, I couldn’t confirm this was the actual issue, or the only issue, but this just goes to show that you should always have your networks (both internal and external) properly configured using leading practices. In the long run, it saves time and avoids issues.

Enable Windows Update “Features on Demand” and “Turn Windows features on or off” in WSUS Environments

Windows 10, Windows 7, Windows Server, Windows Server 2016 55 Responses »

Oct 082018

If you are running Microsoft Windows in a domain environment with WSUS configured, you may notice that you’re not able to install some FODs (Features on Demand), or use the “Turn Windows features on or off”. This will stop you from installing things like the RSAT tools, .NET Framework, Language Speech packs, etc…

You may see “failure to download files”, “cannot download”, or errors like “0x800F0954” when running DISM to install packages.

To resolve this, you need to modify your domain’s group policy settings to allow your workstations to query Windows Update servers for additional content. The workstations will still use your WSUS server for approvals, downloads, and updates, however in the event content is not found, it will query Windows Update.

Enable download of “Optional features” directly from Windows Update

Open the group policy editor on your domain
Create a new GPO, or modify an existing one. Make sure it applies to the computers you’d like
Navigate to “Computer Configuration”, “Policies”, “Administrative Templates”, and then “System”.
Double click or open “Specify settings for optional component installation and component repair”
Make sure “Never attempt to download payload from Windows Update” is NOT checked
Make sure “Download repair content and optional features directly from Windows Update instead of Windows Server Update Services (WSUS)” IS checked.
Wait for your GPO to update, or run “gpupdate /force” on the workstations.

Please see an example of the configuration below:

Download repair content and optional features directly from Windows Update instead of Windows Server Update Services (WSUS)

You should now be able to download/install RSAT, .NET, Speech language packs, and more!

BSOD “Driver PNP Watchdog” error when upgrading VMware VM to Windows 10 Version 1809 October Update

Windows 10 13 Responses »

Oct 072018

Just a few words of warning when upgrading your VMware vSphere Windows 10 virtual machines to Windows 10 Version 1809 (October Update). When upgrading, after the first restart, you may notice multiple BSOD (Blue Screen of Death) with the error “Driver PNP Watchdog”. This will fail the upgrade. This issue may also occur on the Windows 10 Version 1903 (May Update).

Update – November 14 2018: This issue is still occurring on upgrades using the re-released November version of the October update.

Update and Fix – November 26th 2018: A very big thank you goes out to my reader Werner, who advised that the issue only occurs if the VM is in a snapshotted state. After his comment on this post, I decided to try upgraded without the VM in a snapshot state and it worked! Thanks Werner!

When the upgrade fails, the system will re-attempt until utlimately failing and reverting to the previous version of Windows 10.

In my case, I had a successful upgrade on numerous physical workstations, and a snapshot, so I decided to uninstall both the VMware tools agent, and VMware Horizon View agent. This had no affect and the VM still wouldn’t perform an upgrade.

I’m not sure if it’s the fact that it’s a VM, the VMware tools install, or the VMware Horizon View agent install, however I highly recommend waiting to upgrade until all the bugs get sorted out.

Leave a comment if you have anything to add! 🙂

Older Entries Newer Entries