April 2018

Microsoft Windows Server booting to Safe Mode and Directory Services Restore mode with Veeam Backup

ESXi, Microsoft, Veeam Backup and Replication, VMware, Windows Server 41 Responses »

Apr 292018

Running Veeam Backup and Replication, a Microsoft Windows Server Domain Controller may boot in to safe mode and directory services restore mode.

About a week ago, I loaded up Veeam Backup and Replication in to my test environment. It’s a fantastic product, and it’s working great, however today I had a little bit of an issue with a DC running Windows Server 2016 Server Core.

I woke up to a notification that the backup failed due to a VSS snapshot issue. Now I know that VSS can be a little picky at times, so I decided to restart the guest VM. Upon restarting, she came back up, was pingable, and appeared to be running fine, however the backup kept failing with new errors, the event log was looking very strange on the server, and numerous services that were set to automatic were not starting up.

This specific server was installed using Server Core mode, so it has no GUI and is administered via command prompt over RDP, or via remote management utilities. Once RDP’ing in to the server, I noticed the “Safe Mode” branding on each corner of the display, this was very odd. I restarted the server again, this time manually trying to start Active Directory Services manually via services.msc.

This presented:

Event ID: 16652
Source: Directory-Services-SAM
General Description: The domain controller is booting to directory services restore mode.

Screenshot:

The domain controller is booting to directory services restore mode.

This surprised me (and scared me for that matter). I immediately started searching the internet to find out what would have caused this…

To my relief, I read numerous sites that advise that when an active backup is running on a guest VM which is a domain controller, Veeam activates directory services restore mode temporarily, so in the event of a restore, it will boot to this mode automatically. In my case, the switch was not changed back during the backup failure.

Running the following command in a command prompt, verifies that the safeboot switch is set to dsrepair enabled:

bcdedit /v

To disable directory services restore mode, type the following in a command prompt:

bcdedit /deletevalue safeboot

Restart the server and the issue should be resolved!

Worldwide DNS issues. Failure to resolve or lookup queries.

DNS 1 Response »

Apr 242018

At 5:00AM MST (April 24th, 2018) this morning, I noticed DNS (Domain Name Service) name resolution is failing on numerous internet domains. After further troubleshooting, I realized it’s the root servers that provide DNS that are failing to resolve these hosts.

Some users on providers that cache records may not immediately notice the issues as their ISP has cached records. Doing a search on Twitter confirms numerous people are reporting issues with DNS across numerous providers and ISPs, Google DNS Servers, and Amazon DNS Servers.

Update 6:22AM MST:

On my DNS server, I’m noticing the problematic domains that aren’t providing DNS records, are loading NS records that use awsdns-XX dns servers. This could show a problem with Amazon AWS DNS servers. I will continue to try to identify where the issues are.

Update 6:32AM MST:

I’ve noticed that Amazon AWS has since added a “Recent Event” on their service status page for “Amazon Route 53”: “5:19 AM PDT We are investigating reports of problems resolving some DNS records hosted on Route53 using the third party DNS resolvers 8.8.8.8 and 8.8.4.4 . DNS resolution using other third-party DNS resolvers or DNS resolution from within EC2 instances using the default EC2 resolvers are not affected at this time.”

Update 6:45AM MST:

It appears that there are issues with Amazon’s Route 53 DNS service which provides cloud based DNS services. Trying to view https://aws.amazon.com/route53/ results in page load errors.

Update 7:06AM MST:

Another update from Amazon on their service status page for “Amazon Route 53”: 5:49 AM PDT We have identified the cause for an elevation in DNS resolution errors using third party DNS resolvers 8.8.8.8 / 8.8.4.4 and are working towards resolution. DNS resolution using other third-party DNS resolvers or DNS resolution from within EC2 instances using the default EC2 resolvers continues to work normally.

Update 3:50PM MST:

It appears this was actually a malicious attack including DOS, a man in the middle attack, and an attempt to compromise users accounts. Information can be found at https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f (thanks Juk for this post), and https://www.bleepingcomputer.com/news/security/hacker-hijacks-dns-server-of-myetherwallet-to-steal-160-000/ .

Upgrading to VMware vSphere 6.7? Remember that vSphere vDP is no longer supported!

VMware, vSphere No Responses »

Apr 232018

Ready to jump the gun and upgrade to vSphere 6.7? Hold on a moment…

You’ll remember some time ago VMware announced they are dropping support for vSphere vDP (vSphere Data Protection). If you’re running this in your environment, it will break when upgrading to vSphere 6.7.

A better idea would be to migrate over to a product like Veeam, however please note that as of this date, Veeam does not officially support vSphere 6.7. Support should be coming in the next major update.

VMware vSphere 6.7 released! Here’s the download link…

ESXi, HPE, VMware, vSphere 21 Responses »

Apr 172018

With the news of VMware vSphere 6.7 being released today, a lot of you are looking for the download links for the 6.7 download (including vSphere 6.7, ESXi 6.7, etc…). I couldn’t find it myself, but after doing some scouring through alternative URLs, I came across the link.

VMware vSphere 6.7 Download Link

Here’s the link: https://my.vmware.com/web/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/6_7

HPE Specific (HPE Customization for ESXi) Version 6.7 is available at: https://www.hpe.com/us/en/servers/hpe-esxi.html

Unfortunately the page is blank at the moment, however you can bet the download and product listing will be added shortly!

UPDATE 10:15AM MST: The Download link is now live!

More information on the release of vSphere 6.7 can be found here, here, here, here, here, and here.

An article on the upgrade can be found at: https://blogs.vmware.com/vsphere/2018/05/upgrading-vcenter-server-appliance-6-5-6-7.html

Happy Virtualizing!