Veeam WAN Accelerator Service Fails to Start

Nov 232024

In some scenarios, you may encounter an issue where the Veeam WAN Accelerator service fails to start.

This will cause backup and backup copy jobs to fail that use the Veeam WAN Accelerator, which is how this issue is usually first diagnosed.

In this post I’ll explain the problem, what can cause it, and how to resolve the issue.

The Problem

When this issue occurs, and when a Backup or Backup Copy job runs, it will usually first fail with the following error from the Veeam console:

Error: The RPC server is unavailable. RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [IP.OF.WAN.ACC:6464].

Failed to process (VM Name).

See below for a screenshot of the example:

Veeam Backup Copy Job Failing due to Veeam WAN Accelerator Service failing

From the error above, the next step is usually to check the logs to find out what’s happening. Because this Backup Copy job uses the WAN accelerator, we’ll look at the log for the Veeam WAN Accelerator Service.

Svc.VeeamWANSvc.log

[23.11.2024 11:46:24.251] <  3440> srv      | RootFolder = V:\VeeamWAN
[23.11.2024 11:46:24.251] <  3440> srv      | SendFilesPath = V:\VeeamWAN\Send
[23.11.2024 11:46:24.251] <  3440> srv      | RecvFilesPath = V:\VeeamWAN\Recv
[23.11.2024 11:46:24.251] <  3440> srv      | EnablePerformanceMode = true
[23.11.2024 11:46:24.255] <  3440> srv      | ERR |Fatal error
[23.11.2024 11:46:24.255] <  3440> srv      | >>  |boost::filesystem::create_directories: The system cannot find the path specified: "V:\"
[23.11.2024 11:46:24.255] <  3440> srv      | >>  |Unable to apply settings. See log for details.
[23.11.2024 11:46:24.255] <  3440> srv      | >>  |An exception was thrown from thread [3440].
[23.11.2024 11:46:24.255] <  3440> srv      | Stopping service...
[23.11.2024 11:46:24.256] <  3440> srv      | Stopping retention thread...
[23.11.2024 11:46:24.257] <  4576>          | Thread started.  Role: 'Retention thread', thread id: 4576, parent id: 3440.
[23.11.2024 11:46:24.258] <  4576>          | Thread finished. Role: 'Retention thread'.
[23.11.2024 11:46:24.258] <  3440> srv      | Waiting for a client('XXX-Veeam-WAN:6165')
[23.11.2024 11:46:24.258] <  3440> srv      | Stopping server listening thread.
[23.11.2024 11:46:24.258] <  3440> srv      |   Stopping command handler threads.
[23.11.2024 11:46:24.258] <  3440> srv      |   Command handler threads have stopped.
[23.11.2024 11:46:24.258] <  4580>          | Thread started.  Role: 'Server thread', thread id: 4580, parent id: 3440.

In the Veeam WAN Service log file above, you’ll note a fatal error where the service is unable to find the paths configured, which caused the service to halt and stop.

In some configurations, iSCSI is used to access Veeam backup repository storage hosted on iSCSI targets. Furthermore, in some iSCSI configurations special vendor plugins are used to access the iSCSI storage, and configure items like MPIO (multipath input output), which can take additional time to initialize.

In this scenario, the Veeam WAN Accelerator Service was starting before the Windows iSCSI service, MPIO Service, and Nimble Windows Connection Manager plugin had time to initialize, resulting in the WAN accelerator failing because it couldn’t find the directories it was expecting.

The Solution

To resolve this issue, we want the Veeam WAN Accelerator Service to have a delayed start on the Windows Server operating system bootup sequence.

Open Windows Services
Select “Veeam WAN Accelerator Service”
Change “Startup Type” to “Automatic (Delayed Start)”
Click “Apply” to save, and then click “Start” to start the service.

As per the screenshot below:

Veeam WAN Accelerator Service Properties

The Veeam WAN Accelerator Service will now have a delayed start on system bootup, allowing the iSCSI initiator to establish and mount all iSCSI Target block devices, before starting the WAN service.

VMware Explore 2024

Sep 012024

I’m finally back at home after an amazing week at VMware Explore 2024, in Las Vegas!

As always, VMware Explore 2024 was the place to be, and THE most important conference of the year, at least for me, my clients, and those who need to be in the know.

See below for my insights on the conference, information and links, as well as some videos I created to talk about!

The Focus moving forward

This was the first VMware Explore with VMware by Broadcom. With the acquisition of VMware by Broadcom, there’s been a number of changes they have implemented, including their new strategy moving forward, SKU consolidation, as well as solution refinement.

The focus this year was on VCF, also known as VMware Cloud Foundation, with the announcement of VCF 9.0, and enabling technologies like AI (Artificial Intelligence) with VMware Private AI Foundation, and NVIDIA AI enterprise.

Why VMware Explore is important for me

For me, VMware Explore is important because of a number of different things. A brief of this is:

VMware By Broadcom
- Strategy
- Product Releases
- Technical Sessions (Learning, Training)
- Hands on Labs (HoL)
Community
- VMUG – VMware User Group
- VMware vExpert
Networking
- Reconnecting in person with clients in other cities (and other countries)
- Meeting customers face to face for the first time
- New Business

I can’t recommend enough how important VMware Explore is for career development. It’s a face paced conference, and you won’t have time to do everything you plan and want to, but you just need to go with the flow and let it happen naturally.

Technical Sessions

There was a number of AMAZING technical sessions, which are all available via the VMware Explore content catalog (click the link to view).

You can also view various community sessions, by going to VMware TV Live (click for link).

Check out my videos on VMware Explore 2024

I hope you enjoy the videos I created covering VMware Explore during the experience, and after.

VMware Explore 2024: A chat with Matt Heldstab about VMUG and Community

In this video, I sat down with Matt Heldstab, Global VP for VMUG (VMware User Group), to talk about the importance of community, VMUG, and how VMware Explore brings everyone together in person. This provides opportunities to reconnect with old friends, make new friends, and share a passion of technology (and VMware By Broadcom solutions).

VMware Explore 2024: A chat with Matt Heldstab about VMUG and Community

VMware Explore 2024: Day 1, Let’s Explore

On Day 1, I chatted with Daniel Keer (owner and writer at theDXT), to talk about his experience at VMware Explore. This was Daniel’s first VMware conference, and large tech conference for that matter.

Daniel was having a blast, commenting on how he was able to connect with VCF experts even before the conference started, and how friendly and welcoming everyone was.

VMware Explore 2024: Day 1, Let’s Explore

Daniel and I covered:

The Conference
Networking Oppurtunities
Broadcom Aquisition of VMware
NVIDIA AI Enterprise (NIMs and the NGC Catalog)
Private AI
VMware EUC split to Omnissa
World of EUC
- EUC World: Independence Conference
VMware Communities
- VMUG
- VMware vExpert Program
- Community Theatre

I was able to catch up with Daniel later, and he told me he was enjoying back to back technical sessions, and VMware Hands on Labs.

VMware Explore 2024: Day 3, Talking VCF and AI with Johan van Amersfoort

Johan van Amersfoort (owner/writer at vHojan.nl – Johan van Amersfoort) is a very well known presenter, often presenting on technologies such as EUC (with Omnissa Horizon), and now on AI and the endeavors both in his own home lab, as well as the work his organization is doing for customers with AI.

Johan and I talked about:

VMware Explore recap
Discussion on VCF, Private AI Foundation, and NVIDIA AI Enterprise (NIVIDIA NIMs)
Johan’s Presentations (see below for links)
Enterprise AI Workloads
VMware EUC divestiture (separation of Omnissa)
GPU Discussion
Tanzu, Cloud Foundry
Simplification of IT and Virtualization with VMware VCF
Community

At VMware Explore, Johan had two sessions (click the links to view):

VMware Explore 2024: Debrief and takeaways

Getting back home, there was a lot to take in and process, however I wanted to do a quick debrief and takeaway video.

In this video, I talk about:

Broadcom’s vision for VCF
Evolving world of IT
Corporate IT strategy shift
- Private AI
- Private Cloud
Practical AI use cases

Towards the end of the video, I also shared my thoughts on the last session I attended, called “GenAI-Powered Insights in Avi LB: A Journey into Next-Gen Observability“. You can click on the session name to find out more information and watch the video.

The session covered a real-world practical use of AI in IT, using an LLM to provide insight, analysis, administration, monitoring, and management of the VMware Avi Loadbalancer, using LLMs, with the choice of using Google Gemini in the cloud, or your own LLM on-prem using NVIDIA AI Enterprise, NIMs, and VMware Private AI Foundation.

Cool stuff!

VMware Explore, moving forward…

Moving forward, I’m aligned with and believe in the strategy of Private Cloud first, enabling technologies such as AI, while providing existing solutions such as compute.

I can’t wait for VMware Explore 2025!

vGPU Mixed Size Virtual GPU types with VMware vSphere 8U3

NVIDIA, NVIDIA vGPU, Omnissa Horizon, vSphere 12 Responses »

Jun 262024

I’m happy to announce today that you can now deploy vGPU Mixed Size Virtual GPU types with VMware vSphere 8U3, also known as “Heterogeneous Time-Slice Sizes” or “Heterogeneous vGPU types”.

VMware vSphere 8U3 was released yesterday (June 26th, 2024), and brought with it numerous new features and functionality. However, mixed vGPU types deserves it’s own blog post as it’s a major game-changer for those who use NVIDIA vGPU for AI and VDI workloads, including Omnissa Horizon.

NVIDIA vGPU (Virtual GPU) Types

When deploying NVIDIA vGPU, you configure Virtual GPU types that provide Workstation RTX (vWS Q-Series), Virtual PC (vPC B-Series), or Virtual Apps (vApps A-Series) class capabilities to virtual machines.

On top of the classifications above, you also needed to configure the Framebuffer memory size (or VRAM/Video RAM) allotted to the vGPU.

Historically, when you powered the first VM, the physical GPU that provides vGPU, would then only be able to serve that Virtual GPU type (class and Framebuffer size) to other VMs, locking all the VMs on running on that GPU to same vGPU type. If you had multiple GPUs in a server, you could run different vGPU types on the different physical GPU, however each GPU would be locked to the vGPU type of the first VM started with it.

NVIDIA Mixed Size Virtual GPU Type functionality

Earlier this year, NVIDIA provided the ability to deploy heterogeneous mixed vGPU types through the vGPU drivers, first starting with the ability to run different classifications (you could mix vWS and vPC), and the later adding support for mixed-size frame buffers (example, mixing a 4Q and 8Q profile on the same GPU).

While the NVIDIA vGPU solution supported this, VMware vSphere did not immediately add support so it couldn’t take advantage of this until the new release of VMware vSphere 8U3, VMware vCenter 8U3, and VMware ESXi 8U3.

To configure different classifications (vWS mixed with vPC), it requires no configuration other than using a host-driver and guest-driver that support it, however to use different sized framebuffers, it needs to be enabled on the host.

To Enable vGPU Mixed Size Virtual GPU types:

Log on to VMware vCenter
Confirm all vGPU enabled Virtual Machines are powered off
Select the host in your inventory
Select the “Configure” tab on the selected host
Navigate to “Graphics” under “Hardware”
Select the GPU from the list, click “Edit”, and change the “vGPU Mode” to “Mixed Size”

Screenshot showing the "Graphics Properties" for GPU adapters on VMware ESXi 8U3 with the "vGPU Mode" set to "Mixed Size"

Once you configure this, you can now deploy mixed-size vGPU profiles.

When you SSH in to your host, you can query to confirm it’s configured:

[root@ESXi-HOST:~] nvidia-smi -q

    vGPU Device Capability
        Fractional Multi-vGPU             : Supported
        Heterogeneous Time-Slice Profiles : Supported
        Heterogeneous Time-Slice Sizes    : Supported
        vGPU Heterogeneous Mode           : Enabled

It’s supported, and enabled!

Additional Notes

Please note the following:

When restarting your hosts, resetting the GPU, and/or restarting the vGPU Manager daemon, the ESXi host will change back to it’s default “Same Size” mode. You will need to manually change it back to “Mixed Mode”.
When enabling mixed-size vGPU types, the number of some types of vGPU profiles may be reduced vs running the GPU in equal-size mode (to allow other profile types). Please see the additional links for information on Mixed-Size vGPU types inside the “Virtual GPU Types for Supported GPUs” link.
Only “Best Effort and “Equal Share” schedulers are supported with mixed mode vGPU. Fixed Share scheduling is not supported.

Additional Links

Join me at VMware Explore 2024

VMware, VMware Explore, VMware Explore 2024 No Responses »

Jun 222024

I hope that you’ll have a chance to Join me at VMware Explore 2024 this year!

VMware Explore 2024, is being held at the Venetian and Palazzo in Las Vegas, on August 26 to 29th, 2024.

Image showcasing that Registration is open for VMware Explore 2024

VMware Explore is one of my favorite, and most important annual conferences for a number of reasons.

Networking

Through technology we make friends, connections, and friendships with those in our community. Chances are, you’ll see all your favorite people and that community, at VMware Explore.

You’ll have the chance to catch up with communities like VMUG (VMware User Group), the vExpert community, the vCommunity and I’m sure even some folks from World of EUC (like myself).

Additionally, you’ll get to network with like minded people passionate about the technology, experts in the field, and a diverse group of individuals from all over the world.

Learning

I can’t say how important the technical sessions are…

The sessions at VMware Explore help you learn in so many ways:

Learn about products you have interest in, but no experience
Learn more about the products you’re familiar with, become an expert!
Have a chat and dialogue with Product Managers, Presenters, and Staff about the solutions you work with, or are curious about
Catch up with and connect with experts (like vExperts)!

You can also save on certification by getting certified and taking exams at VMware Explore at a 50% discount!

Business

As the President and Owner of Digitally Accurate Inc. (a VMware and Broadcom Partner), attending this conference is crucial as it allows me to connect with customers, vendors, and VMware/Broadcom staff.

Deals get done, laughs are had, and these interactions really help advance and move business forward.

You get to have fun

And don’t forget, this event is FUN! There’s numerous events and parties that are held by vendors and community programs (such as vCommunity, VMUG, and more).

I highly recommend you keep your eyes glued to Discord, Slack, E-Mails, and the web to find the invite links to all the parties. Ask around!

Join me at VMware Explore 2024

With all that said, I hope to see you there!

Follow these hashtags to stay up to date

#VMwareExplore
#VMwareExploreHOL
#VMwareExploreSelfie
#VMwareExploreParty

Follow the official VMware Explore Social Media pages

VDI Gaming Demo with NVIDIA vGPU and Omnissa Horizon 8

ESXi, NVIDIA, NVIDIA vGPU, Omnissa Horizon, vSphere No Responses »

May 252024

Here’s a fun quick VDI Gaming Demo with NVIDIA vGPU and Omnissa Horizon 8, using an NVIDIA L4 GPU and the L4-12Q Profile.

This video is just for fun, and is just to show some of the capabilities of the technology, hardware, and software, in this case, with Cloud Gaming.

The NVIDIA vGPU solution provides the ability to “slice” and create multiple Virtual GPU (vGPU) devices for your Virtual Machines and Virtual workloads.

In this video:

Quick Introduction to NVIDIA vGPU with Omnissa Horizon 8
Validating NVIDIA vGPU functionality (with DirectX Diagnostics, Horizon Performance Monitor Tracker)
MechWarrior 5 Cloud Gaming
Heaven Benchmark

Environment Details:

2 x HPE DL360p Gen8 Servers (2 x 10 Core Procs, 384GB of RAM)
- 1 Server with NVIDIA A2
- 1 Server with NVIDIA L4
VMware vSphere 8U2
Omnissa Horizon 8

Hope you enjoy the video and demo!

Older Entries