qga-vss: Exit with non-zero code when register fail

QGA installer uses rundll32 to run the DLLCOMRegister function
from qga-vss.dll and perform VSS provider registration.
rundll32 ignores the return value of the function and always
exits with a zero exit code. This causes a situation where
the installer does not know the status of VSS provider registration.

This commit forces to change exit code when the VSS provider
registration fails.

https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/rundll32

Reviewed-by: Yan Vugenfirer <yvugenfi@redhat.com>
Tested-by: Dehan Meng <demeng@redhat.com>
Message-ID: <20250620083132.28347-1-kkostiuk@redhat.com>
Signed-off-by: Kostiantyn Kostiuk <kkostiuk@redhat.com>

1 files changed, 4 insertions, 1 deletions

diff --git a/qga/vss-win32/install.cpp b/qga/vss-win32/install.cpp
index 5cea5bc..6ee2f44 100644
--- a/qga/vss-win32/install.cpp
+++ b/qga/vss-win32/install.cpp
@@ -385,7 +385,10 @@ out:
 STDAPI_(void) CALLBACK DLLCOMRegister(HWND, HINSTANCE, LPSTR, int);
 STDAPI_(void) CALLBACK DLLCOMRegister(HWND, HINSTANCE, LPSTR, int)
 {
-    COMRegister();
+    HRESULT hr = COMRegister();
+    if (FAILED(hr)) {
+        exit(hr);
+    }
 }
 
 STDAPI_(void) CALLBACK DLLCOMUnregister(HWND, HINSTANCE, LPSTR, int);