VS Packages

Code signing a VSIX package targeting multiple Visual Studio versions

You know that I am such a big fan of targeting multiple Visual Studio versions with the same VSIX file that I wrote an article for MSDN Magazine explaining the approach to target  from Visual Studio 2012 to Visual Studio 2017. In that article I didn’t explain how to code sign such VSIX file to work with multiple Visual Studio versions but this week I have got a (personal) code signing certificate.

First things first, if you don’t code sign your VSIX file, the VSIX installer shows that the digital signature is none:

To code sign a vsix file, you can’t use the regular signtool.exe tool of the Windows SDK. You need to use the vsixsigntool.exe that you can get from NuGet:

And the official documentation to sign VSIX packages is here:

Signing VSIX Packages

In the samples that follow, I will assume that you have the vsixsigntool.exe file and a code signing certificate named CodeSigningCertificate.pfx in the same bin\debug folder that contains a vsix file named MyVSIXProject.vsix.

If you specify in the source.extension.vsixmanifest file that your VSIX package only targets Visual Studio 2012 (11.0) and 2013 (12.0):

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[11.0,12.0]" />
 </Installation>

and you sign the VSIX file with with the /fd sha1 option to use the SHA1 algorithm (do not confuse with the /sha1 option, which is used to select the correct certificate):

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha1 MyVSIXProject.vsix

then, although initially you can get this “invalid certificate” error:

after a few attempts while connected  to internet then it works as expected (I don’t know yet why the initial error, does anyone know?):

However, if you sign the VSIX file with with the /fd sha256 option to use the recommendable SHA256 algorithm:

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha256 MyVSIXProject.vsix

then you get an error because the algorithm is not supported:

Now, if you specify in the source.extension.vsixmanifest file that your VSIX package only targets Visual Studio 2015 (14.0) and 2017 (15.0):

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[14.0,15.0]" />
 </Installation>

and you sign the assembly with with the /fd sha1 option to use the SHA1 algorithm:

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha1 MyVSIXProject.vsix

then the VSIX installer shows “Invalid signature”:

That is because Visual Studio 2015 and higher no longer support the SHA1 algorithm, only the SHA256 algorithm. If you sign the VSIX file with with the /fd sha256 option to use the SHA256 algorithm:

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha256 MyVSIXProject.vsix

then the VSIX installer shows the correct signature:

But, what if you specify in the source.extension.vsixmanifest file that your VSIX package targets from Visual Studio 2012 (11.0) to Visual Studio 2017 (15.0)?:

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[11.0,15.0]" />
 </Installation>

Of course, using the SHA256 algorithm would case the “invalid signature” warning when installing the VSIX package for Visual Studio 2012 or 2013. But if you use the SHA1 algorithm:

vsixsigntool.exe sign /f CodeSigningCertificate.pfx /sha1 "<sha1 bytes>" /p MyPassword /fd sha1 MyVSIXProject.vsix

then, because the VSIX installer notices that the VSIX package is also valid for Visual Studio 2012 and 2013, it no longer shows “Invalid signature” for Visual Studio 2015 and 2017:

Note: according to Ed Dore in this post, this approach wouldn’t work with older versions of the VSIX Installer if the manifest states explicitly the higher version:

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[11.0,15.0]" />
 </Installation>

and the workaround was not to specify the higher version:

 <Installation>
 <InstallationTarget Id="Microsoft.VisualStudio.Professional" Version="[11.0,)" />
 </Installation>

But that bug is already fixed and you can specify the higher version.

VS Extensibility

The VisualStudio-TestHost project to execute interactive UI tests in Visual Studio

3 Comments

I started with automated tests for my MZ-Tools extension early in the development of version 7.0 (then an add-in, not a package), ten years ago, around the year 2008 or so. At that time Visual Studio 2008 provided a Visual Studio TestHost dll (even with source code, if I remember correctly) to run automated tests of Visual Studio packages using the own Visual Studio. I remember that it was so painful to use that approach (crashes, hangings, etc.) that after months of investment I threw all the stuff and started to build my own test framework infrastructure and test runner:

(yep, I have 3,354 automated integration tests for my extension for Visual Studio)

It was perfect in most aspects because I owned the code and could adapt it to my needs but it had one important inconvenient: it didn’t allow Continuous Integration (CI). Instead, I ran the tests manually before each (monthly) build. It had another inconvenient: I learned automated testing by myself without formal training and I ended with tons of integration tests (which use the real Visual Studio to host the extension), no system tests (that would mock Visual Studio) and no unit tests (that just test a single method or class). I know now that, for performance reasons, it is much better to have have a pyramid with tons of unit tests, lots of system tests and few integration tests.

Last year I started to focus on DevOps (even if I am a solo developer) and continuous integration, and decided to make my test framework infrastructure compatible with VS Test / MSTest, so I could keep using my own test runner for integration tests but use them 100% as system tests mocking Visual Studio with stubs:

As you can see, my progress is quite modest at this point (only 133 system tests vs 3,354 integration tests) because I need to mock the full EnvDTE.FileCodeModel for most tests. But it is a feasible approach because I have fully mocked with stubs the whole VBA / VB6 IDEs that I use for MZ-Tools 8.0 for VBA / VB6. So, for example I have 1,822 integration tests for VB6 with my own test runner (an add-in for VB6):

And the same 1,822 tests as system tests using the test runner of Visual Studio:

Once I finish the system tests for Visual Studio, the next step is to give another try to run the integration tests using the test runner of Visual Studio, rather than my own test runner, which will allow me Continuous Integration not only for system tests, but also for integration tests. For such approach, I learned in the Two videos about building High Performance Extensions by Omer Raviv about the Microsoft/VisualStudio-TestHost project on GitHub. I have still to read and learn about it, but I think it is based on the code of the VS TestHost of 2008 and hopefully these 10 years have been used to solve the problems that caused me to abandon that approach.

Update (July 29, 2018): yesterday I finished the last stubs for Visual Studio and now all the integration tests for the real Visual Studio are also system tests with stubs that simulate the behavior of Visual Studio:

VS Extensibility

Two videos about building High Performance Extensions by Omer Raviv

Omer Raviv, author of the OzCode extension for Visual Studio, has recorded two videos with Robert Green on Channel 9 about building high performance extensions for Visual Studio. If creating extensions for Visual Studio is already tricky, creating high performance extensions is quite difficult and Omer has some advices and techniques about it.

The first video is this one:

Building High Performance Extensions Part 1

And the second one is this:

Building High Performance Extensions Part 2

.NET Compiler Platform ("Roslyn")

BUG: EnvDTE.CodeElement.GetStartPoint(vsCMPartBody) / GetEndPoint(vsCMPartBody) throw COMException for expression-bodied methods and properties

Some days ago I got a bug report from a user that was using the new expression-bodied methods and properties introduced by C# 6.0, which have the following form:

public class Class1
{
   public int Function1() => 0;
   public int Property1 => 0;
}

After some investigation, it turned out that the problem was in the calls to get the start/end points of the body of the EnvDTE.CodeElement representing them:

codeElement.GetStartPoint(vsCMPartBody)
codeElement.GetEndPoint(vsCMPartBody)

For those kind of code elements, the code model throws COMException (not even NotImplementedException), when it should return valid values since those expression-bodied members have…well, a body.

Today I have reported the bug on the GitHub project for Roslyn, since Microsoft changed the implementation of the automation code model (EnvDTE.Project.CodeModel, EnvDTE.ProjectItem.FileCodeModel) for Visual Studio 2015 to be based on the underlying .NET Compiler Platform (“Roslyn”).

EnvDTE.CodeElement.GetStartPoint(vsCMPartBody)/GetEndPoint(vsCMPartBody) throw COMException for expression-bodied methods and properties

This implementation change caused some bugs in Visual Studio 2015 betas that fortunately were fixed after I reported them, and some calls to GetStartPoint / GetEndPoint are not implemented as I reported two years ago:

EnvDTE.CodeElement.GetStartPoint(part)/GetEndPoint(part) not implemented for many parts

As a workaround for this bug, I assumed that the expression-bodied member expands to a single line, and I searched the string “=>” to find the start of the body.

VS Extensibility, VS Packages

MSDN Magazine article: Creating Extensions for Multiple Visual Studio Versions

4 Comments

Back in April or May I started to write a post on this blog about creating extensions for multiple Visual Studio versions. I have been struggling to support multiple versions of Visual Studio with my MZ-Tools add-in since Visual Studio .NET (2002) / 2003, and other IDEs such as VB6/VB5/VBA before that. So, I know how tricky it can be to share as much code as possible between versions, and using a single binary dll / setup if possible. It seems that I am not the only one wanting to target multiple Visual Studio versions with a single vsix package because I have seen lots of questions in the forums, chat rooms and private email asking about it. The post on this blog was never finished, because it was becoming so long that I thought it would be… a good article for MSDN Magazine! And finally today it has been published in the August issue:

Visual Studio – Creating Extensions for Multiple Visual Studio Versions

VS 2017

About the new privateregistry.bin file of Visual Studio 2017

3 Comments

As I explained in the post Some implications of the new modular setup of Visual Studio 2017 for VSX developers, Visual Studio 2017 introduces among others two significant changes compared to Visual Studio 2015:

  • It allows several Visual Studio editions (Community, Professional, Enterprise) to coexist at the same time on the same machine. For VSX developers, this means that Visual Studio 2017 installations now use different folders on disk, and instance Ids.
  • It uses its own private registry. This post is about this.

From Visual Studio .NET 2002 to Visual Studio 2008, Visual Studio used two registry keys:

  • HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VisualStudio\<version>: this per-machine entry was created when Visual Studio was installed (which required admin rights) and 3rd party extensions could add registry entries to it (to register packages or DDEX providers, etc.) because it was never deleted (otherwise Visual Studio would become unusable and would require reinstallation).
  • HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\<version>: this per-user entry was created the first time Visual Studio was launched for a user account, and contained per-user settings. It was never deleted (otherwise user settings would be lost).

This worked fine for a few years, until Microsoft wanted Visual Studio not to require admin rights to run or even to install 3rd party extensions, and to allow file-based registration of such extensions using .pkgdef files. So, a third registry key was introduced:

  • HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\<version>_Config
that per-user entry was created by Visual Studio when it deemed it convenient merging two sources:
  1. The per-machine Registry configuration, HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VisualStudio\<version>
  2. Per-user and per-machine .pkgdef files on disk.

So, from time to time, Visual Studio 2010 copied to a per-user configuration the per-machine configuration (source #1) along with per-machine and per-user .pkgdef files on disk (source #2). This process happened when Visual Studio was launched for the first time on a user account, after a change of configuration, etc. And for this to work, at run-time the devenv.exe process redirects  HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VisualStudio\<version> to HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\<version>_Config.

All that means that 3rd party setups of extensions shouldn’t write directly to the HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\<version>_Config key, because it could be deleted and created again at any time. Instead, setups should either write directly to HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VisualStudio\<version> (which requires admin rights), to HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\<version> (which doesn’t require admin rights), or, better yet, avoid completely writing to the registry and use instead .pkgdef files on disk, either on per-machine folders (which requires admin rights) or on per-user folders (which doesn’t require admin rights).

So far so good, but now Visual Studio 2017 stops using those three registry keys and it uses its own private registry. This is a file named privateregistry.bin which is located in the (hidden) folder C:\Users\<user>\AppData\Local\Microsoft\VisualStudio\<version>_<instance-id>, where <version> is 15.0 and <instance-id> is a random value determined when each Visual Studio 2017 edition is installed:

If you want to use regedit.exe to load that private registry, follow the steps detailed in How to examine Visual Studio 2017 registry. You will end seeing this:

At this point you should be aware of all this in Visual Studio 2017:

  • The private registry (privateregistry.bin file) is per-user, not per-machine. As such, it is not created when Visual Studio is installed, but when Visual Studio is launched for the first time for a user account.
  • The private registry provides two keys: 15.0_<instance-id> and 15.0_<instance-id>_Config.
  • The key 15.0_<instance-id> is equivalent to HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\<version> in previous versions. As such, it is never deleted (otherwise a user would lose her per-user configuration).
  • The key 15.0_<instance-id>_Config is equivalent to HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\<version>_Config in previous versions. As such, it can be deleted and recreated by Visual Studio when it needs to update the configuration.
  • There is no per-machine private registry. So, there is no equivalent of HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VisualStudio\15.0 (well, such key exists, but it stores minimal settings). The installation of Visual Studio 2017 uses exclusively per-machine .pkgdef files instead that are scanned when Visual Studio is loaded for the first time for a user account to create the key 15.0_<instance-id>_Config in the per-user private registry.
  • At run-time, the devenv.exe process of Visual Studio 2017 redirects registry operations on Visual Studio keys to the privateregistry.bin file. So, this change is transparent for extensions (DLLs) that run in the devenv.exe process. Setups (which are external processes) are not so lucky. While they could use the RegLoadAppKey function to write to the 15.0_<instance-id> key for per-user extensions (never for 15.0_<instance-id>_Config, which would be overwritten later), it is much better to switch to .pkgdef files on disk (either per-machine, or per-user). If you really need to mess with the Visual Studio setup, then see Microsoft/vs-setup-samples.
VS 2017, VS Extensibility

Sample code and utilities to get installed Visual Studio 2017 editions programmatically

As I explained in the post Some implications of the new modular setup of Visual Studio 2017 for VSX developers, Visual Studio 2017 has changed all that you knew about installations of Visual Studio. In this episode of Channel 9, Art Leonard explains to Robert Green the internals of this re-architecture of Visual Studio:

The use of a private registry file causes that if you want to know programmatically the installed editions of Visual Studio 2017, the old approaches don’t work. For example, my article HOWTO: Detect installed Visual Studio editions, packages or service packs is now obsolete.

Fortunately, Microsoft provides a new Setup API to query the installed editions of Visual Studio 2017 or the highest VSIXInstaller.exe, along with sample code and utilities:

Visual Studio Setup Configuration Samples
Microsoft/vs-setup-samples
“This is a sample in various programming languages that demonstrates how developers can use the new Visual Studio setup query API. The included samples show how to use the new setup configuration API for discovering instances of Visual Studio 2017”.

Visual Studio Locator
Microsoft/vswhere
“Over the years Visual Studio could be discovered using registry keys, but with recent changes to the deployment and extensibility models a new method is needed to discover possibly more than once installed instance. These changes facilitate a smaller, faster default install complimented by on-demand install of other workloads and components. vswhere is designed to be a redistributable, single-file executable that can be used in build or deployment scripts to find where Visual Studio – or other products in the Visual Studio family – is located.”

Visual Studio Setup PowerShell Module
Microsoft/vssetup.powershell
“This PowerShell module contains cmdlets to query instances of Visual Studio 2017 and newer. It also serves as a more useful sample of using the Setup Configuration APIs than the previously published samples though those also have samples using VB and VC++.”

VSIX Installer Bootstrapper
Microsoft/vsixbootstrapper
“An installer that can be chained with other packages to locate the latest VSIXInstaller.exe to use for installing VSIX extensions. One of the great new features of Visual Studio 2017 is an initial smaller and fast install. To compliment a smaller – but powerful – initial feature set, installing additional workloads and components on-demand is supported for both end users and package developers. Package developers can install their VSIX extensions for Visual Studio using this bootstrapper to find the latest version of VSIXInstaller.exe and install their extension(s). This may be preferable for extensions that support Visual Studio 2017 or newer than installing extensions in Windows Installer .msi packages, since MSIs cannot run concurrently in separate processes. Other deployments may also benefit since they no longer have to find where VSIXInstaller.exe is installed. The command line passed to VSIXBootstrapper.exe is passed through to VSIXInstaller.exe.”

VS 2017, VS Extensibility, VS Packages

Building a VSIX extension with the Visual Studio 2017 Build Tools

As I explained in the post Migrating the build of a VSIX project to a build server if you are a solo developer, I am taking the steps to build my MZ-Tools extension on a build/release server. As part of that process, I realized than rather than installing Visual Studio 2017 Community edition on the server, I could use the Visual Studio 2017 Build Tools that were thought, well, for build servers that don’t need the overhead of a Visual Studio 2017 installation. They are a lightweight version of Visual Studio 2017 without the IDE (devenv.exe executable). They can be used to build either managed (C#, VB.NET, etc.) projects or native (C++) projects. Incidentally my MZ-Tools solution has both type of projects.

The Visual 2017 Build Tools can be downloaded from here. Once you install them on a clean machine, you will notice that they provide only the following:

  • A built-in (non-optional) set of components to build MSBuild-based projects (for example managed projects).
  • An optional workload “Visual C++ build tools”.
  • An optional workload “Web development build tools”.

There are also optional individual components to install .NET Framework 4.6.1 support Windows SDKs, ATL support, etc.:

In my case my extension needs to use .NET Framework 2.0 for some projects (I still support Visual Studio 2005). Since that version is not installed by default on modern versions of the Windows OS, I need to install it going to “Control Panel”, “Programs and Features” item, “Turn Windows Features on or off” link:

My obfuscator tool needs the .NET Framework 3.5 SDK (or the .NET Framework 2.0 SDK). While the Visual Studio 2017 Community installer provides the optional individual component “.NET Framework 3.5 developments tools”, the installer of Build Tools 2017 doesn’t. That is not only a pity but also causes a bug if you install yourself the Windows 7.0 SDK that contains the .NET Framework 3.5 SDK that I reported here: the resource .resx files of a .NET project targeting .NET Framework 2.0 are compiled using the Assembly Linker (al.exe tool) of the .NET Framework 4.0, which will cause them to fail silently at run-time. Microsoft fixed the bug just in time for RTM in Visual Studio 2017, but the Build Tools 2017 still has the bug due to the lack of the “.NET Framework 3.5 developments tools”. There is a fix that I explained in the bug report if you find this problem. I have also requested to Microsoft to include the “.NET Framework 3.5 developments tools” in the installer of Build Tools 2017.

My extension for Visual Studio targets version 2012, so I need to stick to .NET Framework 4.5, not some higher version. Since that version is not provided by the Build Tools 2017, I need to install the Windows 8.0 SDK, that contains the .NET Framework 4.5 assemblies and SDK.

For the C++ projects, I needed to install:

  • “Visual C++ ATL Support”: required to get files such as atlbase.h, etc.
  • “Windows 8.1 SDK”: I could upgrade to some Windows 10 SDK version but they occupy much more space on disk.
  • “UCRT SDK”: the Universal Common RunTime that provides files such errno.h and other files in the folder “C:\Program Files (x86)\Windows Kits\10\Include\10.0.10240.0\ucrt”.
  • “.NET Framework 4.6.1 SDK”: to get files such as mscoree.h / mscoree.lib in the folder “C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\”.

My build script uses tf.exe to set a workspace and download the latest sources. Alas, tf.exe is not installed with Build Tools 2017 (Visual Studio 2017 installs the Team Explorer extension, that contains that file in the folder “C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer”). Now there is a standalone installer for Team Explorer (that includes tf.exe) announced here. Since that’s a bit overkill, I just copied tf.exe and the required dlls.

The build script needs also nuget.exe, but that’s easy to get.

So, when I thought that I had all the required components installed, I tried to build the extension. I got the following problems:

First, the error “Error’MSB4226: The imported project “(…)\VSSDK\Microsoft.VsSDK.targets” was not found.” Initially I thought it was a bug, that I reported to Microsoft here, but I discovered that the problem was solved setting the “VisualStudioVersion” MSBuild property, something that a machine with the full Visual Studio 2017 does and that a machine with the Build Tools 2017 does if you open a developer command prompt. Since I was not using it, I passed it as a parameter to the MSBuild script. It can be defined too inside the .csproj file, something that previous Visual Studio versions did automatically but recent versions don’t.

Then I got an error about a missing Microsoft.VisualStudio.Settings.15.0 file. How is that a file from Visual Studio is required by the NuGet package that provides the Visual Studio SDK?. It happens that Microsoft.VsSDK.Build.Tasks.dll, the file that contains the MSBuild tasks needed when creating a VSIX file, references it. But on a machine without Visual Studio, that dll is not present. I discovered that despite the error, the VSIX file was generated correctly, so that DLL was required for a task after generating the VSIX file. That task is to deploy the generated VSIX file to the experimental instance for debugging. Since on a release server that step is not needed, I knew how to instruct the .csproj project file to avoid it:

<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
   ... 
   <!-- Do not deploy the extension in Release configuration -->
   <DeployExtension>False</DeployExtension>
</PropertyGroup>

Finally, when I thought that all obstacles were solved, I got an error about the Microsoft.Visualstudio.Shell.Interop.dll being delay-signed rather than strong-signed. Since my development machine has tons of Visual Studio versions and SDKs, I discovered that all the Microsoft.Visualstudio.Shell.Interop.dll files were strong-named except an old one, which somehow Microsoft shipped delay-signed, and that was the one that I was referencing. So, I only need to change it by the correct version. And finally, my extension generates a VSIX file on a release server with only the Build Tools 2017 plus some additional components, but without Visual Studio 2017 installed.

The next step is to install some agent to connect to the Build/Release Management of Visual Studio Team Services.

VS Extensibility

Microsoft, the Visual Studio Shells and the old versions

2 Comments

Visual Studio 2008 started to offer a new form of extensibility: a whole IDE for your app! That is, reusing the core of the Visual Studio IDE for your own tool instead of reinventing the wheel creating a new IDE from scratch. This is named Visual Studio Shell and there are two flavors: the “Isolated Shell“, where your app has its own instance of Visual Studio Shell even if the Visual Studio IDE is already installed (because the user is a developer); and the “Integrated Shell“, where if the Visual Studio IDE is already installed, your Visual Studio Shell integrates with it (otherwise is installed anyway).

I have never worked with the Visual Studio Shell, nor I have any knowledge, but I provide a page on this site for the download links. Alas, it seems that Microsoft doesn’t pay much attention to keep the links working, so I get questions on forums and by email about their availability. If the links of that page fail, you can try the following:

It would be nice if Microsoft provided a centralized, public page with the downloads that VSX developers may require to develop and test extensions (Visual Studio SDKs, .NET Framework SDKs, Visual Studio Shells, Community editions, etc.) and, very important, from the current Visual Studio version going back to… Visual Studio 2005 and .NET Framework 2.0. Not everyone wants or can use the latest bits. I keep a couple of hard disks at home and at my basement with all the required Visual Studio versions, Visual Studio SDKs, Windows SDKs, .NET Framework SDKs, Windows OS versions,  etc. fearing that they would disappear from the web some day (which certainly is happening with the shells). And now that Microsoft uses web-based installers for Visual Studio with no .iso file provided, I am even afraid that my Visual Studio 2017 installer won’t work in a few years, so I have to create an offline installer.

The strange case of...

The strange case of VB6 getting “access denied” building ActiveX DLL running with Registry virtualization

This is one of those posts that I write mainly for myself in the future, but that maybe can be useful for others developers that want or need to run Visual Basic 6.0 without admin rights on Windows 7 or Windows 10.

One of the flavors of my MZ-Tools add-in is for Visual Basic 6.0 (yes, there is still quite a few people using it). Visual Basic 6.0 was created and used heavily in a time (Windows 98-Windows XP) when every Windows user was an administrator, so it was not a problem that when building an ActiveX DLL project some registry entries were added to the HKEY_CLASSES_ROOT (HKCR) registry hive, actually to HKEY_LOCAL_MACHINE (HKLM) internally. On Windows Vista and higher with the User Account Control (UAC) that changed, so if you run Visual Basic 6.0 without admin rights you get this error when building an ActiveX DLL:

A solution is of course to run Visual Basic 6.0 with admin rights, and I have been doing it so for some years. But that requires Visual Studio running also with admin rights, to be able to debug an add-in project loaded on Visual Basic 6.0 (if Visual Studio is not running with admin rights, it prompts you to restart with admin rights, but that’s quite annoying). But running Visual Studio all the time with admin rights only because of Visual Basic 6.0, when all the IDEs (Visual Studio, VBA) of my other add-ins wouldn’t require it, is somewhat overkill, and I really wanted to avoid it.

So, I was aware of the Registry virtualization mechanism of UAC, which Microsoft invented precisely for this scenario: an application that required admin rights to write in some keys of the HKLM\Software key of the Registry but that was not allowed to run with admin rights. Since Registry virtualization was not working for Visual Basic 6.0, my first thought was that it no longer existed on Windows 10 (documents say that apps cannot rely on this mechanism forever since it will be removed in some Windows version), or that it needed to be activated at global level. But no, if you open the Local Security Policy and go to Local Policies > Security Options, the setting exists and file and registry write failures are virtualized by default:

My second thought was that it needed to be activated per-application. But no, all the documents say the opposite, applications can disable registry or file virtualization by adding a .manifest, which signals to the Windows OS that the application is UAC-compliant (and therefore virtualization is not required). And indeed Process Explorer revealed that vb6.exe runs with virtualization:

So, what was happening? Process Monitor to the rescue. I noticed that Visual Basic 6.0 was trying to write to HKCR\TypeLib, getting “access denied”, not directly to HKLM:

But HKCR is actually a merged view of some keys of HKLM and some keys of HKCU, and clearly the failure was due to the HKLM side.

Finally, a closer reading of the page Registry Virtualization on MSDN, section Controlling Registry Virtualization, gave me the clue:  it happens that not all the keys of HKLM\Software are virtualized by default. In particular, the keys and subkeys of HKLM\Software\Classes, which are used when registering an ActiveX component, are not virtualized by default. Fortunately, you can use the reg.exe tool to query and virtualize them.

So I launched an elevated (with admin rights) command prompt and queried the HCKR\Classes\TypeLib (actually HKLM\Software\Classes\TypeLib) in 32-bit Registry (Visual Basic 6.0 is a 32-bit executable) with this command:

reg.exe FLAGS HKLM\Software\Classes\TypeLib QUERY /reg:32

Note: the fact that Visual Basic 6.0 is a 32-bit executable and that Process Monitor showed HKCR\TypeLib (that would belong to 64-bit) and not HKCR\WOW6432Node\TypeLib (that would belong to 32-bit) means that both keys (32-bit and 64-bit) are the same.

I got that it is not virtualized (REG_KEY_DONT_VIRTUALIZED: SET):

To clear that flag I executed the following, that clears the DONT_VIRTUALIZED flag but keeps set the RECURSE_FLAG as before:

reg.exe FLAGS HKLM\Software\Classes\TypeLib SET RECURSE_FLAG /reg:32

Trying to build again the ActiveX DLL causes the same “Error accessing the system registry”, but this time Process Monitor revealed that the problem was in HCKR\WOW6432Node\Interface:

Notice that in this case HCKR\WOW6432Node\Interface is shown, instead of HCKR\Interface, which means that the 32-bit and the 64-bit keys are different.

So I queried the flags with this command:

reg.exe FLAGS HKLM\Software\Classes\Interface QUERY /reg:32

And I got that certainly is not virtualized:

Again, to clear that flag I executed the following, that clears the DONT_VIRTUALIZED flag but keeps set the RECURSE_FLAG as before:

reg.exe FLAGS HKLM\Software\Classes\Interface SET RECURSE_FLAG /reg:32

Trying to build again the ActiveX DLL caused the same “Error accessing the system registry”, but this time Process Monitor revealed that the problem was in HCKR\Interface, so I executed:

reg.exe FLAGS HKLM\Software\Classes\Interface SET RECURSE_FLAG /reg:64

And finally building the ActiveX didn’t cause any error.

VS SDK, packages, add-ins, macros and more…