I started with automated tests for my MZ-Tools extension early in the development of version 7.0 (then an add-in, not a package), ten years ago, around the year 2008 or so. At that time Visual Studio 2008 provided a Visual Studio TestHost dll (even with source code, if I remember correctly) to run automated tests of Visual Studio packages using the own Visual Studio. I remember that it was so painful to use that approach (crashes, hangings, etc.) that after months of investment I threw all the stuff and started to build my own test framework infrastructure and test runner:
(yep, I have 3,354 automated integration tests for my extension for Visual Studio)
It was perfect in most aspects because I owned the code and could adapt it to my needs but it had one important inconvenient: it didn’t allow Continuous Integration (CI). Instead, I ran the tests manually before each (monthly) build. It had another inconvenient: I learned automated testing by myself without formal training and I ended with tons of integration tests (which use the real Visual Studio to host the extension), no system tests (that would mock Visual Studio) and no unit tests (that just test a single method or class). I know now that, for performance reasons, it is much better to have have a piramyd with tons of unit tests, lots of system tests and few integration tests.
Last year I started to focus on DevOps (even if I am a solo developer) and continuous integration, and decided to make my test framework infrastructure compatible with VS Test / MSTest, so I could keep using my own test runner for integration tests but use them 100% as system tests mocking Visual Studio with stubs:
As you can see, my progress is quite modest at this point (only 133 system tests vs 3,354 integration tests) because I need to mock the full EnvDTE.FileCodeModel for most tests. But it is a feasible approach because I have fully mocked with stubs the whole VBA / VB6 IDEs that I use for MZ-Tools 8.0 for VBA / VB6. So, for example I have 1,822 integration tests for VB6 with my own test runner (an add-in for VB6):
Once I finish the system tests for Visual Studio, the next step is to give another try to run the integration tests using the test runner of Visual Studio, rather than my own test runner, which will allow me Continuous Integration not only for system tests, but also for integration tests. For such approach, I learned in the Two videos about building High Performance Extensions by Omer Raviv about the Microsoft/VisualStudio-TestHost project on GitHub. I have still to read and learn about it, but I think it is based on the code of the VS TestHost of 2008 and hopefully these 10 years have been used to solve the problems that caused me to abandon that approach.