There’s an area of the system that I’m about to make big changes to. I had started off by creating characterization tests as a way of understanding how the current system works while also weaving a safety net that will quickly give me feedback on the effects of my code changes.
I have a good suite of tests and was to the point where it would be helpful to me to know what areas of the code are still not exercised by any tests. I was hankering for a code coverage tool. I had tasted the goodness of EclEmma in Java/Eclipse land — “Is there anything available for C++?” I wondered.
I resisted stopping to learn a new tool.
I worried that support for C++ might be clunkier, less seamless than what I had experienced in Java.
I wondered what the company approval process would involve to purchase a license.
BUT, the alternatives seemed to be:
- Visually trace through my existing tests to get an idea of the code coverage, or
- Proceed without knowing what can break without the tests showing it
The tedium and error-proneness of alternative 1 combined with the spectre alternative 2 presented of an extended stabilization time trying to figure out afterwards what my code changes had caused overcame my reluctance, worries, and wonderings. Besides, if I could get a code coverage solution in place, next time I could conceivably just use it, whereas if I do alternative 1 or 2 this time I’ll be faced with this same dilemma next time.
OHHHHkay. I was moved to go ahead and try to figure this out.
Not as bad as I thought
It didn’t turn out nearly so bad as I feared.
When I asked around internally I found that as a company we already have some experience with a product called AQTime from a company called AutomatedQA.
Having gotten a license, I installed AQTime 6.3.0.
(Let me pause to mention that I find the approval process here to be remarkable. The IT department seems to have something in common with Jimmy John’s: “Service so fast you’ll freak”.)
1. Gathering the Coverage Data
I looked confusedly and dazedly at a few tutorials, then decided to just try something. Following is what I did (minus the dead ends and rabbit trails):
- (Previously) Had written a suite of unit tests (using the Boost unit test framework) and built the test executable (it might need to be a debug build)
- Dropped to a command prompt and ran the command we use to set up environment variables for our build environment
- Started AQTime from that command prompt (otherwise the DLLs needed by the test executable were not in the path, so AQTime was not able to start the executable)
- File -> New Project From Module… -> browsed to the unit test executable
- Selected the Coverage Profiler (I think the default was the Performance Profiler)
- In the Setup tab on the AQTime main window, expanded the tree view of my executable. In my case, my tests focus on the contents of one .obj file, so I context-clicked on that and said Add Selected to Area -> Add to New Area…
(The profiling level needs to be set to “Line” for code coverage profiling (the default is Routine level))
- Pressed the Run button (the program run lasted 10 or 12 seconds instead of the usual of about 2 seconds from the command line)
2. Displaying the Results
I initially had trouble finding the visual coverage output with the source code. It is available — I got to it thusly:
- Went to AQtime’s Results tab
- Double-clicked the thread under Last Results -> Source Files
- Selected the source file in the Report tab
- Clicked Editor in the Analyze Results panel
This Editor window provided just what I had envisioned. Now I can quickly see what’s covered, sometimes covered, and not covered. I browsed through, looking for red dots.
The first coverage gaps I saw were in error logging code:
I knew that my tests were probably not exercising these error logging sections, but this is a good confirmation and reminder to me. Those sections being lower risk, I’m not sure whether I’ll take the time to exercise them all… now I can make a more informed decision.
More interestingly though, here is a whole branch I didn’t realize I wasn’t testing:
This coverage check quickly showed me five to ten such branches that lack tests. These are areas where if I made changes to the production code, I could too easily miss finding out about the breakage. Good to know about them!
And next time I want a test coverage check, I shouldn’t have to go through all the preliminaries. I’m glad to have a coverage checker in place.
I found the AQtime application help to be quite…helpful in getting up and running, as I was not familiar with the concept of an “area” or how to get to an editor.