AQTime for C++ code coverage analysis

There’s an area of the system that I’m about to make big changes to.  I had started off by creating characterization tests as a way of understanding how the current system works while also weaving a safety net that will quickly give me feedback on the effects of my code changes.

I have a good suite of tests and was to the point where it would be helpful to me to know what areas of the code are still not exercised by any tests.  I was hankering for a code coverage tool.  I had tasted the goodness of EclEmma in Java/Eclipse land — “Is there anything available for C++?” I wondered.

I resisted stopping to learn a new tool.

I worried that support for C++ might be clunkier, less seamless than what I had experienced in Java.

I wondered what the company approval process would involve to purchase a license.

BUT, the alternatives seemed to be:

1. Visually trace through my existing tests to get an idea of the code coverage, or
2. Proceed without knowing what can break without the tests showing it

The tedium and error-proneness of alternative 1 combined with the spectre alternative 2 presented of an extended stabilization time trying to figure out afterwards what my code changes had caused overcame my reluctance, worries, and wonderings.  Besides, if I could get a code coverage solution in place, next time I could conceivably just use it, whereas if I do alternative 1 or 2 this time I’ll be faced with this same dilemma next time.

OHHHHkay.  I was moved to go ahead and try to figure this out.

Not as bad as I thought

It didn’t turn out nearly so bad as I feared.

When I asked around internally I found that as a company we already have some experience with a product called AQTime from a company called AutomatedQA.

Having gotten a license, I installed AQTime 6.3.0.

(Let me pause to mention that I find the approval process here to be remarkable.  The IT department seems to have something in common with Jimmy John’s: “Service so fast you’ll freak”.)

1. Gathering the Coverage Data

I looked confusedly and dazedly at a few tutorials, then decided to just try something.  Following is what I did (minus the dead ends and rabbit trails):

1. (Previously) Had written a suite of unit tests (using the Boost unit test framework) and built the test executable (it might need to be a debug build)
2. Dropped to a command prompt and ran the command we use to set up environment variables for our build environment
3. Started AQTime from that command prompt (otherwise the DLLs needed by the test executable were not in the path, so AQTime was not able to start the executable)
4. File -> New Project From Module… -> browsed to the unit test executable
5. Selected the Coverage Profiler (I think the default was the Performance Profiler)
6. In the Setup tab on the AQTime main window, expanded the tree view of my executable.  In my case, my tests focus on the contents of one .obj file, so I context-clicked on that and said Add Selected to Area -> Add to New Area…
   
   (The profiling level needs to be set to “Line” for code coverage profiling (the default is Routine level))
7. Pressed the Run button (the program run lasted 10 or 12 seconds instead of the usual of about 2 seconds from the command line)

2. Displaying the Results

I initially had trouble finding the visual coverage output with the source code.  It is available — I got to it thusly:

1. Went to AQtime’s Results tab
2. Double-clicked the thread under Last Results -> Source Files
3. Selected the source file in the Report tab
4. Clicked Editor in the Analyze Results panel

This Editor window provided just what I had envisioned.  Now I can quickly see what’s covered, sometimes covered, and not covered. I browsed through, looking for red dots.

The first coverage gaps I saw were in error logging code:

I knew that my tests were probably not exercising these error logging sections, but this is a good confirmation and reminder to me.  Those sections being lower risk, I’m not sure whether I’ll take the time to exercise them all… now I can make a more informed decision.

More interestingly though, here is a whole branch I didn’t realize I wasn’t testing:

This coverage check quickly showed me five to ten such branches that lack tests.  These are areas where if I made changes to the production code, I could too easily miss finding out about the breakage.  Good to know about them!

And next time I want a test coverage check, I shouldn’t have to go through all the preliminaries.  I’m glad to have a coverage checker in place.

Acknowledgment

I found the AQtime application help to be quite…helpful in getting up and running, as I was not familiar with the concept of an “area” or how to get to an editor.

That great but pesky highlighting

The Eclemma coverage tool that comes with Eclipse is really nice — it lets you quickly see what code is being run by your tests, by coloring the code green for covered and red for not covered.  After running the coverage tool, my code looks like this:

The problem is, now all the code everywhere in my project is red or green, which is distracting!  You can cause an individual file’s coverage highlighting to go away by modifying the file… but it’s annoying to have to do that to each file you look at as you browse around your project after a coverage session.

A Better Solution

It turns out there’s a handy Remove All Sessions button on the Coverage toolbar:

Press that, and our whole project returns to a state of coverage-unhighlightitude: