AQTime for C++ code coverage analysis

There’s an area of the system that I’m about to make big changes to.  I had started off by creating characterization tests as a way of understanding how the current system works while also weaving a safety net that will quickly give me feedback on the effects of my code changes.

I have a good suite of tests and was to the point where it would be helpful to me to know what areas of the code are still not exercised by any tests.  I was hankering for a code coverage tool.  I had tasted the goodness of EclEmma in Java/Eclipse land — “Is there anything available for C++?” I wondered.

I resisted stopping to learn a new tool.

I worried that support for C++ might be clunkier, less seamless than what I had experienced in Java.

I wondered what the company approval process would involve to purchase a license.

BUT, the alternatives seemed to be:

  1. Visually trace through my existing tests to get an idea of the code coverage, or
  2. Proceed without knowing what can break without the tests showing it

The tedium and error-proneness of alternative 1 combined with the spectre alternative 2 presented of an extended stabilization time trying to figure out afterwards what my code changes had caused overcame my reluctance, worries, and wonderings.  Besides, if I could get a code coverage solution in place, next time I could conceivably just use it, whereas if I do alternative 1 or 2 this time I’ll be faced with this same dilemma next time.

OHHHHkay.  I was moved to go ahead and try to figure this out.

Not as bad as I thought

It didn’t turn out nearly so bad as I feared.

When I asked around internally I found that as a company we already have some experience with a product called AQTime from a company called AutomatedQA.

Having gotten a license, I installed AQTime 6.3.0.

(Let me pause to mention that I find the approval process here to be remarkable.  The IT department seems to have something in common with Jimmy John’s: “Service so fast you’ll freak”.)

1. Gathering the Coverage Data

I looked confusedly and dazedly at a few tutorials, then decided to just try something.  Following is what I did (minus the dead ends and rabbit trails):

  1. (Previously) Had written a suite of unit tests (using the Boost unit test framework) and built the test executable (it might need to be a debug build)
  2. Dropped to a command prompt and ran the command we use to set up environment variables for our build environment
  3. Started AQTime from that command prompt (otherwise the DLLs needed by the test executable were not in the path, so AQTime was not able to start the executable)
  4. File -> New Project From Module… -> browsed to the unit test executable
  5. Selected the Coverage Profiler (I think the default was the Performance Profiler)
  6. In the Setup tab on the AQTime main window, expanded the tree view of my executable.  In my case, my tests focus on the contents of one .obj file, so I context-clicked on that and said Add Selected to Area -> Add to New Area…

    (The profiling level needs to be set to “Line” for code coverage profiling (the default is Routine level))
  7. Pressed the Run button (the program run lasted 10 or 12 seconds instead of the usual of about 2 seconds from the command line)

2. Displaying the Results

I initially had trouble finding the visual coverage output with the source code.  It is available — I got to it thusly:

  1. Went to AQtime’s Results tab
  2. Double-clicked the thread under Last Results -> Source Files
  3. Selected the source file in the Report tab
  4. Clicked Editor in the Analyze Results panel

This Editor window provided just what I had envisioned.  Now I can quickly see what’s covered, sometimes covered, and not covered. I browsed through, looking for red dots.

The first coverage gaps I saw were in error logging code:

I knew that my tests were probably not exercising these error logging sections, but this is a good confirmation and reminder to me.  Those sections being lower risk, I’m not sure whether I’ll take the time to exercise them all… now I can make a more informed decision.

More interestingly though, here is a whole branch I didn’t realize I wasn’t testing:

This coverage check quickly showed me five to ten such branches that lack tests.  These are areas where if I made changes to the production code, I could too easily miss finding out about the breakage.  Good to know about them!

And next time I want a test coverage check, I shouldn’t have to go through all the preliminaries.  I’m glad to have a coverage checker in place.

Acknowledgment

I found the AQtime application help to be quite…helpful in getting up and running, as I was not familiar with the concept of an “area” or how to get to an editor.

Advertisements

, , ,

  1. #1 by Aaron Frost on December 28, 2009 - 8:48 pm

    I have also had huge success with AQTime. Prior to refactoring an entire module in our app, we decided that we would automated all of our tests using AutomatedQA’s TestComplete. We did that, and felt good about it, but weren’t able to quantify how much testing we had actually automated. We knew that we had automated all of the current tests, but what about the holes in our current testing? We couldn’t be sure that were plugging those holes. So… we got AQTime and had it profile our automation, much like you are explaining. We got a 73% by method and 65% by lines score. However, due to the remarkable feedback that AQTime gives you, it told us exactly where we needed to add more automated tests. We filled in the holes with more automated tests. By the time we were done with the automation, we had 99.5% coverage by routine (would have been 100, but we had a dead routine that the developers refused to take out of the code) and 94% coverage by lines. This was much higher than we had expected to achieve. However, once the tests ran on the pre-refactored version of the app, we refactored the app and then ran the automation again. Within 2 days we had found and fixed over 140 bugs. The tests passed the same after the huge refactorization as the did prior to the refactorization. What we had expected to take 6 months took a little over two weeks. We were able to completely increase the stability of our system once we coupled our automation effort with a code coverage profiler.

    I agree with the author, AQTime is easy to use and a highly valuable tool.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s