C++test latest Coverage Collecting Features
C++test has introduced a new functionality that will allow users to combine all coverage findings from their Static Libraries and/or previous runs into one report. We have introduced a new wrapper executable used to collect coverage, cpptestcc, and introduced a new fully customizable Runtime Coverage Library. I have put together a small tutorial to demonstrate some of the capabilities of this new feature.
Some Screenshots in this forum will be compressed to fit the size of the forum post. You can right click on an image that you have trouble seeing and click "Open image in New Tab" to get a better view.
- Add the cpptestcc wrapper with the correct compiler configuration and the indicated coverage type wrapped around your compiler. An example is below
make "CC=cpptestcc -compiler gcc_7-64 -line-coverage -- gcc"
Go to ${CPPTEST_HOME}/engine/coverage/runtime
Run make on the directory to build the C++test runtime library.
A new file called cpptest.o should be generated within the build directory.
- Add the newly generated file, cpptest.o, into the linkage step of your build system. We can modify the Timer Makefile and add the newly generated cpptest.o into the LINKSTEP environmental variable.
The linkage step should look similar to the following:
gcc clock.o driver.o timer.o stdinout.o ${CPPTEST_HOME}/engine/coverage/runtime/build/cpptest.o -o timer.exe
If you would like to use a prebuilt Coverage Runtime library go to ${CPPTEST_HOME}/engine/coverage/runtime/lib.
Windows (x86 and x86-64)
cpptest.a: 32-bit import library to be used with Cygwin GNU GCC compilers. To be added to linking command line.
cpptest64.a: 64-bit import library to be used with Cygwin GNU GCC compilers. To be added to linking command line.
cpptest.lib: 32-bit import library to be used with Microsoft Visual C++ compilers. To be added to linking command line.
cpptest64.lib: 64-bit import library to be used with Microsoft Visual C++ compilers. To be added to linking command line.
Windows CoverageDynamic Library
${CPPTEST_HOME}/engine/coverage/runtime/bin/cpptest.dll 32-bit dynamic-link library.
${CPPTEST_HOME}/engine/coverage/runtime/bin/cpptest64.dll 64-bit dynamic-link library. ${CPPTEST_HOME}/engine/coverage/runtime/bin should be added to PATH environmental variable.
Linux (x86 and x86-64)
libcpptest.so: 32-bit shared library. To be added linking command line. [INSTALL_DIR]/engine/coverage/runtime/lib should be added to LD_LIBRARY_PATH
/libcpptest64.so 64 bit shared library. To be added linking command line.
[INSTALL_DIR]/engine/coverage/runtime/lib should be added to LD_LIBRARY_PATH
Now that we have everything set up we can now collect coverage results.
- Run the following commands:
make clean
make "CC=cpptestcc -compiler gcc_7-64 -line-coverage -- gcc"
./timer.exe
This will execute the instrumented timer executable and generate a cpptest_results.clog. We will need this file for the next part of this tutorial
Import the project into the C++test IDE using any import method. (Via Makefile, bdf, import eclipse project, etc...). This can be done by going to File -> Import -> C/C++ -> Existing Code as Makefile Project.
We can load the coverage results stored within the cpptest_results.clog by using one of the latest C++test Test Configurations, Load Application Coverage. This can be found by navigating to Test Configurations -> Builtin -> Utilities -> Load Application Coverage.
You should see the following results in your Coverage Tab. If the Coverage tab is now available you can open it by going to Parasoft -> Show View -> Coverage.
You will see the clock.c shows 0/6 lines executable lines. We need to run some additional tests to make sure that we receive coverage for the file. However, we don't want to lose the coverage that we have obtained from our system level tests.
We can save now save our Quality Tasks and Coverage using the latest archive feature. We can archive by completing the following steps
Duplicate the Load Application Coverage Test Configuration so that you will be able to see the newly generated Test configuration in your User-Defined folder.
Go to Execution -> Runtime and select "Archive test and coverage results (experimental)".
Change the name of the Test Configuration to "Load Application Coverage and Archive Results" then select Apply.
This Test Configuration should resemble the following:
- Run the newly created test configuration.
This will create a new folder within the .cpptest directory. Right-click on the .cpptest folder and select Refresh to view the results. We will still need to collect more coverage from our clock.c source file.
Now that we have the .clog file archived we can generate more Unit Tests for our clock.c file so that we can receive greater coverage.
Generate Unit Tests on the clock.c file. This can be done by Right-clicking on the file and select** Parasoft -> Test Using -> Builtin -> Unit Testing -> Generate Unit Tests**
Select Generate Stubs (File Scope) on clock.c to isolate the file for testing.
Duplicate the test configuration Run Unit Tests (File Scope) and go to Execution -> Runtime and enable the options "Archive test and coverage results (experimental)" option.
(Optional Rename the Test Configuration quickly identify which Test Configuration will archive the Results)
After running your tests you should see the following:
We now have 100% coverage for clock.c file and the total percentage of Line Coverage reported by the Coverage View has increased.
However, if you try to publish the report you will see the C++test will only report the coverage from the last test execution which shows significantly less coverage than what we what.
We have introduced a new test configuration to resolve this issue.
Go to Builtin -> Utilities -> Load Archived Results and duplicate the test configuration as we have done in previous steps.
Go to Execution -> Runtime and select "Generate detailed coverage report for" then choose Line Coverage since this is the coverage this tutorial is focused on.
Run the test configuration on the top-level of the Timer project.
Go to your Test Configuration view and select the Report and Publish icon at the top-right of the view.
You will now see the generated report will now have the same coverage as reported within the Coverage view.
Click on the LC Detailed Report option at the top-right of the coverage summary section to the coverage reported at the source code level as seen in the following image.
If you would like more information on how cpptestcc and the latest Coverage Runtime Library work please refer to the following documentation.
Cpptestcc Documentation:
https://docs.parasoft.com/display/CPPDESKE1042/Collecting+Application+Coverage+with+cpptestcc#CollectingApplicationCoveragewithcpptestcc-CommandLineReferenceforcpptestccCommandLineReferenceforcpptestcc
Coverage Runtime Library Documentation:
https://docs.parasoft.com/display/CPPDESKE1042/Collecting+Application+Coverage+with+cpptestcc#CollectingApplicationCoveragewithcpptestcc-CoverageRuntimeLibraryCoverageRuntimeLibrary