Parasoft Static Analysis Jobs are Inconsistent
I am running Static Analysis across an entire workspace worth of projects with some key characteristics:
Number of Projects: ~160
Number of Files: ~ 1000 .c/.cpp and ~1000.h
1. Limit is not set on number of violations per rule
2. Check box for "static/Flow Analysis Advanced Settings/Compact Incremental Caches " Checked
3. Depth of analysis: Standard
4. Timeouts are off
5.Enable Swapping of analysis data to a disk
The following are facts about the configuration of my projects:
- Every project is created from an identical base project and is capable of having Parasoft run across it
- Every file exists in only one project
- Files are virtually linked into each project
- Runs are completed by, within the Eclipse GUI, selecting all projects, right clicking, and selecting the configuration to run
- Every run is initiated identically
- I have a script that has been verified to accurately distill the information found in the report.xml generated by the static analysis run into a deterministic easily read text file
When I run static analysis using the configurations/processes listed above, I see inconsistency in the static analysis results.
When I run 3 identical runs consecutively on the same workspace, on the same computer, with the same test configuration, and then compare the results, inconsistencies are found, meaning that each run has violations that were and were not reported on the other runs. These violations should be expected to be consistent, should they not? There are certain violations we know should exist and be caught by the tool, however, those violations do not appear on every run.
Is there any reason, or something that I missed during these runs, that would cause the differences in results?