How to get Jenkins to fail build with test failures?

sbritton

I have SOATest running through Jenkins, however I am seeing that if tests fail, the Jenkins build is still showing as passing. Is there something I need to add to my cli command to have it exit correctly?

Also, I'm having reports published by Jenkins, but when I look in the workspace at the reports, the View Traffic link doesn't pull up the traffic log, the links seem static.

benken_parasoft

I haven't had much success and can't get anything to generate

Make sure you know where the SOAtest XML report is being written in the job's workspace directory. In Jenkins, you can click the Workspace link to nagivate the build workspace to examine what's there, for example. Once you know the path, you must configure the Parasoft Findings "pattern" field with a pattern that will match the path of the SOAtest XML report. Alternatively, some generic pattern like "**/rep*.xml" will likely match.

After that's done, viewing SOAtest results in Jenkins is no different than viewing unit test results. You can click on the job and see the Test Result Trend graph. If you click on a specific build number then you should also see a "Test Result" link to view the test results for that build.

Is there a working example you guys have that can show me the Jenkins configuration used to generate the graphs?

I'm not aware of a public Jenkins that has Parasoft Findings enabled. However, don't sweat it. The Test Result Trend is built in, no different than Jenkins jobs that don't use Parasoft Findings, like ones that run unit tests. Once Parasoft Findings is correctly configured to find the path to the SOAtest XML report, there should be nothing extra to configure to get the trend graphs.

Also, regarding my second question, should the Jenkins workspace where reports are saved be able to display the links within the report?

By default, Jenkins sends a "CSP" HTTP header back to the browser, instructing the browser to restrict certain things. For the SOAtest HTML reports from Jenkins to behave as expected you likely need to disable CSP. To do this you set the hudson.model.DirectoryBrowserSupport.CSP property to empty string. You could do this on the Jenkins command line with -Dhudson.model.DirectoryBrowserSupport.CSP=. Or from the Jenkins script console you could execute System.setProperty("hudson.model.DirectoryBrowserSupport.CSP", ""). For specific details, please visit the Jenkins docs: https://wiki.jenkins.io/display/JENKINS/Configuring+Content+Security+Policy

benken_parasoft

Example post build step in a job config:

Notice the fields for configuring the build status based on the test results.

OmarR

Hey sbritton,

In addition to @benken_parasoft 's suggestions above, You may also be interested in looking over Spencers Debrosse's wonderul tutorial on configuring SOAtest with Jenkins:
https://blog.parasoft.com/continuous-integration-with-parasoft-soatest

OmarR

Hello Sbritton,

The Jenkins build succeeds because it was able to execute your SOAtest tests successfully. I recommend looking into the Jenkins plugin for SOAtest which allows you to integrate your test results directly into your Jenkins Job.

You may download the plugin on Parasoft Marketplace or Jenkins Marketplace.

benken_parasoft

Are you using Parasoft Findings? This will do the job.
https://wiki.jenkins.io/display/JENKINS/Parasoft+Findings+Plugin

Parasoft Findings leverages the xUnit Jenkins plugin which has configurable thresholds for marking builds as unstable or failed.

sbritton

@benken_parasoft @OmarR Thanks I'll look into how to get the plugin to work, I haven't had much success and can't get anything to generate. Is there a working example you guys have that can show me the Jenkins configuration used to generate the graphs?

Also, regarding my second question, should the Jenkins workspace where reports are saved be able to display the links within the report?

sbritton

Interesting! In the help documentation that I had prior to 9.10.3 didn't have the -fail option documented. This weekend I updated to 9.10.3. After reviewing the blog post to verify that option works, I removed all post build actions and adding -fail will cause the build to fail, which is exactly what I was looking for.

Thanks, guys! I'll review your suggestions with my team tomorrow and get this stuff working in Jenkins. Appreciate all the help!

benken_parasoft

In the help documentation that I had prior to 9.10.3 didn't have the -fail option documented

For reference, there is also soatestcli -help which prints:

-fail
    Fail the build by returning a non-zero exit code if any violations are
    reported.

However, I do think Parasoft Findings still offers a lot of value beyond from making builds fail based on test results. We really appreciate feedback about things like this.