Welcome to the new Parasoft forums! We hope you will enjoy the site and try out some of the new features, like sharing an idea you may have for one of our products or following a category.

soatest report "Failed Tests" incorrect

huiraym
huiraym Posts: 24 ✭
in SOAtest

Hi newbie question...
I have a test suite which reads an xlsx with 10 test cases.
Of the 10 test cases, only the 3rd one failed.
I would expect the report should show 9 successful and 1 failed. But instead, it shows 2 successful, 8 failed.
It seems like any test cases executed after the failed one will be counted as failed. Is there an option to change to have the failed tests number correctly?
Thanks.

Comments

  • benken_parasoft
    benken_parasoft Posts: 1,306 ✭✭✭

    If your tests are using a data source then they will execute multiple times, once for each row from the data source. Any violations should indicate which test failed as well as the row number. A single test can run multiple times and fail multiple times. The totals you see in the report reflect the actual number of executions of each test.

    If you have doubts about which tests are passing or failing then you can also get a more detailed breakdown. To do this, click "Parasoft > Preferences... > Reports", enable "Overview of checked files and executed tests", then disable "Only top-level test suites".

  • huiraym
    huiraym Posts: 24 ✭
    edited October 16

    I tried your suggestion but the number in Summary is the same, it doesn't match with the actual...
    In this case, i made my 4th test to fail and now the report says 7 failed Tests :(

  • benken_parasoft
    benken_parasoft Posts: 1,306 ✭✭✭

    I am not able to reproduce the behavior you describe. I suspect you won't see the problem with the current SOAtest release.

  • huiraym
    huiraym Posts: 24 ✭

    Is there a way to check whether this is a bug in version 2021.2?

  • benken_parasoft
    benken_parasoft Posts: 1,306 ✭✭✭
    edited October 17

    Resolved issues are listed in the release notes. You are probably running into this one from the 2022.1 release notes:

    SOA-14977 - Incorrect summary results presented by SOAtest in HTML report

  • benken_parasoft
    benken_parasoft Posts: 1,306 ✭✭✭

    Until you are able to upgrade, you can try enabling the Legacy Report Format in case this helps.

  • huiraym
    huiraym Posts: 24 ✭

    Thank you Benken. Sorry i am not quite undertand how to...

    "This can be done in desktop installations by configuring the following command line argument: -J-Dparasoft.legacy.report.format=true"

    Does it mean i launch SOATest.exe -J-Dparasoft.legacy.report.format=true"

    There is no ini file or on UI i can enable or disable this flag right?

  • huiraym
    huiraym Posts: 24 ✭

    Thank you so much. I added this arg in the shortcut and it works. You are the best!!!!

Categories