A test job is a set of one or more test scenarios that are associated with a specific environment configuration. When a test job is executed—on-demand or via automation tools such as the Environment Manager Jenkins plugin—the designated environment will be provisioned before the tests are executed.

Test jobs are configured and run from the Test Scenarios page.

Reviewing Available Test Scenarios and Jobs

Test Scenarios

The upper left panel lists all of the .tsts that are available in each connected SOAtest Server’s TestAssets folder. This list is automatically populated with the test scenarios from each server’s TestAssets folder (see Integrating Virtualize Server and/or SOAtest Server with CTP for details). You can also extend and modify these test suites directly from CTP; for details, see Building Scenarios and Tests.

To view details about a scenario, select its test case tree node.

Note that you can search for scenarios (top left of page) as well as jobs (middle left of page). The search covers .tst/job names as well as associated metadata.

Jobs

The Jobs panel allows you to create, search filter, review, execute, and remove jobs, as well as review/remove job execution results.

The Jobs panel is automatically populated with jobs for all of the available SOAtest Test Executor component instances. These jobs are automatically synchronized with the associated SOAtest Test Executor component instances: if .tsts are added, removed, or reconfigured in the SOAtest Test Executor, those changes will be applied here as well.

You can filter the available jobs to focus on specific systems and environments of interest.

Adding Test Jobs

To add a test job:

  1. Do one of the following
    • From the list of jobs, click the Add New Job icon.
       
    • From the scenario details view, click the Create Job button.
       
  2. Specify a name for the test job.
  3. If you want to limit the job history that is saved, specify the maximum days and/or runs you want to allow.

  4. Click Add Test Scenarios and specify which test scenarios you want to run, in what order, and with what variables and data sources.

    Specifying test execution settings: The Execute tests setting determines whether test scenarios are batch-processed on each designated server or run separately in the specified order. Batch (fastest) optimizes performance. Sequential allows you to control the execution order and generates a separate report for each test scenario.

    Specifying test execution order: If Execute tests is set to Sequential, test scenarios will be executed in the order in which they are specified here. You can reorder tests by dragging and dropping them to the desired positions.

    Configuring additional options: To see additional configurable options for a test scenario (such as data groups you can activate or SOAtest environment variables you can modify), click the arrow to the right of that scenario’s name.

    Specifying test variable mappings: If a test scenario has variable sets defined, they will automatically be configured to use the existing values. Alternatively, you can specify literal values, use the values of variables defined in CTP, or use whatever value is active in the .tst at the time of the test. If you want to modify the variable mapping for this test job, see Configuring Variable Mappings..

    Specifying data groups: Specify which of the available data sets you want to be active for each test scenario with data groups.



    Specifying environments: By default, each scenario will be added with the variables from the active variable set (also known as a SOAtest environment). If you want to control what set of variable set is added with each scenario, hover over the .tst name, click the Environments link that appears...

    then click the Add link for the variable set(s) you want to add. 

    The green ball icon marks the variable set that is currently active for the given test scenario (on the SOAtest server).

  5. (Optional) If you want an environment provisioned upon test execution, specify the appropriate system, environment, and set of component instances in the upper right side of the page. Under Select an Instance, select a specific environment instance to use that snapshot of component instance settings or select Custom if you want to manually configure the active component instance settings.
     

Executing Jobs

Jobs can be executed directly from the UI, or as part of an automated Jenkins job.

Executing Test Jobs from the UI

To use the UI to kick off a test job and provision any associated environments:

  1. From the Jobs panel, open the job you want to modify.
  2. Click Execute.

The specified test environment will be provisioned, then the tests will be executed. Progress and results will be indicated in the Jobs panel.

Tip: Executing Your Own Version of a Saved Job—without Impacting Team Members

Sometimes you might want to quickly run an existing job with new environment and/or variable settings—without saving changes to that job and potentially impacting other team members who are also working with that job. 

In this case, just select the job, configure the desired settings (environment context, variables, etc.), and click Execute without first saving the job.

Be sure to select the top-level job node rather than a time-stamped job history node.

Executing Test Jobs Automatically

The Parasoft Environment Manager Plugin for Jenkins (available on Parasoft Marketplace) can automatically run test jobs as part of a Jenkins job. This plugin is designed to help you rapidly configure various actions needed for automated, continuous testing across your software delivery pipeline. For more details, see the description and documentation available on Marketplace.

Reviewing Test Job Results

To review a test job’s execution results:

  1. From the Jobs panel, expand the associated test job, then click the test run you want to review.
  2. Click View report to open the execution report.

Icons in the Jobs panel indicate test outcomes.

Managing Test Jobs

Modifying Test Jobs

To modify a test job’s .tsts or execution settings:

  1. From the Jobs panel, open the job you want to modify.
  2. Make the desired modifications.
  3. Click Save.

Cloning Test Jobs

To clone an existing test job:

  1. From the Jobs panel, open the job you want to duplicate.
  2. Click the Clone Job icon.

Clearing a Test Job’s History

To clear a job’s history:

  1. From the Jobs panel, open the job whose history you want to clear.
  2. Click Clear Job History.

Removing Test Jobs or Test Runs

To remove a test job or test run:

  • Click the Delete icon in the Jobs panel.

Distributing Test Job Execution Across a Cluster of Servers (Execution Group)

If you want to distribute test job execution across an "execution group" (a cluster of SOAtest servers grouped under the same server name), ensure that those servers all have:

  • The same name (this is specified in the SOAtest Preferences> Environment Manager panel).
  • All of the .tst files you want executed in this manner.

The first of these servers to connect to CTP will be treated as the primary server in the execution group; the others will be considered alternates. The SOAtest server page shows only the primary servers (one server for each unique server name).

The page for a specific SOAtest server contains a table of the other servers in this same "execution group," along with their current status (online or offline). When the primary server is refreshed, all servers in that execution group will be refreshed.


The list of servers in the execution group is also shown when you select the primary server in the Test Scenarios page.

To run the distributed tests, simply ensure that all servers are running, then configure and execute the desired job to run on the primary server. If that primary server is busy executing another job, CTP will execute it on one of the other servers within that cluster.

 

  • No labels

Need assistance? Visit our support page