How to manage testing

LegacyForum

Among multiple developers using WebKing

Here are some practices to ease testing with a number of developers using WebKing:

1) Make use of comments, notes and names:
Many elements in WebKing allow you to add comments, notes and set their names. For example, make sure to comment paths to show what they do in the web application. Rename tests in your test suite to indicate what they are designed to test. Choose your project names carefully so that they reflect the kinds of tests you will find in them. If you are using any custom scripting, make sure that the scripts are well commented also. All of this will help others understand what is happening when they begin working on a project they did not create.

2) Make use of test suites:
Test suites provide a useful, top-level approach to dealing with a test you want to perform on your site. A test can abstract away the details of how a path is constructed or a tool is configured and allow others to worry about the details only when necessary.
Tests are also easy to organize. They use a directory/file structure where multiple tests can be grouped under a test suite, and multiple test suites can be grouped together into a project. This allows you to categorize sets of tests under a common header, again making them easier to work with for those unfamiliar with the project.

3) Organize similar tests into projects:
Use projects as a higher level construct to represent a collection of tests that share some common aspect. For example, you may have a project that tests one function or a small subset of functions in your web application, or you may have a project that tests one subsection of your site. You may also divide projects by the kinds of tests they run, such as into static tests and functional tests.
The benefit again is that it is easier to understand what the project is testing, why it is useful, and what portion of the web application broke if the project is reporting an error. Also, when projects are kept focused and specific, it is less likely that one developer's changes will affect another developer's tests in other aspects of the web application.
Well-organized projects may be organized into a directory structure if that level of abstraction might be useful.

4) Check your projects into a source control system:
Source control systems like CVS are designed to make concurrent modification of a collection of files easier. Among other things, they provide a common ground for multiple users to get the same set of files and keep those files up to date, allow users to follow changes to files over time, and make it easy to revert unwanted changes.

5) Make use of WebKing's batch mode (command line):
(An example script for this topic is atteched below.)
WebKing's batch mode allows you to run a script from the command line. That means that with a small amount of scripting you can have WebKing scan your test directory, run all the tests suites it encounters and report the results, project by project. This is a handy way to regularly run all of your tests to check for errors that show up in your web application during development and maintenance.

This is what we do here at Parasoft. For example, when a tool is developed, a project file or a collection of project files are created for that tool. Each project file contains multiple tests, organized into test suites and commented to make it easy for other developers to figure out what the test's function and purpose is, should they need to make changes or investigate a failure. The projects and all other related files (custom rules, for example) are checked into CVS. Every night each developer updates his or her local copy of the test files by grabbing them from source control. Also nightly, an updated copy of the tests are run using the WebKing batch mode script to scan for all the tests in our test directory tree, and the next day we check the results for failures.

This is a solid system that allows us to catch regressions in our application as soon as they happen. All developers are on the same page with nightly updated tests, and all developers are able to investigate a failure when one occurs. Each developer can run all the tests by having WebKing run the batch mode script (the script is also checked into CVS) that searches for and runs all test suites. When a problem is discovered that is not a part of our tests, either a new project is created or a test is added to an existing project to ensure that bug does not resurface.

LegacyForum

The following example script will recursively scan a directory for WebKing project files and execute all the test suites it encounters.

In order to run this script, you must first install Jython and configure WebKing to use Jython by configuring the preferences under File:Customize Preferences:Scripting:Python:Python Home and Python Path. Save this script to a file and run it with the following command:

wk.exe -cmd -run C:\path\to\NameOfMyScript.py

CODE

import os
from os import *
from com.parasoft.api import *
from java.lang import *
import sys

def find_tests(dir, tests):
   for name in os.listdir(dir):
       name = os.path.join(dir, name)
       ext = os.path.splitext(name)[1]
       if (os.path.isdir(name)):
           find_tests(name, tests)
       elif (ext == '.wkj'):
           tests.append(name)

testdir = 'c:/path/to/tests/directory'
tests = []
find_tests(testdir, tests)
print 'Running test suites:'
for t in tests:
  name = str(t)
  print 'Running '+ name + ':'
  Application.execute('runTest ' + t)
print 'All tests complete.'