Using LoadTest Continuum in DevOps delivery pipeline
Running performance tests as a part of the continuous delivery pipeline is becoming a best practice in DevOps environments. You can start integrating performance tests into the continuous delivery pipeline by adding selected performance tests to Jenkins or a continuous integration tool of your choice and having them run regularly. Depending on your needs you can run performance tests at one or more of the following points in the build/test infrastructure:
After each build with a reduced set of performance “smoke” tests.
Once a day with a more comprehensive set of performance tests.
Once a weekend or based on infrastructure availability with a set of long running tests for endurance testing or high volume load tests for stress testing.
This by itself, however, is not enough. Manually analyzing load test reports can be time consuming and may require special skills not every developer possesses. Without the ability to automate load test report analysis, reviewing performance test results becomes a tedious time sink. Vital performance information may also get overlooked. In such scenarios you may be running performance tests continuously, but the benefit of them will be limited.
To get the full benefit of continuous performance testing you need to set up an effective mechanism to analyze performance test results. LoadTest and LoadTest Continuum (LTC) provide you with tools that help automate performance tests results collection and analysis and give you insights into the performance of your application. The following steps will help you set up your environment for continuous performance test execution with LoadTest and LoadTest Continuum:
Review and configure LoadTest project QoS metrics for automation.
Deploy and configure LoadTest Continuum for load test report collection.
Configure LoadTest projects into batches for execution.
Start running LoadTest project batches as a part of continuous integration, and use LoadTest Continuum to regularly review and analyze performance test results.
Review and Configure QoS Metrics for Automation
LoadTest Quality of Service (QoS) metrics are one of the key features for automating the analysis of performance test results. QoS metrics reduce large amounts of data in a load test report to a set of success/failure answers about your application performance. LoadTest offers a rich set of QoS metrics that range from ready-to-use threshold metrics to custom scripted metrics that allow you use the LoadTest API for advanced load test data analysis.