The Transaction Threshold command—which was made available in the SOASTA CloudTest Plugin, version 2.17—provides users with the ability to automate the validation of Transactions in a composition using CloudTest metrics, such as Average Response Time or Errors per Transaction.
For example, if a threshold of 3000 ms is set on Average Response Time, any result outside that limit will fail the test.
Transaction Thresholds are automatically validated during each Jenkins build and can be set on any non-nested transactions you've placed into your test composition(s).
Transaction Threshold validation status is posted in the Jenkins build log (shown above) for each threshold that has been set. Using this feature provides a simple means to track regression(s) that are caused by the build actions that changed (whether deployment, code change, etc.). By setting Transaction Thresholds, the plug-in can be made to easily track regressions, or other types of issues caused by this recent change.
Using Transaction Thresholds in Jenkins
The Transaction Threshold feature appears in the CloudTest Jenkins Plugin's Play Composition(s), Additional Options section, which appears whenever you click the Advanced button (shown below).
- To add a Transaction Threshold, click Add.
- Once Add is clicked, the Transaction Thresholds form appears.
- Enter the Transaction Name to track. If you're not sure of the transaction's name, go to your CloudTest/TouchTest instance and inspect the test clip that contains it, and then note its name to enter here. For example, Transaction1.
- Click the Threshold drop down to select from the available Thresholds.
Transaction Threshold Metrics
Average Response Time - The average amount of time spent establishing the connection, sending the request, and receiving and processing the response for objects in this transaction. Response time excludes any time spent waiting for a connection to be available from the internal connection pool
90th Percentile - CloudTest calculates a 1-tailed confidence interval, meaning that the outliers removed to obtain the 90th percentile are the slow response times, but not the really fast ones.
Min Response Time - The smallest value of the transaction’s duration for all actions in it and any containers it contains that did complete a request.
Max Response Time - The largest value of the transaction’s duration for all actions in this container and containers it contains that did complete a request.
Total Bytes Sent - The sum of all of the Total Bytes Sent values for all actions in this transaction and containers it contains that did complete a request.
Errors Per Transaction - Non-zero if any items in this transaction or the containers it contains had errors. "Having errors," means an item that completed successfully but generated events of level "Error" during play, as well as items that failed during play.
Minimum Transaction Count – The smallest sum of the transactions that completed a request.
- Once the Threshold is selected, enter a value. For example, 234 (expressed in milliseconds) for Max Response Time below.
- If more than one Transaction will be tracked, click Add to place additional Transaction Threshold forms into the Jenkins job.
- Click Save to save and exit the Jenkins job after making changes, or click Apply to save and continue editing the job configuration.