Setting Transaction Thresholds in Jenkins

Document created by DPM Admin Employee on Jul 20, 2017Last modified by Jijius Jabez on Aug 10, 2017
Version 3Show Document
  • View in full screen mode

The Transaction Threshold command—which was made available in the SOASTA CloudTest Plugin, version 2.17—provides users with the ability to automate the validation of Transactions in a composition using CloudTest metrics, such as Average Response Time or Errors per Transaction.

 

For example, if a threshold of 3000 ms is set on Average Response Time, any result outside that limit will fail the test.

 

Transaction Thresholds are automatically validated during each Jenkins build and can be set on any non-nested transactions you've placed into your test composition(s).

 

Note: The Service-Level Agreement feature is an alternative to setting up a Transaction Threshold using the Jenkins Plugin. Please see the Service-Level Agreement help article for more details.

 

Tip: The CloudTest Jenkins Plugin uses the scommand utility to output the JUnit xml output to display validations on transaction. Any JUnit-ready tool can use this JUnit XML output. For more information, refer to the Knowledge Base article, sCommand Reference (Play command, Validations argument).

 

 

Transaction Threshold validation status is posted in the Jenkins build log (shown above) for each threshold that has been set. Using this feature provides a simple means to track regression(s) that are caused by the build actions that changed (whether deployment, code change, etc.). By setting Transaction Thresholds, the plug-in can be made to easily track regressions, or other types of issues caused by this recent change.

 

Using Transaction Thresholds in Jenkins

 

The Transaction Threshold feature appears in the CloudTest Jenkins Plugin's Play Composition(s), Additional Options section, which appears whenever you click the Advanced button (shown below).

 

  • To add a Transaction Threshold, click Add.

  • Once Add is clicked, the Transaction Thresholds form appears.

 

  • Enter the Transaction Name to track. If you're not sure of the transaction's name, go to your CloudTest/TouchTest instance and inspect the test clip that contains it, and then note its name to enter here. For example, Transaction1.
  • Click the Threshold drop down to select from the available Thresholds.

 

Note: When you set up a Transaction Threshold using this section, you are setting up an implicit validation on the test composition(s) defined in the same Jenkins job. When the threshold result is outside the limit defined here, the test validation will fail.


Transaction Threshold Metrics

 

Note: The Service-Level Agreement feature is an alternative to setting up a Transaction Threshold using the Jenkins Plugin. Please see the Service-Level Agreement help article for more details.

 

Average Response Time - The average amount of time spent establishing the connection, sending the request, and receiving and processing the response for objects in this transaction. Response time excludes any time spent waiting for a connection to be available from the internal connection pool

 

90th Percentile - CloudTest calculates a 1-tailed confidence interval, meaning that the outliers removed to obtain the 90th percentile are the slow response times, but not the really fast ones.

 

Min Response Time - The smallest value of the transaction’s duration for all actions in it and any containers it contains that did complete a request.

 

Max Response Time - The largest value of the transaction’s duration for all actions in this container and containers it contains that did complete a request.

 

Total Bytes Sent - The sum of all of the Total Bytes Sent values for all actions in this transaction and containers it contains that did complete a request.

 

Errors Per Transaction - Non-zero if any items in this transaction or the containers it contains had errors. "Having errors," means an item that completed successfully but generated events of level "Error" during play, as well as items that failed during play.

 

Minimum Transaction Count – The smallest sum of the transactions that completed a request.

  • Once the Threshold is selected, enter a value. For example, 234 (expressed in milliseconds) for Max Response Time below.

jenkins_plugin_6.png

  • If more than one Transaction will be tracked, click Add to place additional Transaction Threshold forms into the Jenkins job.

  • Click Save to save and exit the Jenkins job after making changes, or click Apply to save and continue editing the job configuration.

Attachments

    Outcomes