Understanding Metrics in CloudTest Analytics

Document created by Chris Sommerstad Employee on Jul 22, 2017
Version 1Show Document
  • View in full screen mode

The following table defines CloudTest Analytic metrics as they appear in the Results Details and other dashboard widgets in test results. Definitions are categorized by the order of hierarchy in the test composition itself, first by action, then by Containers inside of a Clip, and then by the Clip itself).

 

Metric LevelUser Interface NameDefinition

Actions

Metrics for an individual action

  
 Start TimeThe time that play started as an offset from the start of the test, in milliseconds (ms).
 Delta From Scheduled TimeIf in a timed container, the offset of the actual start time from the scheduled time, in milliseconds (ms).
 Response Time

The complete amount of time spent establishing the connection, sending the request, and receiving and processing the response. Excludes any time spent waiting for a connection to be available from the internal connection pool. If no response is received from the server, then this will be 0.

For WebSocket Send, the response time is very small or zero because sending is handled asynchronously.

For WebSocket Message (received), the response time will depend on whether a server-side timestamp (set by the target server) is present in the response.

  • If a timestamp is present in the response, the response time measures from the timestamp to CloudTest receiving the message.
  • If there is no timestamp in the response, the response time is very small or zero.
 Bytes SentThe total number of bytes sent to the remote server, including the request line, the headers, the body, and all separating whitespace.
 Bytes ReceivedThe total number of bytes received from the remote server, including the status line, the headers, the body, and all separating whitespace. If the response requires any post-processing (e.g. decompression or binary-to-text-conversion), this will be the number of bytes *before* the post-processing took place.
 Retries due to unavailable local portsThe total number of times that the request needed to be re-tried due to insufficient local ports on the SOASTA server. Typically, 0.
 Retry time due to unavailable local portsThe total amount of time that the request spent on requests that failed due to insufficient local ports on the SOASTA server. Does not include the time spent on the request that succeeded. Typically, 0.
 Connection Establish TimeThe amount of time spent establishing the HTTP connection, including the SSL handshake (if any) and DNS lookup. If a previously-opened connection is re-used, then this will be 0.
 Send TimeThe total amount of time spent sending the request content to the server (including request line, headers, and body). Does not include any of the following: establishing the HTTP connection, waiting for the server to respond, or processing the server response.
 Receive TimeThe total amount of time spent receiving and processing the response content from the server. The term "processing" includes de-compressing (if the response was compressed), performing any binary-to-text conversions (e.g. AMF), and/or parsing the bytes according to the selected character set.
 Total TimeThe total amount of time spent sending a Message and receiving the response. Includes "Send Time", "Receive Time" and any necessary re-tries. Messages that did not receive a response due to connection timeouts or other errors, will still have a total time. For Non-Messages, the Total Time is the time to complete the action. Delays, Browser Actions, App Actions, and Database Actions will have a total time.
 TTFBThe amount of time from the start of the request to the first byte being received from the server. Includes the following: establishing the HTTP connection, sending the request to the remote server, and waiting for the first response byte.
 TTLBThe amount of time from the start of the request to the last byte being received from the server. Includes the following: establishing the HTTP connection, sending the request to the remote server, and receiving the complete response. Does not include response "processing" (e.g. de-compressing, etc.).
 DNS Lookup TimeThe amount of time spent resolving the host name using DNS. If a cached DNS result is re-used, or if the request is being sent directly to an IP address, then this will be 0. This is a subset of "Connection Establish Time".
 Wait for Connection Pool TimeThe amount of time spent waiting for a free connection from the HTTP connection pool. If there is no need to wait (i.e. new connection is opened, or free connection is available immediately), then this will be 0.
 StatusCompleted, Stopped or Failed.
   
   

Containers inside of a Clip

Metrics for an individual Container inside of a Clip

  
 Start

The time that play started as an offset from the start of the test, in milliseconds (ms).

 Delta From Scheduled TimeIf in a timed container, the offset of the actual start time from the scheduled time, in milliseconds (ms).
 Full Duration (Shown by default as Effective Duration)The length of time for the container to start until it completed, in milliseconds (ms). This is not the time for the elements within the container. The duration can be greater than or less than the sum of the elements within it. For example, a Page container will contain Messages which run in parallel so the sum will often be greater than the duration.
 Message Play CountThe number of Messages that were completed in any way (success or fail) in this container and any containers it contains
 Browser Action Play CountThe number of Browser Actions that were completed in any way (success or fail) in this container and any containers it contains
 App Action Play Count The number of App Actions that were completed in any way (success or fail) in this container and any containers it contains
 Database Action Play CountThe number of Database Actions that were completed in any way (success or fail) in this container and any containers it contains
 Script Play CountThe number of Scripts that were completed in any way (success or fail) in this container and any containers it contains
 Response CountThe number of any type of operation (Message, Browser Action, App Actions, Database Action) that completed a "request" (success or fail). This number could be different than the number of Actions completed. For example, a Message might have failed to create the request due to some error, such as an improper ISSE in the request. This would result in the count of Messages being played going up, but not in the Response Count going up, since no request could be sent. For actions in this container and containers it contains.
 Min Response TimeThe smallest value of the clip element’s duration for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Max Response TimeThe largest value of the clip element’s duration for all actions in this container and containers it contains that did complete a request (see the collection's Response Count).
 Total Response TimeThe sum of all of the clip element’s duration values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count). Note that actions can execute in parallel, especially in Pages, so this sum can be longer than the duration of the container itself.
 Base Page Average Response TimeThe first clip element in a Page container is the Base Page (e.g. index.html). The Base Page Avg Response Time is the mean time to get that base page not including its assets (i.e. .png, .js, .xml, etc.).
 Total Bytes SentThe sum of all of the Total Bytes Sent values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count).
 Total Bytes ReceivedThe sum of all of the clip element's Total Bytes Received values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Effective Message ThroughputThe number of messages per second, and bytes per second sent and bytes per second received, for all messages in this container and containers it contains that did complete a request.
 Min Port Retry CountThe smallest value of the clip element's Total Port Retry Count for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Max Port Retry CountThe largest value of the clip element's Total Port Retry Count for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Total Port Retry CountThe sum of all of the clip element's Total Port Retry Count values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Min Port Retry TimeThe smallest value of the clip element's Total Port Retry Time for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Max Port Retry TimeThe largest value of the clip element's Total Port Retry Timet for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Total Port Retry TimeThe sum of all of the clip element's Total Port Retry Time values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Delay Play CountThe number of Delay Clip Elements that were completed in any way (success or fail) in this container and any containers it contains
 Main Message Response CountThe number of Messages in a Main (Document) section of a Page that completed a "request" (success or fail) (see the collection's Response Count). This number could be different than the number of Pages completed. For example, a Message might have failed to create the request due to some error, such as an improper ISSE in the request. This would result in the count of Pages being played going up, but not in the Response Count going up, since no request could be sent even though the Page did start. For Pages in this container and containers it contains.
 Main Message Response TimeThe sum of all of the clip element’s duration values for all Messages in a Main (Document) section of a Page that completed a "request" (success or fail) (see the collection's Response Count). For all Pages in this container and containers it contains.
 Effective DurationThe total amount of time, in milliseconds (ms), that this container was "effective" (doing work). Doesn't include the time between items (such as inter-item delays or inter-repeat delays). Takes into account overlapping executions -- the effective time is when at least one or more Clip Elements are executing anywhere within the container or its child containers.

Clip

Metrics for an individual Clip

  
 StartThe time that play started as an offset from the start of the test, in milliseconds (ms).
 Response Count

The number of any type of action (Message, Browser Action, App Actions, Database Action) that completed a "request" (success or fail). This number could be different than the number of Actions completed. For example, a Message might have failed to create the request due to some error, such as an improper ISSE in the request. This would result in the count of Messages being played going up, but not in the Response Count going up, since no request could be sent. For actions in this clip and containers it contains.

 Min Response TimeThe smallest value of the clip’s duration for all actions in this container and containers it contains that did complete a request (see the collection's Response Count).
 Max Response TimeThe largest value of the clip’s duration for all actions in the clip and containers it contains that did complete a request (see the collection's Response Count).
 Total Response TimeThe sum of all of the clip’s duration values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count). Note that actions can execute in parallel, especially in Pages, so this sum can be longer than the duration of the container itself.
 DurationThe length of time for the container to start until it completed, in ms. This is not the time for the elements within the container. The duration can be greater than or less than the sum of the elements within it. For example, a Page container will contain Messages which run in parallel so the sum will often be greater than the duration.
 Delta From Scheduled TimeIf in a timed container, the offset of the actual start time from the scheduled time, in milliseconds (ms). Generally, used only for timed clips.
 Delta Scheduled DurationThe difference between the scheduled duration and the actual one. Generally, used only for timed clips.
 Total Bytes SentThe sum of all of the Total Bytes Sent values for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count).
 Total Bytes ReceivedThe sum of all of the clip's Total Bytes Received values for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count).
 Min Port Retry CountThe smallest value of the clip's Total Port Retry Time for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count)
 Max Port Retry CountThe largest value of the clip's Total Port Retry Count for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Total Port Retry CountThe sum of all of the clip's Total Port Retry Count values for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count)
 Min Port Retry TimeThe smallest value of the clip's Total Port Retry Time for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count)
 Max Port Retry TimeThe largest value of the clip's Total Port Retry Timet for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
 Total Port Retry TimeThe sum of all of the clip's Total Port Retry Time values for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count)
 Start Offset from ContainerThe offset of this clip's start time from its parent's start time, in milliseconds (ms).
 Total Main Message Response CountThe sum of all of the clip’s duration values for all Messages in a Main (Document) section of a Page that completed a "request" (success or fail) (see the collection's Response Count). For all Pages in this container and containers it contains.
 Total Main Message Response TimeThe sum of all of the clip’s duration values for all Messages in a Main (Document) section of a Page that completed a "request" (success or fail) (see the collection's Response Count). For all Pages in this container and containers it contains.
 Effective DurationThe total amount of time, in milliseconds (ms), that this clip was "effective" (doing work). Doesn't include the time between items (such as inter-item delays or inter-repeat delays). Takes into account overlapping executions—the effective time is when at least one or more Clip Elements are executing anywhere within the container or its child containers.
 Errors in Descendants

Attachments

    Outcomes