Nested Compositions and the Test Suite Dashboard are distinct features that can be used in tandem for debugging functional tests and to quick identify regression errors or issues. Both are relevant to functional testers using CloudTest and TouchTest.
The Nested Compositions feature introduced a new (SOASTA 53.05) clip element type, Launch Composition. This clip element can be placed into the test clip the same as any other clip element. Using this feature CloudTest and TouchTest functional testers can launch compositions from other compositions, and view Test Suite results in the new Test Suite Dashboard. The ability to leverage Jenkins CI and other CI tools remains undiminished.
Test Suite Dashboard
This System Dashboard (also introduced in SOASTA 53.05) is purpose-built for web and mobile Functional Testing and allows the tester to easily browse through the history of a test suite and the test compositions in it, right down to the action level where validations pass/fail, and where regressions can be detected and fixed.
Using the Launch Composition Clip Element
The Launch Composition clip element is a call to a Composition from another Composition (via a Test Clip in the calling Composition). A Composition is added into a Test Clip via the Clip Editor lower panel, Compositions tab, and the resulting clip element is a Launch Composition clip element.
Adding a Launch Compositions clip element into the Clip works precisely the same as adding any other clip element—double click the item to place it at the insertion point, or, drag it into place in the Clip Editor workspace.
More than one Launch Composition clip element can be placed into a Test Clip(s) in order to create a Test Suite.
Additionally, the Set Launch Composition context command can be used to set another Composition as a Launch Composition.
When this command is selected, the Choose a Composition box appears. Select the Composition to set as a Launch Composition.
Launch Compositions have all of the following capabilities:
- They can Repeat, be placed in Transactions, have properties, and all of the things that Clip elements do.
- They have properties including "whether or not to wait for the comp to complete", "whether failure should fail the parent", "optional name for the result", etc.
- When the parent Composition that contains the Clip is played its result can be viewed in the Test Suite Dashboard (which loads by default); an individual result is also created for each Launch Composition.
- Results of parent Compositions show the Launch Composition elements in the tree, but not its children.
Note: In this release, Device Selection is done in the nested composition itself and cannot be overridden in the parent composition.
Launch Composition Options
Launch Compositions, like Nested Clips, have their own in situ properties that are applicable only to the nested instance (these options are distinct from the stand-alone properties that apply to the stand-alone Composition). After adding a Launch Composition to your Clip, expand the new clip element in Clip Editor, List View to display its Options. Alternately, select the Launch Composition clip element and view it in the Selected: <""> panel.
- Location – For non-functional tests specify the Maestro Location to use. Refer to Specifying Maestro Locations for more information. Locations doesn't apply to functional testing so functional testers can accept the default.
- Errors Should – This drop-down specifies which failure action to take if an error occurs in the Launch Composition. Select one of the standard Failure Actions that apply to all clip elements
- If the failure action is set to record only, the validation outcome will be recorded in the result, and the launch composition will continue whether the validation passes or fails.
- If the failure action is set to fail the parent, the parent clip will stop if the validation fails.
- Wait For Completion – Checked by default.
- If checked, this Clip Element will not complete execution until the Composition completes.
- If unchecked, this Clip Element will complete once the Composition has been started, and will not wait for its completion.
- Verify Successful Play – Unchecked by default.
- If checked, this Clip Element will fail if the launched Composition does not complete with status Completed. What then happens is determined by the value of the Failure Action attribute.
- This attribute only has meaning if the Wait For Completion attribute is checked. If that attribute is unchecked, this attribute is not relevant.
- Allow Partial Load – Unchecked by default.
- Checked indicates that Composition play should proceed even if all of the necessary servers were not found. Locations doesn't apply to functional testing so functional testers can leave this unchecked.
- Result Name – Optionally, specify an alternate name for the Result name (e.g. other than the default name).
- Result Description – Optionally, specify a description for the result.
Building a Test Suite
Test Suites can be built in a number of ways, including adding all of the relevant Compositions as Launch Compositions into a test clip that resides in a parent test composition with no other test clips.
Playing this parent Composition creates a result for the Test Suite (although individual test composition results are also created).
More sophisticated Test Suites can be built that contain many such Clips, each of which contains one or more Launch Compositions. For example, a Test Suite for mobile apps and devices such as the example below can be arranged by Track, each of which corresponds to a device type, OS version, or to a network type. The Clips that have Launch Compositions are organized into the appropriate Track.
When you play a parent Composition (e.g. your Test Suite), the Composition Editor opens to the Play tab with the Test Suite Dashboard in display.
The Test Suite Dashboard presents functional data as a summary of the output from a "test suite." The Test Suite is used to analyze the results of a functional test across many results, and in the TouchTest use case, mobile devices. The Test Suite Dashboard incorporates key metrics, and the results of a functional test, including Success Summary [the X and number of successful Compositions, Clips, Validations]; Details about the Compositions and its test results [pass/fail, time/date of last test run], and the ability to drill down into details.
Test Suite Dashboard
As part of the Test Suite Dashboard, a new Widget Type category—Functional—has been added to the Widget Type list.
The Test Suite Dashboard includes the following widgets:
Functional Summary – This widget displays Summary Statistics, including the Success percentage of successful out of the total for Compositions, Validations, and Clips. The Functional Summary can also be added to custom dashboards.
Compositions Overview – This widget shows the components of the parent Composition and their history by component.
The Composition Overview widget includes the following columns:
- A Composition column that can be expanded to drill into child components
- A Last Result column that shows the most recent result (pass/fail) of the specific item for that line (e.g. in the result that was opened)
- A Prior 10 Results column that shows the pass/fail result for each row before the Last Result (most recent from left to right).
- A Start Time column that shows the last start time for the item in the row
- A Time to Complete column that shows the total time that the item (a Composition, Clip, or Validation) last took to execute
The Compositions Overview tree starts from the Clips inside the parent Composition. Only Compositions, Clips, and Validations are shown here, while Tracks, Bands, Messages, App, and all grouping Clip Elements are omitted.
The selected dot in Last 10 Results drives the Result Details selection below.
Create a Report Using Composition Overview
As of SOASTA 57, reports render the Composition Overview widget using native tables. You can generate a report to view results in HTML or MS Word.
You can create a report by logging into your CloudTest instance and clicking Compositions on the leftmost panel. In your list of Compositions, double-click your Composition and select Create a New Report.
When the Create a New Report dialog will appears, choose between a MS Word or HTML report template. In the Widgets section, expand All Other Widgets > Functional, and then select Compositions Overview.
For more information on reports, please visit the Creating Report Templates and Reports help article.
Result Details – This is the existing Result Details dashboard for that Result without the cover flow and with the left-hand tree starting at the selected element from above. Users can also select in the Result Details tree itself to navigate through test elements.
As already noted, the Result Details widget is also driven by the selected result dot in the Last 10 Results column (shown below).
In this iteration of the Result Details widget, the Navigation Tree contains all Clip Elements (Groups, Chains, etc.) that are children of the Clip in the existing Result Details dashboard.
Note: Errors indicated in the Result Details UI will "bubble up" to the parent item(s) of the failed element. This was true of errors in the prior release only if the Failure Action for the given element was set to "Also fail the parent." An element such as a Composition may succeed, but if any of its children have failed, then the Composition icon in the tree will show a red-x. This change applies to all contexts of the Result Details widget—whether it is in view in the full Result Details Dashboard, here in the Test Suite Dashboard, or in any custom dashboard that includes the Result Details widget. Additionally, bubble-up errors are shown all the way to the composition-level (in the prior release, they were shown only up to the track-level if the Failure Action was set to fail the parent).
Selection in the Compositions Overview will navigate to the corresponding object in Result Details below. Additionally, the Result Details widget in this context shows only the relevant panels (for example, if the selection is a Validation, the relevant Accessor panels are shown).
Accessing the Test Suite Dashboard
The tester can enable to access the Test Suite Dashboard by any one of the following ways:
- Opening a parent Composition in the Composition Editor, Results tab.
- Opening the result of a parent Composition from Central (in which case, the Test Suite Dashboard is shown by default)
- Opening the Build in Jenkins and viewing the test result of a parent Composition