Load server calibration is an iterative process to determine the appropriate number of virtual users to run from each load server. The process must be followed for each test clip that will run on a cloud provider (and each instance size on a cloud provider). Depending on the permutations of these variables, the calibration process might take some time.
The below table that classifies scenarios by type and provides a high level indicator for the development effort and potential impact on VUs per server.
|Low||Low||Browse only: will generally get more users per 'large' load generator. This should be the baseline assumption if there is one scenario and it's browse only. This could get over 1000+ users if think times are extended between browsed pages.|
|Low to Medium||Medium||Number of Steps: Most scenarios are between 5-15 steps, though some can have 1-200 steps or more. The actual recording of the scenarios is not drastically more difficult. However, the more steps, the greater the likelihood that scripting and tuning will be required. An hour to start, and then an hour for every 10 steps is a good rule of thumb. At runtime, generally, longer scripts will require more resources because there's a greater likelihood of script complexity. However, a long script doesn't automatically mean more processing power is required.|
|High||Low||Number of Scenarios: an obvious impact on development time. When sizing, total number of users is generally not impacted greatly as long as the scenario characteristics are consistent. There is very little difference between 5 similar scenarios with 200 users each and 1 scenario with 1000 users.|
|Medium||Medium||Parameterization: using a set of values from the customer is generally not a huge impact assuming you get good data from the customer. Login is the most common requirement, others include account numbers, product IDs, etc. The more variability there is in the use of runtime parameters, the fewer users will be supported.|
|High||High||Correlation and Session based data: can be high if the application is not well understood or the dynamic content requires scripting to properly enter the data. If the data needs to be retained throughout the session, such as for transactions or shopping cart checkout, this will add complexity to the scenario and impact the user count. Will impact sizing if there is more than one or two correlations that need to be carried through the scenario. We see this often with logged in user flows where the user ID must be constant throughout the user flow.|
|Low to High||High||Custom Logic: in some cases there may be very simple custom logic involved, but in others the requirement might be for some complex financial calculation, for example. The resource utilization of the logic will drive the sizing impact and the dev impact will likely be related to whether or not the script/logic is already created. We see this often in random browse then add to cart as there must be logic for any of the items that may be added randomly. For instance there may be options for color, waist size and leg length if a pair of jeans were to be added to a cart but a dress shirt may have neck and arm sizing only.|
|Low||Low to Medium||Type of Test: Stress and load tests are often designed to throw load quickly at a site, with faster ramps. This will likely result in fewer users per generator. Soak, capacity and performance tests generally are built to reflect real world scenarios and can handle greater user counts.|
|Medium||Medium||Type of Content: This will impact the size of the test primarily in the area of bandwidth consumption. A browse scenario that downloads a video will probably accommodate more users simply because the 'think' times are longer. However, content that is spread out among a number of URLs and aggregated may increase complexity slightly.|
|Low||Low||Geographic Distribution of Load: Very little impact on either test development or sizing, other than that an appliance/load generator is required for each physical location.|
|Low||High||Think times: shorter think times per user will reduce the number of users that a server can generate. However, shorter think times may also decrease the number of users necessary to simulate a specific load.|