How to handle large file uploads (png, jpg, pdf, etc.) in CloudTest

Document created by Dave Murphy Employee on Jul 19, 2017
Version 1Show Document
  • View in full screen mode

Let's first define what a large image is in regards to a memory footprint in a load generator in CloudTest. The rule is anything larger than 2 MB should be consider 'large' and the below logic should be implemented as opposed to having the payload in the Data part of the message of the clip.

 

There are two main ways to handle this and this article will concentrate on the first method, but it should cover some of the steps of the second method too and the remaining steps can be deducted from the steps given in all this post. Let us know of any specific questions.

 

1. Base64 encode the files and put them in a single line (no carriage returns) in a csv, one file per line and upload them to an ftp server, Amazon S3 for instance. Later create a seed data file with the paths to those files in Amazon S3.

2. Base64 encode the files and put them in a single line (no carriage returns) in a csv, one file per line and create a new seed data in CloudTest with this data.

 

The main reason for doing the above is due to memory limitations if you keep the data in the clip in the Data section. Since CloudTest loads the clip before starting the composition, you might get an out of memory error when just trying to load a composition with a clip that is uploading a large file.

 

For the first method we have two variances depending on the application; the application can be uploading the file in chunks or the complete file at once. I'll explain in chunks and point out what to do different in the case of uploading the whole file, which is a simpler case.

 

Also the example here will concentrate when uploading multiple files in a single clip, if you just want to upload a single file per clip iteration, then you should be able to eliminate some extra steps described below and I'll be pointing out those differences too.

 

The steps are below:

 

Step 1. Base64 encode your files. I have a small shell script that you can run in terminal for MAC (if in windows, you'll have to somehow get to a similar one) and you can modify it according to your needs:

 

#!/bin/bash  # Declaring variables declare -r chunkSizeInBytes=6291456; declare -r startingFolder=Files; declare -r fileType=jpg;  find $startingFolder -name "*.$fileType" | while read filePath; do      fileName=$(echo $filePath | sed "s|$startingFolder/||")      folderName=${filePath%.$fileType}      echo "\nWorking on file "$filePath      echo "Creating Folder"      mkdir $folderName      echo "Splitting file into "$chunkSizeInBytes "bytes..."      split -b $chunkSizeInBytes $filePath $folderName/$fileName      echo "64encode all files and removing splitted files..."      find $folderName -name $fileName* | while read filePathSplittedFile; do           echo "64encoding "$filePathSplittedFile           # -A option is so that base64 processes the data on one line, this means NO carriage returns          openssl enc -base64 -A < $filePathSplittedFile > $filePathSplittedFile.csv           rm $filePathSplittedFile      done      echo "Transporting file list to csv and appending AWS URL"      ls $folderName | sed 's/^/https:\/\/s3-us-west-1.amazonaws.com\/projectcloudtest\//g' > $startingFolder/$fileName.csv done echo "\nAll DONE\n"

Important Notes Before running the shell script:

- Copy and paste the above code to a text file and save it with the extension sh.

- Create a new folder called Files in the same path you saved the sh file.

- The shell file can only work with single specific file types, so if the file types you want to work with are pdf, you need to modify the 6th line from jpg to pdf and so on for other file types.

- Put the files before running the shell script under the Files.

- As said before the shell is specific to the chunk of the files to 6291456 bytes, but configure this according to your application needs or just set it to a higher value than the biggest file to 64encode so that it puts them in single files if that is what is needed.

- Change the text on line 24 projectcloudtest to the bucket name you will create in step 2 in Amazon S3.

- You might also need to update line 24 west-1 for whatever region in S3 you are.

- The two last points don't check or upload anything on S3, they are just to create a url for you based on that.

 

So for example your folder structure should look like the on Figure 1:

 

Figure 1. Sample Folder Structure.

 

Where:
- The sh file and the Files folder are in the same path.

- The Files folder contains the files to 64encode, in this case 3 png images.

 

Once you run the shell script in terminal with the command "sh bashBase64Encode.sh" (your terminal session should be where the shell script is) you'll get something similar to Figure 2:

 

Figure 2. Shell output messages.

 

And the files and folders created should look like in Figure 3:

 

Figure 3. Files and Folders created after running the script.

 

Where:
- The csv files with the original file names contain paths you can use to reference them in CloudTest in a seed data object after being uploaded to Amazon S3.

- The folder with the original file names contain the Base64 encoded files as csv in a single line and are the files to upload to Amazon S3.

- When the chunkSizeInBytes is overpassed by any file, multiple files will be created as shown in Figure 4:

 

Figure 4. Multiple files created when chunking.

 

Step 2. Upload the Base64 encoded files to Amazon S3. Just access your S3 Amazon console, create a Bucket to put all the Base64 files there and upload them. Something very important is to set the Make everything public under Set Permissions in the upload wizard so that CloudTest can access them. Figure 5.

 

Figure 5. Check the box Make everything public when uploading the files.

 

Step 3. Create a seed data object in CloudTest that will have the urls to the Base64 encoded files.

 

If the files are not uploaded in chunks, you'll need to create a seed data file with the data needed to be uploaded according to your application, for instance see Figure 6. When just uploading a single file, then you will have a single row.

 

 

Figure 6. Uploading of the complete files, not in chunks.

 

You need to specify the URL and depending on your application other parameters, such as the file name and the file size.

 

If the files are uploaded in chunks, then you need to use a special character so that you can read one file per line, in my example I use the pipe "|" and the character to divide the URLs per file and then you'll have to write a JavaScript in your clip to separate them. See Figure 7 to see an example. When just uploading a single file in chunks, then you will have a single row.

 

Figure 7. Uploading of files in chunks.

 

Sample row in the above set up below, notice that there is a pipe in between each url:

 

https://s3-us-west-1.amazonaws.com/projectcloudtest/natur_1.pngaa.csv|https://s3-us-west-1.amazonaws.com/projectcloudtest/natur_1.pngab.csv|https://s3-us-west-1.amazonaws.com/projectcloudtest/natur_1.pngac.csv|https://s3-us-west-1.amazonaws.com/projectcloudtest/natur_1.pngad.csv

Step 4. Finally set it up in the clip. For this you need to do the below steps:

 

Step 4a. Declare a clip property seed data referring to the seed data created in step 3 and create a clip property for storing the actual Base64 Encoded data, in this case I called it binaryData.

 

Important Note: If uploading a single file per clip iteration, skip all below steps and get to step 4d.

 

When doing the uploading for multiple files in the same clip iteration, do steps 4b and 4c.

 

Step 4b. Wrap all the steps involved in the upload process into a group.

Step 4c. Create a seed data object in CloudTest that will have the urls to the Base64 encoded files. Set the Row count for that seed data property to the number of uploads per clip iteration you want. Set the group to repeat For Each and select the seed data property created in step 4a.

 

Figure 8 shows an example when the files are uploaded as a whole file:

 

Figure 8. Grouping main steps for uploading a single file.

 

Step 4d. "readFromURL Photo Data" JavaScript from Figure 8 is as below and for single uploads per iteration it should be set at the very beginning of the clip.

 

// Getting the value on the group iteration var URL = $context.currentGroup.forEachValue.URL; // Getting the data from the URL var dataList = $context.readFromURL(URL, "CSV", true); // Posting the length $context.result.postMessage($context.result.LEVEL_INFO, "Rows in input file: " + dataList.length);  // Setting the actual Base64 encoded data to the clip property binaryData var binaryData = dataList[0]; $prop.set("clip", "binaryData", binaryData);

For single file upload per iteration change to get the URL from the clip property instead of the group:

 

var URL = $prop.value("clip", "ClipSeedData").URL;

And the JavaScript for changing the transaction name "S20: Append fileName to nextItem Transaction" in Figure 8 is not required but recommended for analytics:

 

// Change Name of the next Transaction var nextItem = $context.currentItem.nextItem; var name = nextItem.name + " " + $context.currentGroup.forEachValue.fileName; nextItem.name = name;

 The same goes for the one changing the page name "S20: Change Name nextItem to fileName" in Figure 8:

 

// Change Name of the next Item var nextItem = $context.currentItem.nextItem; var name = $context.currentGroup.forEachValue.fileName; nextItem.name = name;

 Remember to update when using a single upload per clip iteration to:

 

var name = $prop.value("clip","ClipSeedData").fileName;

 

Figure 9 shows an example when the files are uploaded in chunks, skip to step 4e if not applicable. In addition to the creating the group and repeating it For Each, you'll need to create a clip property called URLsArray and set the actual step that uploads the file into a chain and set the chain to repeat For Each URLsArray:

 

Figure 9. Grouping main steps for uploading a file in chunks.

 

"Saving URLs to URLsArray" JavaScript is as:

var URLs = $context.currentGroup.forEachValue.URLs;
var URLsArray = URLs.split("|");

$prop.set("clip", "URLsArray", URLsArray)

The JavaScript "Setting up binaryData with readFromURL" is:

 

// Getting the value on the group iteration
var URL = $context.currentChain.forEachValue;
$context.result.postMessage($context.result.LEVEL_INFO, "URL: " + URL);
// Getting the data from the URL
var dataList = $context.readFromURL(URL, "CSV", true);
// Posting the length
$context.result.postMessage($context.result.LEVEL_INFO, "Rows in input file: " + dataList.length);
// Setting the actual Base64 encoded data to the clip property binaryData
var binaryData = dataList[0];
//var binaryData = dataList[0];
$prop.set("clip", "binaryData", binaryData);

Step 4e. Open the message where the actual data should go and update the Data part with the property binaryData. Also update any other data from the seed data property such as file name and file size for this message and other messages that use them accordingly. Figure 10 shows the example of uploading multiple files in the same clip since it uses {%% sys-prop : group : ForEachValue.fileName %%}. If a single upload then change to {%% prop : parent-clip : /ClipSeedData.fileName %%}, etc. The {%% prop : parent-clip : /binaryData %%} should be the same for any situation.

 

Figure 10. Updating the uploading part of the clip and other elements.

 

Finally make sure the Base64 Encoding icon is checked and not grayed out:

 

Figure 11. Base64 Encoding must be checked.

Attachments

    Outcomes