With the increased focus on DevOps, as well as our own DPM methodology of 'Monitor-Optimize-Validate', a reasonable follow-up question might be 'How do we use mPulse data in an automated DevOps process?'.
Fortunately, there's an answer to that: mPulse has a pretty well developed and documented query API (here ) which allows you to query mPulse data and get performance data from any time period (and with any filters you'd like). This means that if you've just made a configuration change a couple of hours ago and would like mPulse to 'validate' that performance hasn't degraded, you can query mPulse data of the past week and compare it to data for the past 2 hours to see if there's any degradation in performance.
To make this a little simpler, I've created a Java .jar file (attached at the bottom of this article) which you can execute from the command line. You'll need to pass in two time periods, as well as a threshold, and user credentials. It will query the two time periods to see if performance in the 'test' time period was more than x% above performance in the 'baseline' time period.
A sample command is as follows:
java -jar DynamicAlerting.jar baselineStartDate=-2weeks baselineEndDate=-2days baselineMultiplier=1 testStartDate=-3hours testEndDate=now globalThreshold=100 apiToken=ea57e6bf-8ac5-***-YYYY-892dd912beca allowedPercentOfPageGroupsFailing=20.0 minMeasurements=10 name="mPulse Demo" apiKey=PH7E4-H9YBZ-6NM8T-L4UQK-ZJAPW tenantName="SOASTA SE Demo"
Some notes on the command line arguments:
- The apiToken can be found by logging into mPulse and going to 'My Settings' and clicking the 'generate API token' button.
- The apiKey is the apiKey for the mPulse app whose measurements you'd like.
- The tenantName is necessary for mPulse users who have more than one tenant that they can login to.
- baselineMultiplier indicates what you consider acceptable performance for the 'test' pageGroups. A value of 1.1 indicates that the test pageGroups will fail if pageload time is more than 1.1*pageload time of baseline. In the example above, I put baselineMultiplier=1 which will cause a page group to fail if it is any slower than the baseline time.
- allowedPercentOfPageGroupsFailing=20.0 indicates we will allow up to 20% of the page groups to fail before we mark the build as 'OVERALL_FAIL'
- minMeasurements indicates that we won't bother calculating pass/fail if there are less than that many measurements.
A full list of the command line arguments is listed below. Output is an HTML file, an XML file and a few csv files. The HTML file looks like this:
You can see that only one of the pageGroups failed to meet the threshold. However, since we allow the build to proceed if less than 20% of pageGroups had failures, so the overall job would be considered a 'PASS'.
So far, the HTML document is good for visibility, but to use this information in an automated process. To run this in Jenkins you'll need to copy the .jar file to a Jenkins workspace directory and then just execute it as a shell command in Jenkins:
To tell Jenkins to pass or fail based on the overall status, probably the easiest option is install a free plugin called TextFinder and have it pass the job if the text 'OVERALL_PASS' appears in the HTML:
Alternately, if you would like Jenkins to report the test/fail status of every page group (and it would automatically call the build unstable if any pageGroup failed) then create a post-build step to 'Publish JUnit test result report' and point it to the XML file created by the .jar file:
Although the 'Junit XML' method of reporting data into Jenkins will automatically fail if only one pageGroup fails, it does have the added benefit of reporting every test page as a result into Jenkins (which may be desired by some customers):
Lastly, if you would like to plot the RUM data from build to build, you can install a free Jenkins plugIn called the 'Plot Plugin' and point it to any of the .csv files that are created in the workspace by the .jar file:
This will allow you to plot build-to-build performance differences in Jenkins as measured by mPulse:
Please note that the .jar file has several other options available from the command line. If you run 'java -jar DynamicAlerting.jar', it will give you a full list of options which I may cover in a separate Blog post.
Note: Updated 3/16/2018 to better handle scientific notation in the returned JSON results.